LITERACY, LANGUAGE AND LEARNING INITIATIVE (L3) NATIONAL FLUENCY ASSESSMENT OF RWANDAN SCHOOLS MIDLINE REPORT January 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Education Development Center, Inc. and do not necessarily reflect the views of USAID or the United States Government. National Fluency Assessment of Rwandan Schools: Midline Report 2015 UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT (USAID) LITERACY, LANGUAGE AND LEARNING INITIATIVE (L3) NATIONAL FLUENCY ASSESSMENT OF RWANDAN SCHOOLS: MIDLINE REPORT Cooperative Agreement AID-696-A-11-00006 EDUCATION DEVELOPMENT CENTER (EDC) Date: January 2016 DISCLAIMER: This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Education Development Center, Inc. and do not necessarily reflect the views of USAID or the United States Government. iii EXECUTIVE SUMMARY The Literacy, Language and Learning Initiative (L3), funded by the U.S. Agency for International Development (USAID) and implemented by a partnership led by the Education Development Center, Inc. (EDC), is designed to help improve Rwandan early grade learners’ language and mathematics skills. L3 is assisting the Rwandan Ministry of Education (MINEDUC) in the implementation of a comprehensive early literacy and mathematics program, including support for transition to English as a medium of instruction in the 4th grade (P4). To gather information on learner achievement and support Rwandan Education Board (REB) in establishing a system of regular national assessments, L3 conducts annual literacy and mathematics assessments with a random sample of learners drawn from a nationally representative sample of schools. Assessments are conducted in the language of instruction (Kinyarwanda in grades P1 through P3, and English in P4), and were developed by a team of experts from the REB and L3 based on a) international standards for testing and measuring learners’ oral reading fluency in the early grades, and b) existing Rwandan grade level standards in literacy and mathematics. This report presents results of the October 2015 L3 midline assessment of 2580 learners, 433 teachers and 60 head teachers from 60 schools. The results of this assessment are compared with the baseline assessment conducted in October 2014. The endline assessment is planned for October of 2016. Oral Reading Fluency Assessment (FARS). Analysis of baseline and midline FARS results showed that learners in all tested grades showed improvement in reading from baseline to midline. Primary 4 was assessed for the first time; their scores will serve as a baseline. Primary 1, 2 and 3 showed statistically significant gains (p<.001) in average FARS scores from baseline to midline. In measuring improvements in the number of words read correctly per minute, P1 and P3 learners showed most gains. P1 showed an average increase in FARS of 2.7 wcpm (±1.1 wcpm). P2 demonstrated gains from baseline BOX 1. AVERAGE NUMBER OF WORDS READ CORRECTLY IN A MINUTE, BY GRADE 4.8 19.2 wcpm 22.1 wcpm 40.6 wcpm 7.5 21.5 wcpm 25.1 wcpm English baseline, 26.2 wcpm P1 P2 P3 P4* Baseline Midline * Baseline data for P4 were collected in 2015; endline will be collected in 2016 along with P1-P3 National Fluency Assessment of Rwandan Schools: Midline Report 2015 iv to midline with an average increase of 2.3 (±2.0 wcpm) P3 showed average gains of 3.0 wcpm (± 1.8 wcpm). The assessment found that more P2 and P3 learners could read fluently compared to their peers from the same grades who were tested the previous year, according to the proficiency standards established by REB. The percent of P1 learners who could read 20 words correct per minute or faster more than doubled between the baseline and the midline. The percent of P2 learners who could read a grade level text with some oral reading fluency (over 20 words per minute) increased from 50% to 56%. The percent of P1 learners with non-zero scores increased by 10%. All changes are statistically significant at p<.001 level. The table below shows the changes between the baseline and the midline in oral reading fluency expressed as words correct per minute and percent of learners with zero scores, along with the associated effect size. BOX 3. BASELINE-MIDLINE GAINS ON FARS, BY GRADE Grade Subtest BASELINE MIDLINE GAIN EFFECT SIZE (Cohen’s d) P1 P2 P3 Words Per Minute 4.76 7.48 2.70 (±1.06) 0.29 (± .12) Zero Score 60.3% 50.4% - 10.10 (±0.06 ) 0.20 (± .11) Words Per Minute 19.2 21.5 2.30 (± 2.00) 0.13(± .11) Zero Score 32.7% 25.5% - 7.20 (± 0.05) 0.16(± .11) Words Per Minute 22.1 25.1 2.99 (±1.79) 0.19(± .11) Zero Score 21.3% 18.6% - 2.71 (±0.05) 0.07(± .11) An analysis of proficiency rates by sex in P2 and P3 showed that girls were more likely to complete the reading of the passage than boys. At baseline, the difference was statistically significant among learners, at p<.001. At midline, this difference is statistically significant in oral reading in Kinyarwanda in all four tested grades. An analysis of learner proficiency results by sex showed that girls both started significantly higher and improved more than boys BOX 2. PERCENT OF P2 LEARNERS READING AT GRADE LEVEL zero 33% zero 25% 1 to 19 17% 1 to 19 18% 20 to 32 25% 20 to 32 27% 33 to 47 20% 33 to 47 22% over 47 5% over 47 7% zero scores and developing readers proficient and advanced readers P2 baseline P2 midline National Fluency Assessment of Rwandan Schools: Midline Report 2015 v between the baseline and the midline. Further investigation is needed to establish the reasons why girls are learning to read better than boys. Oral reading comprehension results were mixed. At midline, 13% of P1 learners met the threshold of 80% comprehension, 12% of P2 learners, and 17% of P3 learners. Almost 40% of P4 learners were able to meet the threshold of 80% in Kinyarwanda, but only about 11% of P4 learners answered four or five questions in English. P2 and P3 learners who answered 4 or 5 comprehension questions read the text with the average speed of 37.5 and 49.0 wcpm, respectively; 12.6% of P3 learners read with the grade-level speed of 33 wcpm or faster and answered 4 or 5 comprehension questions, compared to 5% at baseline. Only a handful of P1 learners were able to read over 80% of P1 text correctly, and those learners answered all or almost all comprehension questions correctly. The report also details innovations in testing procedures, in response to REB’s queries on timed and untimed testing and growing interest in this area internationally. The data collectors administered a timed and untimed test. During the second round, assessors gave the text back to the learners and allowed them to finish reading the passage (if they hadn’t done so already), and then asked them comprehension questions without taking the text away. As shown in Box 4, removing time and memory barriers notably increased comprehension scores among all groups. Mathematics Procedural Fluency Assessment (MARS). The mathematics assessment was developed by EDC mathematics experts based on a review of the Rwandan mathematics curriculum in early grades relative to international standards of mathematics instruction. All tasks test grade-level procedural fluency in basic mathematical concepts, hence tasks for each grade were more difficult than tasks for the previous grade. BOX 4. COMPARISON OF TIMED AND UNTIMED READING COMPREHENSION RESULTS, BY GRADE timed, 1.01 timed, 1.48 timed, 2.21 timed, 2.83 timed, 0.98 untimed, 1.61 (out of 5) untimed, 2.14 (out of 5) untimed, 3.58 (out of 6) untimed, 4.23 (out of 5) untimed, 1.4 (out of 5) P1 P2 P3* P4 KR P4 English * P3 test had 6 comprehension questions. All other grades had 5 comprehension questions. National Fluency Assessment of Rwandan Schools: Midline Report 2015 vi Analysis of baseline and midline MARS results found that learners in P1, P2, and P3 showed improvement in mathematics after one year of L3 implementation. Box 5 shows the average percent of MARS tasks solved correctly at baseline and midline, by grade. All grades showed statically significant gains (p<.001). P1 showed the largest gains from baseline to midline with an average increase of 22.2% (± 3.1%) in the percent of MARS tasks solved correctly. P2 and P3 also demonstrated significant gains from baseline to midline with an average increase of 8.4% (±2.9%) and 7.4% (± 2.7%), respectively. Overall, analysis showed a very large effect size (d=0.81) for P1 learners, which suggests that, at midline, 79% of P1 learners scored higher on the MARS assessment than P1 baseline learners. Analysis of gains for P2 and P3 showed effect sizes of 0.33 and 0.31 respectively, meaning that, at midline, 62% of P2 and P3 learners scored higher on the MARS than at baseline. Analysis of MARS assessment results showed that the percent of P1, P2 and P3 learners with zero scores decreased from baseline to midline. About 14 percent of P1 learners and 7 percent of P2 learners could not solve a single mathematics problem at baseline. By midline, the percent of P1 learners with zero scores had decreased significantly (p<.01) to 8.3%. P2 and P3 learners did not show statistically significant decreases in zero scores on the MARS assessment. The L3 program contributes to making the learning environment more gender-balanced and is sensitive to how teaching practice and learning materials impact learning among girls and boys. One year of program exposure was found to have a beneficial effect on girls learning math. On average, girls in P1, P2 and P3 showed larger gains than boys from baseline to midline. Girls in P1 demonstrated largest gains between the baseline and midline, more than doubling the percent of problems answered correctly. The effect sizes of the change between baseline and midline by sex were large across the board, ranging from d=.29 for P3 boys to d=1.03 for P1 girls. Impact of Contextual Factors on Achievement. Data analysis revealed a variety of factors associated with learners’ performance in oral reading, in math, or in both. Both at baseline and at midline learners’ age was found to negatively correlate with the achievement in reading and BOX 5. AVERAGE PERCENT CORRECT ON MARS SUBTESTS, BY GRADE 25.7% 33.9% 39.0% 51.9% 47.9% 42.2% 46.4% P1 P2 P3 P4* Baseline Midline * Baseline data for P4 was collected in 2015; midline data not available National Fluency Assessment of Rwandan Schools: Midline Report 2015 vii math. The older the tested learner was, the lower his/her results would be. Having a literate mother, having someone reading to a learner at home, and checking homework regularly were found to be positively associated with reading results. Repeating a grade, having a sibling who repeated a grade, missing school or being late for school were, unsurprisingly, found to be negatively associated with reading results. Finally, among school characteristics, teacher absenteeism and a distance of the school to the District Office were found to be negatively associated with both reading and mathematics results of learners: the farther away from the District Office the school was located, the higher was the teacher absenteeism and lower learner results in reading and math. School-Level Results. An analysis of school averages in each grade showed an improvement of average results across schools in each grade on both tests. Assessor interviews indicated that schools on the lower end of average achievement are located in the areas with higher levels of poverty, while schools that perform better, on average, have excellent leadership from their head teachers. Some schools that do not perform well are located either near the border where there is a high proportion of transient population, or in very sparsely populated areas where learners are obliged to travel far to school and attendance (and thus time on task and opportunity to learn) suffers as a result. The analysis of average learner performance in study schools also showed that, on average, the same schools did well at baseline and at midline. Statistical comparisons of background characteristics of the top performing and bottom performing schools demonstrated a few significant differences in these schools. The most notable differences were in the distance from school to District Office, and in the average percent of absent teachers on the day of the test and the preceding day. The average distance to District Office among top performing schools was 12.8 kilometers, while the average distance to District Office among bottom performing schools was 44.7 kilometers. The average percent of absent teachers over two days was only 3.7% in the top performing schools, compared to 18.9% in the bottom performing schools. National Fluency Assessment of Rwandan Schools: Midline Report 2015 viii The correlation between the distance to District Office and teacher absenteeism is very strong (r = .511) in the general dataset. Repeaters. On average, teachers reported that 15.6% of learners in their classrooms were repeaters. To better understand the impact of repeating a grade, the study tracked participants from baseline to midline. Assessment results of those 175 who were found at midline to be repeating the same grade they were in at baseline were analyzed to determine how effective grade repetition was in improving learner achievement. Most repeaters were found to be in P1 and to be, on average, older than their non-repeating peers. The proportion of boys and girls was similar to that of non-repeaters. Teachers reported that the majority of repeaters were not orphans and did not have learning barriers. However, these learners were reported to miss school or be late for school more often than their non-repeating peers. The study was able to track and re-test fewer than a half of repeating learners (6.8% out of teacher-reported 15.6%). Most of re-tested learners were found to have made substantial gains in reading and math over the course of one academic year. These findings are moderated by existing research that shows that initial achievement gains that occur during the year the student is retained will decline within 2-3 years of retention, such that retained children either do no better or perform more poorly than similar groups of promoted children. Many studies have shown that grade retention had a negative impact on all areas of achievement (reading, math and language) and socio-emotional adjustment (peer relationships, self-esteem, problem behaviors, and attendance). Based on these findings, the report’s general recommendation is to provide support such as remediation for learners who fall behind to help advance them together with their age cohort. Schools, Teachers and Learners. In addition to testing learners, the assessment team also collected a vast amount of data on the school, teacher and learner-level factors that might impact learning. The study found that most schools fall within the range between poor and adequate infrastructure. About a third of study schools reported receiving support from local or international organizations/NGOs, mostly in the form of provision of teaching and learning materials, teacher training or infrastructure. Schools were found to have large classrooms, particularly in earlier grades: average learner to teacher ratio in P1 was found to be 66 learners to one teacher, 58 learners to a teacher in P2, and 54 learners to a teacher in P3. On examination of the school registers, it was noted that many learners were found not to attend regularly; on the day of the assessment, about a third of a class was not found to be present, on average. Data collectors also reported that most teachers had difficulty identifying the learners in their class by name and they often had to ask other learners to determine if a learner had repeated, had left the school or was just absent for the day. National Fluency Assessment of Rwandan Schools: Midline Report 2015 ix Teachers reported that overcrowding was an important barrier to effective teaching. On average, teachers reported that 15.6% of learners in their classrooms were repeaters or have repeated the same grade. The official rate of grade repetition in Rwanda was reported to be 12.5% in 2012 (Mineduc Rwanda Education Statistical Yearbook, September 2014) Teachers reported that learner lateness and absenteeism were the main challenges to teaching both reading and math. In many schools, teachers reported that learners come to school hungry. This is confirmed by the learner interview which found that 28% of learners did not have anything to eat before coming to school on the day of the assessment. Observed conditions in sampled classrooms showed that the majority of classrooms were in adequate or good condition with respect to blackboards, clean classroom space, good lighting, desks for learners, and reading and writing materials for learners. In more than half of observed schools print materials (posters, signs, etc.) were observed on school or classroom walls. Teachers reported having received materials and instructional technology from L3. Teaching and learning materials were observed in use by teachers; in 18% of schools they were found in the library. L3-provided learner books were also observed in schools. In the majority (76%) of observed schools learner books were observed in use by learners or on the classroom shelves. In a few schools, learner books were found in the library or the headmaster’s office. In nearly all schools, learner books looked used. Teachers reported having received cell phones, speakers, and SD cards from L3; the majority said they use technology at least once a week. Most teachers reported using technology two to four times a week. Teacher attendance records showed that on average, on a given day, 7.5% of all P1, P2, P3 teachers were absent. The percent of absent teachers was found to be much higher in more remotely located schools. CHALLENGES TO TEACHING PRIMARY GRADES:  Learner absence  Learner lateness  Learners come to school hungry  Different levels of reading ability of learners in the classroom  Overcrowded classrooms/large class size  Lack of electricity  Learners’ parents are not engaged and do not participate in reading development of their children  Age differences between learners  Not enough books/materials  Learners did not attend pre-primary school National Fluency Assessment of Rwandan Schools: Midline Report 2015 x Summary of Findings. Overall, oral reading fluency assessment results show that the proportion of learners reading on grade level increased very substantially over the course of the first year of L3 implementation nation-wide. Most of those learners who were able to read the text were also able to answer some or all comprehension questions. Since literacy instruction is conducted in the mother tongue of the vast majority of learners, it is probable that the major obstacle to reading is decoding. Comparisons in comprehension rates between timed and untimed reading showed a significant improvement in comprehension rates when learners are allowed to finish reading the text and refer to the text when answering comprehension questions. Mathematics assessment results show that, at midline, a large proportion of P1, P2 and P3 learners are still developing basic mathematics skills that would enable them to perform grade￾level number operations with accuracy and speed. The majority of P1, P2 and P3 learners were able to work out several adding and subtracting problems correctly at midline, but very few learners demonstrated grade-level procedural fluency on elementary mathematics operations. However, analysis of MARS results shows that learners in P1, P2, and P3 showed statistically significant gains (p<.001) in average scores from baseline to midline. Notably, girls demonstrated significant gains between the baseline and midline, reducing or eliminating the gender gap that was observed at baseline. These findings suggest significant improvement in mathematic achievement of P1, P2 and P3 learners since the roll-out of the L3 nationwide intervention, particularly among girls. Recommendations. Review and revision of system-level policies that impact the process of instruction, learner and teacher attendance, learner repetition, teacher class assignments, among other, could have a positive impact on overall learner achievement. Specifically, the review/revision of policies relating to the following issues is recommended: - At the heart of the Rwandan curriculum lies an explicit emphasis on the development of skills and competences for lifelong learning and for operating effectively in society. The curriculum also promotes formative assessment to monitor learner progress and make appropriate instructional decisions. Currently early grades do not have class teachers, only subject teachers. During data collection, teachers and data collectors observed that there is little or no time to gather, analyse and use assessment information to improve learning and inform planning. This inhibits teachers’ ability to get to know their pupils personally, differentiate appropriately, as well as the effectiveness of instructional practice. An approach where a single teacher teaches all subjects to an assigned class is recommended. This would enable teachers to provide a broad and balanced curriculum for all learners and allow them to frequently integrate formative assessments during typical daily activities. National Fluency Assessment of Rwandan Schools: Midline Report 2015 xi - Teacher and learner absenteeism was found to be associated with poor academic results. An important finding of the assessment is that the farther the school is from the District Office, the more likely the teachers and the learners to not come to school. It is recommended that policies be put in place to counteract this trend. - Grade repetition contributes to overcrowding early grade classrooms. Providing class teachers with professional development programmes to effectively implement continuous assessments and remediation activities is recommended. This would enable teachers to differentiate instruction effectively, given large class sizes, provide remediation for struggling learners and support these learners to catch up and progress with the rest of their peers. - Annual competence-based assessments would provide consistent information on learner performance on key indicators, such as grade-level reading procedural mathematics fluency. Data on learner achievement, class size, repetition and other central issues in education should be used to inform policy. Finally, the proposed Teacher Development and Management Policy, November 2015 prioritizes the development and implementation of a well-structured program of Continuous Professional Development for teachers to improve all aspects of the quality of education, especially in literacy, numeracy and essential life skills. Ensuring that learners achieve the foundations of learning in the early grades is a vital way of overcoming early disadvantage; well-trained teachers are key to improving pupils’ early learning. In order to achieve these objectives an explicit focus on early grade instruction and recognition of its importance as a foundation for life-long learning should be incorporated into pre-service and in-service teacher training programmes. Public awareness campaigns promoting the importance of parental and community involvement in the development of early grade literacy and numeracy competences should also be considered. National Fluency Assessment of Rwandan Schools: Midline Report 2015 xii National Fluency Assessment of Rwandan Schools: Midline Report 2015 xiii The Literacy, Language and Learning (L3) Initiative’s strategic objective is to strengthen teaching and learning so that children leave primary school with solid literacy and numeracy skills. L3 works with Rwanda’s Ministry of Education (MINEDUC) to improve students’ reading and mathematical skills in grades one to four, as well as their English language proficiency. Working in collaboration with the MINEDUC, USAID and technical partners, the L3 project works with pre￾service and in-service facilitators to introduce proven reading and mathematics teaching strategies, and with community volunteers to support learning. The project also aims to improve the availability and use of innovative reading and math instructional materials. Teachers’ and students’ reading, math and English language skills is reinforced through interactive audio instruction programs. The L3 initiative has five intermediate results that support the strategic objective, and ultimately contribute to USAID’s goal of improved reading skills for 100 million children in primary grades by 2015. These results and key activities include: IR 1: Improved Quality of Teaching 1. Development of a shared vision of effective literacy/numeracy instruction and tools to measure progress with respect to that 2. Implementation of a School-based Mentoring Program to support enhanced literacy, numeracy and ESL instruction 3. Support to TTCs to become Centers of Excellence for Literacy and Numeracy Instruction 4. Pilot initiatives to improve teachers’ motivation and working conditions IR 2: Improved Availability of Teaching and Learning Materials 1. Develop a complete package of instructional materials to support early grade reading 2. Hold Math Camps for teachers and story writing competitions and Writer’s Workshops to produce locally developed reading materials 3. Distribute over one million supplementary books 4. Introduce “traveling libraries” in low income, rural communities 5. Distribute sustainable technologies to support enhanced literacy/numeracy instructional program 6. Hold local campaigns and activities to promote a culture of reading IR 3: Support for English 1. Develop interactive audio programs for, P1 to P4 2. Develop an instrument to evaluate teachers’ English language proficiency 3. Support the revision of the existing English as a Second Language (ESL) curriculum IR 4: Strengthened Ministry Capacity 1. Embed L3 literacy/numeracy specialists in the central MINEDUC and the 13 TTCs to provide day-to-day support in literacy/numeracy and teacher training reforms 2. Develop a criteria-based classroom observation form to monitor changes in teachers’ literacy/numeracy instructional practices over time 3. Provide short-term technical support to the Examinations division to strengthen student literacy/numeracy assessment programs IR 5: Improved Equity in Education 1. Include new instructional materials with positive images of girls and other marginalized groups 2. Provide additional supports and inputs to students in low-income and rural areas L3 OVERVIEW National Fluency Assessment of Rwandan Schools: Midline Report 2015 xiv TABLE OF CONTENTS Executive Summary ................................................................................................................................................iii Table of Contents ..................................................................................................................................................xiv Table of Figures ................................................................................................................................................ xvi Table of Tables ................................................................................................................................................ xviii Acronyms .................................................................................................................................................................. xx Acknowledgments ................................................................................................................................................ xxi Introduction................................................................................................................................................................ 1 Study Participants .................................................................................................................................................... 4 Oral Reading Fluency Assessment of Rwandan Schools (FARS) ............................................................ 8 Oral Reading Fluency ......................................................................................................................................... 8 Comprehension ................................................................................................................................................ 15 Untimed Reading Results .............................................................................................................................. 18 Impact of Contextual Factors on Reading .............................................................................................. 22 Learner Characteristics and FARS Results .......................................................................................... 22 School Characteristics................................................................................................................................ 25 Summary ............................................................................................................................................................. 26 Mathematics Assessment of Rwandan Schools (MARS) ........................................................................ 27 Summary Findings ........................................................................................................................................... 27 Detailed P1 MARS Findings .......................................................................................................................... 33 Detailed P2 MARS Findings .......................................................................................................................... 34 Detailed P3 MARS Findings .......................................................................................................................... 36 Detailed P4 MARS Findings .......................................................................................................................... 38 Impact of Contextual Factors on Mathematics Achievement ......................................................... 40 Learner Characteristics and MARS Results ........................................................................................ 40 School Characteristics................................................................................................................................ 41 Summary ............................................................................................................................................................. 42 Reading and Mathematics School-Level Results ...................................................................................... 43 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xv Study of Grade Repeaters .................................................................................................................................. 47 Description of the Sample of Grade Repeaters .................................................................................... 47 Performance of Repeaters in Reading ..................................................................................................... 49 Performance of Repeaters in Math ........................................................................................................... 51 Teachers’ Reports about Repeaters .......................................................................................................... 53 Evidence from Other Studies ....................................................................................................................... 54 School, Teacher and Learner-Level Findings .............................................................................................. 56 School Environment ........................................................................................................................................ 56 Teachers ............................................................................................................................................................... 57 School Leadership (Policies, Practices, and Monitoring) ................................................................... 60 Learning Environment .................................................................................................................................... 63 L3 Materials and Technology....................................................................................................................... 64 Teaching Practices and Beliefs .................................................................................................................... 66 L3 Training and School-Based Mentors .................................................................................................. 68 Parental and Community Involvement .................................................................................................... 68 Learner Context Interview Findings ................................................................................................................ 71 Recommendations ................................................................................................................................................ 79 Appendix A: Methodology ................................................................................................................................ 81 Assessment Questions ................................................................................................................................... 81 Fluency Assessment of Rwanda Schools (FARS) Tools ................................................................. 82 Mathematics Assessment of Rwanda Schools (MARS) Tools..................................................... 83 Data Analysis ...................................................................................................................................................... 86 Sampling Parameters ...................................................................................................................................... 86 Sampling Process ............................................................................................................................................. 86 Limitations of Design ...................................................................................................................................... 88 Appendix B: Data Collection ............................................................................................................................. 93 Appendix C. Data Collection Tools ................................................................................................................. 96 Student Context Survey ................................................................................................................................. 96 P1 Assessment ................................................................................................................................................. 100 P2 Assessment ................................................................................................................................................. 106 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xvi P3 Assessment ................................................................................................................................................. 112 P4 Assessment ................................................................................................................................................. 119 School Monitoring Form ............................................................................................................................. 129 Grade Monitoring Form ............................................................................................................................... 139 School Observation Form ........................................................................................................................... 145 Teacher Questionnaire about Repeaters in the Study ..................................................................... 147 Appendix D. Detailed Results of Statistical Analyses of FARS and MARS Midline Results ..... 149 Detailed Results for FARS Subtests ......................................................................................................... 149 Detailed Results for MARS Subtests ....................................................................................................... 151 TABLE OF FIGURES Figure 1. Demographic characteristics of sampled learners (n=2,405) ............................................... 6 Figure 2. Age by grade ........................................................................................................................................... 7 Figure 3. Average number of words read correctly in a minute, by grade ........................................ 8 Figure 4. Average percent correct on FARS, by grade ............................................................................... 9 Figure 5. Percent of learners reading at grade level, by grade ............................................................ 11 Figure 6. Percent of words read correctly, grouped by grade ............................................................. 13 Figure 7. Percent of words read correctly grouped, by sex and grade............................................. 14 Figure 8. FARS comprehension results, by grade ..................................................................................... 16 Figure 9. Comprehension results among learners who read 80-100% of the text ...................... 18 Figure 10. Measuring reading comprehension......................................................................................... 19 Figure 11. Comparison of timed and untimed reading comprehension results, by grade ....... 19 Figure 12. Comparison of timed and untimed reading comprehension results ........................... 20 Figure 13. Percent of P2 learners reading and understanding grade level text ............................ 21 Figure 14. Average percent correct on MARS subtests, by grade ...................................................... 28 Figure 15. Percent of learners with zero scores on all MARS tasks .................................................... 30 Figure 16. Average automaticity/ fluency in solving mathematics problems at midline, by grade ...................................................................................................................................................................................... 30 Figure 17. Average percent correct on MARS tasks, by grade and sex ............................................ 32 Figure 18. Overall P1 MARS scores at baseline and midline ................................................................ 33 Figure 19. Average percent correct on P1 MARS subtests at baseline and midline .................... 33 Figure 20. Number of problems solved on P1 MARS subtests ............................................................ 34 Figure 21. Overall P2 MARS scores at baseline and midline ............................................................... 35 Figure 22. Average percent correct on P2 MARS subtests at baseline and midline .................... 35 Figure 23. Number of problems solved on P2 MARS subtests .......................................................... 36 Figure 24. Overall P3 MARS scores at baseline and midline ............................................................... 37 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xvii Figure 25. Average percent correct on P3 MARS subtests at baseline and midline .................... 37 Figure 26. Number of problems solved on P3 MARS subtests at baseline and midline .......... 37 Figure 27. Overall P4 MARS scores at baseline.......................................................................................... 38 Figure 28. Average percent correct on P4 MARS subtests at baseline ............................................. 38 Figure 29. Number of problems solved on P4 MARS subtests at baseline ................................... 39 Figure 30. MARS results and language preference for doing math, P4 ........................................... 39 Figure 31. Correlation between Learner Age and mars results ........................................................... 40 Figure 32. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P1 learners on oral reading fluency test in study schools ..................................................................... 43 Figure 33. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P2 learners on oral reading fluency test in study schools ..................................................................... 43 Figure 34. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P3 learners on oral reading fluency test in study schools ..................................................................... 44 Figure 35. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P1 learners on MARS in study schools .......................................................................................................... 45 Figure 36. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P2 learners on MARS in study schools .......................................................................................................... 45 Figure 37. Average percent correct at baseline (SY 2014) and midline (SY 2015) among tested P3 learners on MARS in study schools .......................................................................................................... 45 Figure 38. Factors correlated with school performance (n=16) .......................................................... 46 Figure 39. Longitudinally tracked learners and repeaters ..................................................................... 47 Figure 40. Percent of repeaters, by grade .................................................................................................... 48 Figure 41. Demographic characteristics of repeaters compared to non-repeaters ..................... 49 Figure 42. Average oral reading fluency results among repeaters and non-repeaters, at midline ...................................................................................................................................................................................... 50 Figure 43. Changes in fluency over the course of 2015 academic year among repeaters........ 51 Figure 44. Average mathematics results among repeaters and non-repeaters, at midline ...... 52 Figure 45. Changes in MARS over the course of 2015 academic year among repeaters .......... 53 Figure 46. Reasons why learner was retained in current grade (n=170) .......................................... 53 Figure 47. Background characteritics of repeaters (n=170) .................................................................. 54 Figure 49. Observed condition of school infrastructure in sampled schools (n=58) .................. 57 Figure 50. Percent of repeaters in P1-P3, on average, in the study schools (n=59) .................... 59 Figure 51. Methods used by prinicpals/head teachers to monitor the reading progress of learners in schools (n=60; multiple response) ........................................................................................... 61 Figure 52. Methods to encourage learners to attend school (n=60; multiple response) .......... 62 Figure 53. Teachers’ reasons that a learner can be held back a year (n=433)............................... 62 Figure 54. Materials the school administration provides teachers with (n=433, multiple repsonse) .................................................................................................................................................................. 63 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xviii Figure 55. Common challenges to teaching and learning in schools (n=60, mutliple response) ...................................................................................................................................................................................... 64 Figure 56. Percent of teachers who received technology from L3, by grade (n=433) ................ 66 Figure 57. How often do you use the technology you received from L3 to teach learners this subject? (n=433) .................................................................................................................................................... 66 Figure 58. Documents teachers use when preparing lesson plans (n=433, multiple response) ...................................................................................................................................................................................... 66 Figure 59. Which items must parents purchase for a learner to attend school? (n=60) ........... 69 Figure 60. PTA members trained by concern worldwide ....................................................................... 69 Figure 61. Home environment Interview results (n=2,405) ................................................................... 73 Figure 62. How often do you miss school or are late? (n=2,405) ...................................................... 73 Figure 63. Which of the following do your parents expect you to do regularly? (n=2,405, multiple response) ................................................................................................................................................ 74 Figure 64. School/teacher factors Interview results (n = 2,405) .......................................................... 74 Figure 65. What do you like about school? (n=2,405, multiple response) ...................................... 75 Figure 66. What do you not like about school? (n=2,405, multiple response) .............................. 75 Figure 67. Socio-economic status Interview results (n = 2,405) .......................................................... 76 Figure 68. What light do you have at home? (n=2,405) ........................................................................ 76 Figure 69. School/Teacher are supportive of data collection (n=223) ............................................. 94 Figure 70. Assessments were conducted with Few interruptions (n=183) ...................................... 94 Figure 71: Percent of Students Who Understood the Language the Assessor USed ................. 95 Figure 72. Problems during data collection (n=183) ............................................................................... 95 TABLE OF TABLES Table 1. Sample of schools and teachers, by grade and province ........................................................ 5 Table 2. Baseline-midline gains on FARS, by grade ................................................................................. 10 Table 3. Oral reading proficiency thresholds .............................................................................................. 10 Table 4. Baseline-midline gains in proficiency rates, by grade ............................................................ 12 Table 5. Baseline-midline gains in proficiency rates, by grade and sex ........................................... 15 Table 6. Correlations between context interview composites and FARS results ........................... 23 Table 7. Correlations between teacher absenteeism and FARS results, at midline...................... 25 Table 8. Correlations between distance to Kigali and FARS results ................................................... 26 Table 9. Correlations between distance to District Office and FARS results, at midline ............ 26 Table 10. Mathematics competencies included in MARS ...................................................................... 27 Table 11. Average percent correct on MARS subtests at baseline and midline, by grade ....... 29 Table 12. Average percent correct on MARS assessment, by sex....................................................... 33 Table 13. Correlations between context interview composites and MARS results ...................... 40 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xix Table 14. Correlations between teacher absenteeism and MARS results, at midline ................. 42 Table 15. Correlations between distance to District Office and FARS results, at midline .......... 42 Table 16. Sample of repeating learners tracked from baseline, by sex ............................................ 48 Table 17. School enrolment and dropout statistics, by grade ............................................................. 58 Table 18. Learner/teacher ratio statistics, by grade ................................................................................. 59 Table 19. Average age range of learners, by grade ................................................................................. 63 Table 20. Average number of L3 materials schools received, by grade ........................................... 64 Table 21. Average number of technology materials received from L3 ............................................. 65 Table 22. Correlations between sections of survey results.................................................................... 77 Table 23.Schedule of national assessments using FARS/MARS .......................................................... 82 Table 24. Criteria for Developing Texts in Kinyarwanda ......................................................................... 83 Table 25. MARS Reliability Analysis ................................................................................................................ 84 Table 26. FARS and MARS subtests................................................................................................................ 84 Table 27. Number of sampled learners by district (2 schools per district) ...................................... 87 Table 28. Criteria For FARS Development: Characteristics of texts according to the level of difficulty .................................................................................................................................................................... 89 National Fluency Assessment of Rwandan Schools: Midline Report 2015 xx ACRONYMS WCPM Words Correct per Minute EDC Education Development Center, Inc. EGRA Early Grade Reading Assessment EGMA Early Grade Mathematics Assessment ESSP Education Sector Strategic Plan FARS Oral reading fluency Assessment of Rwandan Schools MARS Mathematics Assessment of Rwandan Schools IIEP International Institute for Educational Planning L3 Literacy, Language and Learning Initiative LARS Learning Achievement in Rwandan Schools MINEDUC Ministry of Education of Rwanda M&E Monitoring and Evaluation REB Rwanda Education Board UNESCO UN Education, Scientific, and Cultural Organization USAID U.S. Agency for International Development National Fluency Assessment of Rwandan Schools: Midline Report 2015 xxi ACKNOWLEDGMENTS The L3 Monitoring and Evaluation Department wishes to acknowledge the important contributions of the numerous people who made this study possible. Many Ministry of Education officials provided key inputs into the development and implementation of the study, including aligning the reading assessment instruments and methodologies to the Rwandan context and participating in the data collection. Last but not least, the study team owes a debt of gratitude to communities, head teachers, teachers, and learners in the sample schools for their cooperation with the data collection. We sincerely appreciate the help of all concerned. 1 INTRODUCTION Rwanda’s Education Sector Strategic Plan 2013 - 2018 is shaped by a number of national aspirations and international goals embodied in policy declarations and plans. These include Rwanda’s Vision 2020, the Economic Development and Poverty Reduction Strategy (EDPRS 2013-2018), the Millennium Development Goals (MDGs) and Education for All (EFA) Goals. Providing quality education to all is a key priority. Therefore, the government has identified the following goals and priorities over the next five years:  Improved quality and learning outcomes across primary and secondary education.  Qualified, suitably skilled and motivated teachers and trainers to meet the demands of expanding education access. 1 The Literacy, Language and Learning Initiative (L3), funded by the U.S. Agency for International Development (USAID) and implemented by a partnership led by the Education Development Center, Inc. (EDC), is designed to help improve learners’ language and mathematics skills. L3 is assisting the Ministry of Education (MINEDUC) in the implementation of a comprehensive early literacy and mathematics program, including support for transition to English as a medium of instruction in Primary 4 (P4). The exhibit on the next pages shows the main objectives of the project. One of L3’s major goals is to improve the quality of teaching reading in Kinyarwanda and in English, and teaching mathematics in Primary 1-4 (P1-P4). To gather information on learner achievement, as well as to support Rwandan Education Board (REB) in establishing a system of regular national assessments, L3 conducts assessments of learner achievement in literacy and mathematics. The assessments are conducted annually during the project rollout stage (2014- 2016) with a random sample of learners drawn from a nationally representative sample of schools. Assessments are conducted in the language of instruction, which is Kinyarwanda in grades P1 through P3, and English in P4. The following tests are included in the assessment:  Oral Reading Fluency Assessment of Rwandan Schools (FARS) includes a grade-level passage and five comprehension questions. This test measures oral reading fluency (speed and accuracy of reading) and comprehension of a grade-level text.  Mathematics Assessment of Rwandan Schools (MARS) includes grade-level problems designed to measure grade-level procedural fluency. 1 MINEDUC, "Education Sector Strategic Plan 2013/14-2017/18," ed. Ministry of Education (Kigali, Rwanda2013). National Fluency Assessment of Rwandan Schools: Midline Report 2015 2 The Literacy, Language and Learning (L3) Initiative’s strategic objective is to strengthen teaching and learning so that children leave primary school with solid literacy and numeracy skills. L3 works with Rwanda’s Ministry of Education (MINEDUC) to improve students’ reading and mathematical skills in grades one to four, as well as their English language proficiency. Working in collaboration with the MINEDUC, USAID and technical partners, the L3 project works with pre￾service and in-service facilitators to introduce proven reading and mathematics teaching strategies, and with community volunteers to support learning. The project also aims to improve the availability and use of innovative reading and math instructional materials. Teachers’ and students’ reading, math and English language skills is reinforced through interactive audio instruction programs. The L3 initiative has five intermediate results that support the strategic objective, and ultimately contribute to USAID’s goal of improved reading skills for 100 million children in primary grades by 2015. These results and key activities include: IR 1: Improved Quality of Teaching 5. Development of a shared vision of effective literacy/numeracy instruction and tools to measure progress with respect to that 6. Implementation of a School-based Mentoring Program to support enhanced literacy, numeracy and ESL instruction 7. Support to TTCs to become Centers of Excellence for Literacy and Numeracy Instruction 8. Pilot initiatives to improve teachers’ motivation and working conditions IR 2: Improved Availability of Teaching and Learning Materials 7. Develop a complete package of instructional materials to support early grade reading 8. Hold Math Camps for teachers and story writing competitions and Writer’s Workshops to produce locally developed reading materials 9. Distribute over one million supplementary books 10. Introduce “traveling libraries” in low income, rural communities 11. Distribute sustainable technologies to support enhanced literacy/numeracy instructional program 12. Hold local campaigns and activities to promote a culture of reading IR 3: Support for English 4. Develop interactive audio programs for, P1 to P4 5. Develop an instrument to evaluate teachers’ English language proficiency 6. Support the revision of the existing English as a Second Language (ESL) curriculum IR 4: Strengthened Ministry Capacity 4. Embed L3 literacy/numeracy specialists in the central MINEDUC and the 13 TTCs to provide day-to-day support in literacy/numeracy and teacher training reforms 5. Develop a criteria-based classroom observation form to monitor changes in teachers’ literacy/numeracy instructional practices over time 6. Provide short-term technical support to the Examinations division to strengthen student literacy/numeracy assessment programs IR 5: Improved Equity in Education 3. Include new instructional materials with positive images of girls and other marginalized groups 4. Provide additional supports and inputs to students in low-income and rural areas Exhibit 1. L3 OVERVIEW National Fluency Assessment of Rwandan Schools: Midline Report 2015 3 The assessments were developed by a team of experts from the REB and L3 and are based on a) international standards for testing and measuring learners’ oral reading fluency in the early grades, and b) on Rwandan2 grade level standards in literacy and the Rwandan Primary Mathematics Curriculum. The assessments were extensively piloted through a number of pilot activities. The first pilot activity took place in March of 2014 with a sample of 1,237 learners randomly selected from 62 schools from all districts in Rwanda. The results were documented in a detailed report3 ; after the initial pilot the assessment team made appropriate adjustments and revised tools which were piloted again in July of 2014. All revisions were made in close collaboration with REB literacy and mathematics experts. The finalized assessment tools were used in the baseline assessment that took place in October of 2014. Assessment items for the midline and endline assessments of P2 and P3 were piloted in October of 2014 with a sample of 664 learners from 12 schools. Assessment items for P4 were piloted in 2015 with a sample of 240 learners randomly drawn from four schools. All assessment items for school-level longitudinal comparisons have been equated using linear equating method. Since 2012, the REB and L3 worked closely to create national reading performance standards for primary grades 3 and 5. A national assessment of P3 and P5 to validate those standards was conducted at the end of the 2012 school year. In 2014, this work continued with proposing reading standards for Primary 2 (P2) and validating them through national sample-based testing. The present report presents learner achievement data collected at the end of 2015 school year utilizing these recently established national reading performance standards. The assessment had the following main objectives: 1. Impact evaluation of L3 project4 : a. Document changes in P1, P2 and P3 learner achievement in oral reading fluency against established benchmarks, and in mathematics on grade-level procedural fluency tasks after one year of national implementation of L3 intervention. b. Collect baseline data in oral reading fluency in Kinyarwanda, English and in procedural fluency in mathematics for P4 learners. 2. Investigation of factors impacting learner achievement: 2 P2 oral reading fluency standards were proposed by L3 to REB during an L3 steering committee meeting in July of 2015 based on the baseline assessment findings. The proposed standards were based on the data from the baseline assessment conducted in October of 2014. The standards were approved during the next steering committee meeting in August 2015. 3 National Fluency and Mathematics Assessment Report, September 2014. Prepared by EDC for USAID. 4 The impact evaluation is designed based on the principles of the impact attribution articulated in USAID Evaluation Policy (2011), and recommendations in the Technical Notes of the Education Strategy (2012, 2015). The counterfactual for the project impact is captured through the baseline conducted with the nationally representative sample of primary schools in October of 2014. National Fluency Assessment of Rwandan Schools: Midline Report 2015 4 a. Analyze variance in learner achievement using school-level data, such as active PTAs, and classroom-level data such as teacher background characteristics, using instructional technology, teaching experience, etc. b. Analyze variance in learner achievement using learner background characteristics, such as parental literacy, support with homework at home, etc. c. Analyze learner achievement among learners who repeated the grade to establish to what extent repeating a grade impacted learner achievement. 3. Development of recommendations for L3 and REB with regard to support systems needed to accelerate improvements in learner achievement. Recommendations will inform L3 activities in the final year of its implementation. In addition to these L3-related objectives, the assessment also provides an opportunity to begin conversations about how nation-wide periodic assessments based on international standards can inform Education Sector Planning. This report presents results of the L3 midline assessment that was conducted in October of 2015 and included the total of 2580 learners, 433 teachers and 60 head teachers from 60 schools. The results of this assessment will be compared with the endline assessment planned for October of 2016. STUDY PARTICIPANTS The assessment collected nationally representative data on oral reading fluency and mathematics achievement among learners in Primary 1, 2, 3 and 4. The detailed sampling parameters are found in the Methodology section in Appendix A; this section presents a description of schools that were randomly selected to take part in the study, and demographic characteristics of learners and teachers who participated in the assessment. This section also presents findings from the context survey. The sampling approach followed random clustered sampling method to obtain a nationally representative sample of public or government-aided schools. The clustered sampling process involved randomly selecting 2 schools from each of the 30 districts in five provinces, with the Study schools on a map, October 2014, 2015 National Fluency Assessment of Rwandan Schools: Midline Report 2015 5 total of 60 schools randomly selected using Complex Samples module of the Statistical Package for the Social Sciences (SPSS). The same schools participated in both baseline and midline assessments. Because there are a different number of districts in each province, the number of schools in a sample is also different in each province. To compensate for the fact that school districts are different in size, the results of the assessment were weighted during the data analysis. Applying weights to the samples ensures that some provinces or school districts are not over or under-represented in the nation-wide estimates. In each visited school, the Head Teacher was asked to complete the School Survey Form to collect contextual information that could help explain variation in learner results across schools. In addition to that, 433 teachers selected from P1, P2, P3 and P4 classrooms completed a Grade Monitoring Form. Table 1 shows the breakdown of teachers by grade and province. A relatively even numbers of teachers from each grade and subject (Kinyarwanda, English and Math) were selected for the sample. The majority (65.4%) of teachers sampled were female, which is a slight decrease from the 70.7% of teachers who were females, sampled at baseline. The proportion of female teachers is highest in lower primary grade. TABLE 1. SAMPLE OF SCHOOLS AND TEACHERS, BY GRADE AND PROVINCE Province Number of schools P1 P2 P3 P4 TOTAL Eastern 12 28 28 28 27 111 Kigali City 6 12 11 12 11 46 Northern 10 16 15 19 15 65 Southern 18 24 32 30 27 113 Western 14 25 25 23 25 98 TOTAL 60 105 111 112 105 433 During the baseline assessment in 2014, the assessment team collected personal information on the tested learners to allow us to track those learners in subsequent years. Consequently, in the October 2015 midline assessment, it was possible to establish how many learners from the original sample progressed to the next grade and how many were retained in the previous year’s grade. The assessment team was able to locate and test 1,130 learners from the baseline sample of 1,799 learners (62.8%). There were no substantial differences between the male and female learners and across grades. Of the tracked learners who were present on the day of midline testing, 175 learners (6.8% of the sample) were found to be repeating the same grade where they were at baseline testing a year previously. Detailed analysis of repeaters versus non-repeaters is found in the dedicated section of the report. National Fluency Assessment of Rwandan Schools: Midline Report 2015 6 Longitudinally tracked learners who were absent on the day of midline testing in P2, P3 and P4 and those who were retained to repeat the grade were replaced with randomly selected learners of the same sex and grade as the missing ones. All P1 learners for the midline assessment were randomly selected. Repeaters from the baseline were not included in the general analysis, although new, randomly selected learners who happened to be repeaters, were included since the probability of their selection was equal to the probability of the selection of any other learner in the grade in the sampled school. Details of the selection methodology are found in the Appendix A. The final midline sample was 2,580 learners, including 175 baseline sample learners who were repeating the grade. The total number of sampled learners during the midline was larger than during the baseline because P4 was added to the assessment. The midline report is based on the analysis of data 2,405 learners and does not include results of 175 learners tracked from the baseline who were repeating the same grade. Instead, the results of baseline repeaters are compared with their performance at the baseline and reported in the section of report dedicated to the analysis of data on repeaters. The sample was constructed to be nationally representative for P1, P2, P3 and P4. While it is stratified by district to ensure adequate representation of learners from all districts of the country, the province-level or district￾level sub-samples are not large enough to be treated as separate samples. These sub￾samples will be only able to detect very substantial changes or differences. Gender Representation. The sample was designed to select an identical number of boys and girls in each grade, in each school. The final distribution by sex was nearly perfect. Since statistics for overall enrolment in primary grades in the sampled schools show gender parity, no gender weights were applied in the statistical analysis of the results. The following chart shows the provincial representation of the sample. District-level post￾design weights were constructed to compensate for the disproportionate representation of learners from some school districts within provinces, to ensure that the sample is nationally representative. Weights were used in all analyses of oral reading fluency and mathematics assessment data to enable extrapolations from the sample onto the population of Rwandan school children in Primary 1, 2, 3 and 4. FIGURE 1. DEMOGRAPHIC CHARACTERISTICS OF SAMPLED LEARNERS (N=2,405) 23.2% 10.0% 16.9% 26.7% 23.2% Eastern Kigali City Northern Southern Western National Fluency Assessment of Rwandan Schools: Midline Report 2015 7 Age of Sampled Learners. Sampled learners at midline ranged in age from 5 to 17 years old. Similar to the baseline, the median age of Primary 1 learners was 7, for Primary 2 learners was 9, for Primary 3 was 10, and the median age for Primary 4 learners was 12. Figure 2 shows the age distribution of the tested learners. These graphs demonstrate the age diversity of primary grade classrooms. FIGURE 2. AGE BY GRADE 1.7% 14.6% 38.0% 23.7% 11.9% 6.6% 3.5% P1 (n=604) 1.0% 7.6% 28.6% 24.8% 18.1% 9.0% 7.3% 3.7% P2 (n=600) 5.1% 18.8% 26.9% 22.0% 16.5% 6.4% 4.1% P3 (n=600) 5.5% 16.0% 24.1% 28.1% 15.3% 8.2% 2.8% P4 (n=601) National Fluency Assessment of Rwandan Schools: Midline Report 2015 8 ORAL READING FLUENCY ASSESSMENT OF RWANDAN SCHOOLS (FARS) ORAL READING FLUENCY An assessment of oral reading fluency was conducted in Kinyarwanda in Primary 1, 2, 3 and 4 using grade level texts of appropriate length and complexity (see Methodology in Appendix A for details). Primary 4 learners were assessed both in Kinyarwanda and in English, using grade￾appropriate texts in both languages. Learners were asked to read the reading passage, followed by five comprehension questions about the text’s meaning. The reading part of the assessment was timed at 60 seconds; the comprehension questions part of the assessment was not timed. Analysis of baseline and midline FARS results showed that learners in all tested grades showed improvement in reading from baseline to midline. Figure 3 shows the average FARS scores (percent correct) by grade for baseline and midline. Primary 4 was assessed for the first time; their scores will serve as a baseline. Primary 1, 2 and 3 showed statistically significant gains (p<.001) in average FARS scores from baseline to midline. In measuring improvements in the number of words read correctly per minute, P1 and P3 learners showed most gains. P1 showed an average increase in FARS of 2.7 wcpm (± 1.1 wcpm). P2 demonstrated gains from baseline to midline with an average increase of 2.3 (±2.0 wcpm) P3 showed average gains of 3.0 wcpm (± 1.8 wcpm). FIGURE 3. AVERAGE NUMBER OF WORDS READ CORRECTLY IN A MINUTE, BY GRADE 4.8 Baseline, 19.2 wcpm Baseline, 22.1 wcpm Kinyarwanda baseline, 40.6 wcpm 7.5 Midline, 21.5 wcpm Midline, 25.1 wcpm English baseline, 26.2 wcpm P1 P2 P3 P4 Baseline Midline * Baseline data for P4 were collected in 2015; endline will be collected in 2016 along with P1-P3 National Fluency Assessment of Rwandan Schools: Midline Report 2015 9 In measuring improvements in a percent of the grade-level text that the learners were able to read in 60 seconds, P1 showed the largest gains from baseline to midline with an average increase of 8.6% (± 3.5%). P2 and P3 also demonstrated significant gains from baseline to midline with an average increase of 7.4% (±4.4%) and 7.2% (±3.1%) respectively. FIGURE 4. AVERAGE PERCENT CORRECT ON FARS, BY GRADE5 The table below shows the changes between the baseline and the midline in percent of the words read correctly, oral reading fluency expressed as words correct per minute, percent of learners with zero scores, and the associate effect size6 . Effect size is a statistical measure that is used to estimate the magnitude of difference between two measures. It is computed by dividing the differences between the means of the two groups by the pooled standard deviation. 5 FARS Percent Correct is calculated by dividing the number of words read correctly in the passage by the total number of words in the passage. Error bars show 95% confidence interval of means. 6 Effect size is a statistical measure that is used to estimate the magnitude of difference between two measures. It is computed by dividing the differences between the means of the two groups by the pooled standard deviation. Lipsey and Wilson (1993), Vernez and Zimmer (2007), and Hill, Bloom, Black and Lipsey (2007) suggest the following interpretation of effect sizes in education: 0.25 or more as large, 0.15 as medium, and 0.05 to 0.10 as small. Baseline, 17.2% Baseline, 43.1% Baseline, 37.5% Kinyarwanda Baseline, 60.5% Midline, 25.9% Midline, 50.5% Midline, 44.7% English Baseline, 41.9% P1 P2 P3 P4 Baseline Midline * Baseline data for P4 were collected in 2015; endline will be collected in 2016 along with P1-P3 National Fluency Assessment of Rwandan Schools: Midline Report 2015 10 TABLE 2. BASELINE-MIDLINE GAINS ON FARS, BY GRADE Grade Subtest BASELINE MIDLINE GAIN EFFECT SIZE (Cohen’s d) P1 Percent 17.2% 25.9% 8.6 (± 3.48) 0.28 (± .11) Average wcpm 4.76 7.48 2.7 (±1.06) 0.29 (± .12) Zero Score 60.3% 50.4% - 10.1 (±0.06 ) 0.20 (± .11) P2 Percent 43.1% 50.5% 7.4 (± 4.41) 0.19 (± .11) Average wcpm 19.2 21.5 2.3 (± 2.00) 0.13(± .11) Zero Score 32.7% 25.5% - 7.2 (± 0.05) 0.16(± .11) P3 Percent 37.5% 44.7% 7.2% (± 3.06) 0.26(± .11) Average wcpm 22.1 25.1 3.0 (±1.79) 0.19(± .11) Zero Score 21.3% 18.6% - 2.7 (±0.05) 0.07(± .11) One of the objectives of the assessment was to gather data on the percent of learners at different reading proficiency standards at the end of each grade. L3 and REB reading specialists proposed proficiency standards that are based on extensive research in literacy and a data-supported relationship between oral reading fluency and comprehension. According to the fluency standards7 , the minimal oral reading fluency rate of a learner at the end of P2 should be at least 20 words read correctly in one minute. The minimal oral reading fluency rate of a learner at the end of P3 should be at least 33 words read correctly in one minute. While there are no fluency standards for P1, we use non-zero scores as a measure of fluency. Non-zero scores at the end of P1 could be construed as a positive result. Since P4 learners are expected to read in both Kinyarwanda and English, proficiency rates for both languages need to be established. Table 3 shows oral reading proficiency standards used to compute proficiency rates of primary grade learners in Rwandan schools. 7 P2 oral reading fluency standards were proposed by L3 to REB during an L3 steering committee meeting in July of 2015, based on the baseline assessment findings. The proposed standards were based on the data from the baseline assessment conducted in October of 2014. The standards were approved during the next steering committee meeting in August, 2015. Oral reading fluency proficiency Speed in wcpm Proficiency standard Beginning to develop Under 20 wcpm Developing reader 20-32 wcpm Primary 2 Emerging fluent reader 33-47 wcpm Primary 3 Fluent reader Over 48 wcpm TABLE 3. ORAL READING PROFICIENCY THRESHOLDS National Fluency Assessment of Rwandan Schools: Midline Report 2015 11 Overall, the assessment found that more P2 and P3 learners could read fluently compared to their peers from the same grades who were tested last year, according to the proficiency standards established by REB. The percent of P1 learners who could read 20 words correct per minute or faster more than doubled between the baseline and the midline. The percent of P2 learners who could read a grade level text with some oral reading fluency (over 20 words per minute) increased from 50% to 56%. The percent of P1 learners with non-zero scores increased by 10%. All changes are statistically significant at p<.001 level. The following figures show a distribution of results for oral reading fluency according to the proficiency standards for P2 and P3. Since no proficiency standards for P1 have been established at the time of the preparation of this report, all non-zero score learners were assumed to be proficient for the purpose of reporting here. The percent of P3 learners who are proficient is smaller than the percent of P2 learners who are proficient because proficiency standards for P3 are higher. The results are weighted to provide an estimate of the population of Rwanda primary grade learners. FIGURE 5. PERCENT OF LEARNERS READING AT GRADE LEVEL, BY GRADE zero 60% zero 50% 1 to 19 34% 1 to 19 37% 20 to 32 5% 20 to 32 11% 33 to 47 1% 33 to 47 2% over 47 0% over 47 0% zero scores developing and emerging readers P1 baseline P1 midline Average decrease in percent of children not able to read a single word - P1: 10.1% - P2: 7.2% - P3: 2.7% National Fluency Assessment of Rwandan Schools: Midline Report 2015 12 The results of the assessment found that learners in P1 and P2 showed statistically significant reduction in the proportion of learners with zero scores in reading. P1 learners demonstrated the most dramatic drop of 10% (± 0.1%), and P2 learners showed a drop of 8% (± .1%). The change in the proportion of P3 learners with zero scores between the baseline and the midline was not statistically significant and can be attributed to sampling variation. Sampling variation is the variation that occurs between samples of the one population. A measure of the random error of the sampling technique was used. The table below summarizes the data in figure 5. It shows the gains in oral reading fluency proficiency rates between the baseline and midline. TABLE 4. BASELINE-MIDLINE GAINS IN PROFICIENCY RATES, BY GRADE Grade BASELINE MIDLINE GAIN EFFECT SIZE (Cohen’s d) P1* 39.7% 49.6% 9.9% (± .056) 0.20 (± .11) P2** 49.9% 56.2% 6.3% (± .056) 0.13 (± .11) P3*** 24.6% 31.8% 7.2% (± .051) 0.16(± .11) * Reading 1+ words correct per minute zero 33% zero 26% 1 to 19 17% 1 to 19 18% 20 to 32 25% 20 to 32 27% 33 to 47 20% 33 to 47 22% over 47 5% over 47 7% zero scores and developing readers proficient and advanced readers P2 baseline P2 midline zero 21% zero 19% 1 to 19 17% 1 to 19 11% 20 to 32 37% 20 to 32 39% 33 to 47 18% 33 to 47 26% over 47 6% over 47 5% below proficiency level proficient and advanced readers P3 baseline P3 midline National Fluency Assessment of Rwandan Schools: Midline Report 2015 13 ** Reading 20+ words correct per minute *** Reading 33+ words correct per minute An examination of the percentage of learners who were able to read within an allocated minute showed a movement toward better results at midline, although the U-patterned or heavily skewed distributions persist in both P1 and P2, with a high proportion of learners either reading the entire text or not reading a single word. P1 learner results still had a skewed distribution toward zero scores at the midline. P3 assessment results showed about 20% of learners with zero scores, with the remaining results normally distributed. Figure 6 presents these results, with a polynomial line emphasizing the shape of the distribution in each grade. FIGURE 6. PERCENT OF WORDS READ CORRECTLY, GROUPED BY GRADE An analysis of proficiency rates by sex in P2 and P3 showed that girls were more likely to complete the reading of the passage than boys. At baseline, the difference was statistically 0% 20% 40% 60% zero 1-20% 21-40% 41-60% 61-80% 81-100% P1 midline baseline 0% 10% 20% 30% 40% zero 1-20% 21-40% 41-60% 61-80% 81-100% P2 0% 10% 20% 30% 40% zero 1-20% 21-40% 41-60% 61-80% 81-100% P3 0% 10% 20% 30% 40% zero 1-20% 21-40% 41-60% 61-80% 81-100% P4 (baseline only) English Kinywarwanda At midline, girls were found to be reading faster than boys in all four tested grades. National Fluency Assessment of Rwandan Schools: Midline Report 2015 14 significant among learners, at p<.001. At midline, this difference is statistically significant in oral reading in Kinyarwanda in all four tested grades. Figure 7 shows how much of the grade￾level oral reading passage P2 and P3 boys and girls were able to read within the allocated one minute. (P1 results are not presented since only a small proportion of tested learners were able to read the test passage.) The figure shows that girls’ results are skewed toward higher results, comparing to boys. Further investigation is needed to establish the reasons why girls are learning to read better than boys. FIGURE 7. PERCENT OF WORDS READ CORRECTLY GROUPED, BY SEX AND GRADE In fact, an analysis of learner proficiency results by sex showed that girls both started significantly higher and improved more than boys between the baseline and the midline. baseline midline -5% 5% 15% 25% 35% 45% zero less than 20 wcpm 20 to 32 wcpm 33 to 47 wcpm over 47 wcpm percent of learners P2 girls midline baseline 0% 10% 20% 30% 40% zero less than 20 wcpm 20 to 32 wcpm 33 to 47 wcpm over 47 wcpm percent of learners P2 boys midline baseline baseline midline 0% 10% 20% 30% 40% zero less than 20 wcpm 20 to 32 wcpm 33 to 47 wcpm over 47 wcpm percent of learners P3 girls 0% 10% 20% 30% 40% zero less than 20 wcpm 20 to 32 wcpm 33 to 47 wcpm over 47 wcpm percent of learners P3 boys National Fluency Assessment of Rwandan Schools: Midline Report 2015 15 TABLE 5. BASELINE-MIDLINE GAINS IN PROFICIENCY RATES, BY GRADE AND SEX Girls: Proficiency Averages Grade BASELINE MIDLINE GAIN EFFECT SIZE (Cohen’s d) P1* 40.0% 54.8% 14.7% (± .080) 0.30 (± .16) P2** 57.1% 64.2% 7.1% (± .078) 0.15 (± .16) P3*** 26.6% 41.2% 14.6% (± .075) 0.31(± .16) Boys: Proficiency Averages Grade BASELINE MIDLINE GAIN EFFECT SIZE (Cohen’s d) P1 39.5% 44.7% 5.3% (± .078) 0.11 (± .16) P2 42.5% 48.2% 5.7% (± .080) 0.11 (± .16) P3 22.6% 22.4% 0.2% (± .067) -0.01(± .16) * Reading 1+ words correct per minute ** Reading 20+ words correct per minute *** Reading 33+ words correct per minute COMPREHENSION During the assessment, sampled learners were asked five locator8 questions about the text that they just read (see Methodology section in Appendix A for the description). The overall results are presented below in Figures 8 and 9. While a statistical correction was performed to equate9 the results of two different texts used for testing P2 and P3 learners at baseline and midline, it was not possible to equate comprehension questions. The P2 comprehension results, in particular, suffered as a result of uneven comparisons. The baseline P2 text was simpler and the corresponding comprehension questions were easier, as well. The pilot exercise, while extensive and rigorous, was not sufficient in scale to identify how substantial the differences were. Other measures were equated so that the final comparison of the number of words read correctly or the percent of the text read correctly would indeed be equivalent and not biased toward one text or the other. Because of the small number of comprehension questions, it is impossible to statistically correct for difference in questions between the baseline and the midline. Therefore, it is 8 A locator (also called “literal”) question is a type of comprehension question about the passage that invokes a specific reference to the text and not implied meaning or an inference. For example, a question about a name of a character or a place in a story that is specifically mentioned is a locator question. 9 Linear equating was used to render the results of two different tests equivalent. The equating testing was performed in 2014 and involved 376 P2 learners and 288 P3 learners who were asked to read both texts and answer comprehension questions. National Fluency Assessment of Rwandan Schools: Midline Report 2015 16 important to recognize the difference in the difficulty of the text and complexity of the questions when comparing the comprehension subtest results. A learner who “reads with comprehension” is supposed to be able to answer at least 4 out of 5 (80 to 100%) comprehension questions. Figures below show that a larger proportion of tested learners were not able to achieve this benchmark. FIGURE 8. FARS COMPREHENSION RESULTS, BY GRADE zero 69% zero 64% 1 12% 1 11% 2 9% 2 7% 3 4% 3 5% 4 4% 4 5% 5 2% 5 8% zero to 3 questions 4-5 questions answered P1 baseline P1 midline zero 38% zero 41% 1 3% 1 14% 2 8% 2 19% 3 17% 3 15% 4 19% 4 6% 5 15% 5 6% zero to 3 questions 4-5 questions answered P2 baseline P2 midline zero 26% zero 21% 1 16% 1 9% 2 27% 2 21% 3 24% 3 32% 4 4% 4 12% 5 2% 5 5% zero to 3 questions 4-5 questions answered P3 baseline P3 midline National Fluency Assessment of Rwandan Schools: Midline Report 2015 17 Since a true measure of comprehension can only be taken when a learner can read the text about which the questions are asked, an analysis of comprehension results among learners who read more than 80 percent of the text was conducted. The results showed that 13.8 percent of P2 and 55.2% of P3 learners who actually read the text were able to answer four or five literal comprehension questions. All P3 learners who were able to read 80% of the text or more were able to answer at least one comprehension question. P2 and P3 learners who answered 4 or 5 comprehension questions read the text with the average speed of 37.5 and 49.0 wcpm, respectively; 12.6% of P3 learners read with the grade￾level speed of 33 wcpm or faster and answered 4 or 5 comprehension questions, compared to 5% at baseline. Only a handful of P1 learners were able to read over 80% of P1 text correctly, and those learners answered all or almost all comprehension questions correctly. zero 11.3% zero 58.3% 1, 13.6% 1, 19.4% 2, 14.5% 2, 7.0% 3, 24.8% 3, 4.6% 4, 13.8% 4 questions, 3.2% 5 questions 22.1% 5 questions 7.5% zero to 3 questions answered 4 or 5 questions answered P4 Kinyarwanda P4 English P2 and P3 learners who answered 4 or 5 comprehension questions read the text with the average speed of 37.5 and 49.0 wcpm, respectively; 12.6% of P3 learners read with the grade-level speed of 33 wcpm or faster and answered 4 or 5 comprehension questions, compared to 5% at baseline. National Fluency Assessment of Rwandan Schools: Midline Report 2015 18 FIGURE 9. COMPREHENSION RESULTS AMONG LEARNERS WHO READ 80-100% OF THE TEXT Comparisons by sex did not show significant differences beyond the difference in the comprehension of reading. Both boys and girls, who managed to read the test passage, were also able to answer comprehension questions, indicating appropriate vocabulary knowledge for their grade level. Since oral reading fluency is a statistically much more reliable measure than comprehension (due to the number of items included in the measurement), the results of the comprehension subtest should be interpreted with more caution than results of the fluency test. UNTIMED READING RESULTS In response to REB’s queries on timed and untimed testing and growing interest in this area internationally, during the midline testing the data collectors administered the reading comprehension questions in two rounds: one with a timed and one with untimed reading. The first round the administration followed the standard EGRA administration procedures where tested learners had access to the test for 60 seconds and were supposed to answer comprehension questions without referencing the text. The second round immediately followed the first round. During the second round, assessors gave the text back to the learners and allowed them to finish reading the passage (if they hadn’t done so already), and then asked them comprehension questions without taking the text away from the learners. The figure below compares the two rounds: zero 6.9% 1 19.5% 2 35.4% 2 8.2% 3 24.4% 3 36.7% 4 8.9% 4 26.2% 5 4.9% 5 29.0% zero to 3 comprehension questions answered 4 or 5 comprehension questions answered P2 P3 National Fluency Assessment of Rwandan Schools: Midline Report 2015 19 FIGURE 10. MEASURING READING COMPREHENSION The results of the comparison between the two models of testing learners’ comprehension skills are presented in the series of charts below. As seen from the next figure, the average results improved by more than 50% in each grade. The red line shows the confidence interval of the difference in means between the timed and the untimed reading comprehension results. FIGURE 11. COMPARISON OF TIMED AND UNTIMED READING COMPREHENSION RESULTS, BY GRADE The following figures show the change in the number of comprehension questions answered when learners are allowed more time to finish reading the text and have access to the text when answering the questions. In all grades, the percent of learners who were able to meet comprehension proficiency benchmark of 80% increased dramatically. In P2, allowing learners to complete reading and refer to the text when answering comprehension questions caused a drop in comprehension zero scores from 41% to 31%. Note: P3 has 6 comprehension questions while other grades had 5 questions. timed, 1.01 timed, 1.48 timed, 2.21 timed, 2.83 timed, 0.98 untimed, 1.61 (out of 5) untimed, 2.14 (out of 5) untimed, 3.58 (out of 6) untimed, 4.23 (out of 5) untimed, 1.4 (out of 5) P1 P2 P3* P4 KR P4 English * P3 test had 6 comprehension questions. All other grades had 5 comprehension questions. National Fluency Assessment of Rwandan Schools: Midline Report 2015 20 FIGURE 12. COMPARISON OF TIMED AND UNTIMED READING COMPREHENSION RESULTS 64% 11% 7% 5% 5% 8% zero one two three four five P1 comprehension results after timed reading 58% 6% 5% 4% 5% 23% zero one two three four five P1 comprehension results after untimed reading 41% 14% 19% 15% 6% 6% zero one two three four five P2 comprehension results after timed reading 31% 11% 13% 18% 13% 14% zero one two three four five P2 omprehension results after untimed reading 21% 9% 21% 32% 12% 3% 2% zero one two three four five six P3 comprehension results after timed reading 20% 2% 5% 11% 19% 24% 20% zero one two three four five six P3 comprehension results after untimed reading 11% 14% 14% 25% 14% 22% zero one two three four five P4 Kinyarwanda comprehension results after timed reading 9% 2% 3% 4% 10% 72% zero one two three four five P4 Kinyarwanda comprehension results after untimed reading National Fluency Assessment of Rwandan Schools: Midline Report 2015 21 These findings suggest that reading comprehension results from timed reading assessments which have become USAID’s standard in the recent years must be interpreted with caution since scores might be as much impacted by the process of administration as by the abilities of the tested learners. To compute the percent of learners who both can read with fluency and can answer comprehension questions, we combined results of the fluency assessment and the comprehension subtest. The graph below shows the results for both timed and untimed comprehension administration. As the graph demonstrates, the percent of learners who meet the USAID standard indicator “percent of learners who, by the end of two years of schooling, can read and understand grade-level text”, which uses a timed test, is 10.5% at the end of the first year of L3 nationwide intervention, if measuring with timed reading and comprehension without referencing the text. The percent increased to 24% if measuring comprehension with the untimed reading. FIGURE 13. PERCENT OF P2 LEARNERS READING AND UNDERSTANDING GRADE LEVEL TEXT These findings demonstrate that removing time and memory barriers notably increases comprehension scores among all groups. 58% 19% 7% 5% 3% 8% zero one two three four five P4 English comprehension results after timed reading 50% 14% 11% 9% 4% 12% zero one two three four five P4 English comprehension results after untimed reading 10.5% 24.1% Proficiency rate with timed reading Proficiency rate with untimed reading 10.5% of P2 learners read 20+ wcpm AND answer 4+ questions 24.1% of P2 learners read 20+ wcpm AND answer 4+ questions National Fluency Assessment of Rwandan Schools: Midline Report 2015 22 IMPACT OF CONTEXTUAL FACTORS ON READING LEARNER CHARACTERISTICS AND FARS RESULTS Both at baseline and at midline learners’ age was found to negatively correlate with the reading achievement. The older the tested learner was, the lower his/her reading results would be. This relationship between learner’s age and his/her reading results is strong across all grades. Many children, particularly in rural areas, do not start school on time, or do not attend regularly and are more likely to repeat grades. As The Education Sector Strategic Plan10 emphasizes, delaying starting school past the correct age has implications for both the learner and the school. Older learners are less likely to succeed academically, and grade repetition puts a strain on school system resources. One of the reasons for why some children might not be starting school at the correct age is that they are needed at home to help out. Some parents may not see the value of education in their specific circumstances or overall. Research in sub-Saharan Africa shows that for many families in rural areas sending children to school has too high an opportunity cost11. While parents may recognize the value of education in general, they are aware that learners must at the very least complete secondary school to qualify for a position at the government office, and many families feel they cannot afford to do without their children’s contribution to family livelihoods for so many years. Since primary education is compulsory in Rwanda, local education authorities are pressuring those families to still send their children to school, but those children may have irregular attendance, a much higher risk of grade repetition12 and lack of support for education at home. Bivariate statistical analysis found small, but statistically significant correlations between learner results in oral reading and on comprehension tests, and context interview composites13 , as shown in the table below. These results are consistent with the results of the baseline which found similar correlations between home environment, school and teacher, and socio￾economic status composite variables.14 10 Education Sector Strategic Plan 2010-2015, Rwanda Ministry of Education, July 2010. 11 Dennis, C., Stahley, K. “Universal Primary Education in Tanzania: The Role of School Expenses and Opportunity Cost”. Evans School Review. Vol. 2, Num. 1., Spring 2012. Hillman, A., Jenkner, E. “Educating Children in Poor Countries”. International Monetary Fund. 2004. Girls’ Education in Africa. Education and Gender Equality Series, Programme Insights. December 2005. Olaniyan, O. “The Determinants of Child Schooling in Nigeria”. AERC Research Paper 217. African Economic Research Consortium, Nairobi, 2011. 12 The official rate of grade repetition in Rwanda was reported to be 12.5% in 2012 (2013 Rwanda Education Statistical Yearbook, September 2014) 13 Details of the learner context interview and resulting composites are found in the last section of the report. 14 Correlations between oral reading fluency and home environment at baseline were found to be .266 and .121 for P2 and P3, respectively. Correlations between oral reading fluency and school and teacher composite at baseline National Fluency Assessment of Rwandan Schools: Midline Report 2015 23 The largest correlations were found in P4 between oral reading fluency results in English and socio-economic status composite15 . Having something to drink before coming to school and family owning a means of transportation were the two variables out of the socio-economic status composite that had the largest correlations with the oral reading results. Being able to choose stories to read and taking books home were two variables with the most impact on reading out of school and teacher composite. The table below summarizes the findings of the correlation analysis between questions on the context interview and the results of the FARS. The negative correlation coefficients indicate that with increase in variable the results of the test decrease. For example, negative correlation between age and FARS tells us that with the increase in age the results in FARS decrease, on average, among P2, P3 and P4 learners. The positive correlation between age and FARS for P1 learners tells us that with the increase in age among P1 learners, their results go up. TABLE 6. CORRELATIONS BETWEEN CONTEXT INTERVIEW COMPOSITES AND FARS RESULTS P1 P2 P3 P4 KR P4 English fluency comp. fluency comp. fluency comp. fluency comp. fluency comp. Are you repeating this grade? (yes=1) .095* .110** -.137** -.121** -.190** -.151** -.095* Learner's age .112** .119** -.132** -.239** -.196** -.184** -.178** -.224** -.167** Mother literate (1=yes) .163** .244** .092* .093* .112* At home, does someone read stories to you? 1=yes .208** .190** .089* .144** .171** .184** .158** .159** How often do you miss school? (4=a lot, 3= sometimes, 2=rarely, 1=never) -.116** -.096* -.125** -.119** -.128** -.130** -.164** How often are you late for school? (4=a lot, 3= sometimes, 2=rarely, 1=never) .086* .135** .115** .089* .100* Have you or any of your siblings ever repeated a grade? (1=yes) -.114** -.178** -.116** -.083* -.168** Do your parents/caregivers check your homework? (1=yes) .140** .115** .102* .101* .143** .129** .142** .119** When not understanding a lesson, do you ask questions? (1=yes) .123** .117** .101* At school, can you choose which stories to read? (1=yes) .171** .146** .210** .189** .223** .101* Are you allowed to take books home from school? (1=yes) .132** .089* -.094* Do you ever take books from school to read at home? (1=yes) .084* .156** .130** -.088* -.173** were found to be .144 and .106 for P2 and P3. Correlations between oral reading fluency and socio-economic status at baseline were found to be .135 and .139 for P2 and P3, respectively. All correlations were statistically significant at p<.01 level. 15 In social science research correlations below .2 are not considered to be of high importance. Correlations between .2 and .4 are considered small, correlations between .4 and .6 are moderate, and above .6 they are large. National Fluency Assessment of Rwandan Schools: Midline Report 2015 24 Did you have something to drink today (like water, tea, milk or juice)? (1=yes) .153** .110** .120** .114** .169** .109** .116** .150** Did you have something to eat today, like potatoes, rice, bread or beans? (1=yes) Do you have radio or cell phone at home? (1=yes) .113** Does anyone at your house have a bicycle/motorcycle or a car? (1=yes) .094* .132** .115** .270** .253** *Correlations are significant at the >0.05 level (2-tailed) **Correlations are significant at the >0.01 level (2-tailed) Blanks denote no statistically significant association between variables. Weak positive association (.1 to .199) Stronger positive association (.2 and higher) Weak negative association (-.1 to -.199) Stronger negative association (-.2 and higher) Nearly all learners said yes to the following questions: - Do your parents/caregivers want you to go to school every day? - Does your mathematics teacher check your work that you do in class? - Does your mathematics teacher check/mark your homework? - Does your Kinyarwanda teacher check your work that you do in class? - Does your Kinyarwanda teacher check/mark your homework? Additionally, nearly all learners said they speak Kinyarwanda at home. Since these six questions did not have variance in learner responses, they were excluded from correlational analysis. Significant differences were found between learners who had zero scores in reading and those who had non-zero scores. Learners with zero scores had lower composite values on all three composites except the risk factors composite. The difference was statistically significant at p<.05 level. Multivariate linear regression analysis16 showed that the context interview composites do not explain a significant amount of variance in learner achievement in any of the tests. Results of linear regression analysis showed that school/teacher composite explains two to three percent of variance in oral reading fluency results; all three composites combined explain about five percent. The most significant variable found to explain the variance in P4 English oral reading fluency was owning a means of transport, such as a bike or a motorcycle. Learners who said someone in their family owned a means of transport read an English passage 14 words per 16 Linear regression is a statistical analysis procedure that allows computing how much of the change (“variance”) in the variable of interest is explained by the change in other variable(s). National Fluency Assessment of Rwandan Schools: Midline Report 2015 25 minute faster than those learners who said no one in their family owned a means of transport. This relationship was not found in oral reading in Kinyarwanda in any of the tested grades. Comparisons of four composites by sex found no difference between boys and girls. The analysis of midline test results showed that girls did better than boys in oral reading fluency. The difference was statistically significant in all grades in oral reading fluency at p<.01 level. SCHOOL CHARACTERISTICS To help us better understand the variation in learner FARS scores, we looked at the differences in school characteristics, such as distance to Kigali and District Office, teacher absenteeism, and other factors. A variety of factors were found to be associated with learner performance on FARS. Contrary to expectations, such factors as overcrowding (teacher to learner ratio), availability of a school or a community library, school size, and the nursery attached to school were not found to be associated with learner results. However, teacher absenteeism was found to be negatively associated with learner performance on FARS. Table below shows that the higher the proportion of teachers absent, the lower the learner results are. TABLE 7. CORRELATIONS BETWEEN TEACHER ABSENTEEISM AND FARS RESULTS, AT MIDLINE P1 P2 P3 P4 KR P4 English fluency comp. fluency comp. fluency comp. fluency comp. fluency comp. Teacher absenteeism17 -.096* -.100* -.168** -.165** -.137** -.152** -.123** -.158** -.191** -.085** * Correlations are significant at the >0.05 level (2-tailed) **Correlations are significant at the >0.01 level (2-tailed) Shaded correlations are those with r > .1 At baseline, distance between Kigali and the school was found to be negatively correlated with learner achievement in reading, with learner doing better in schools closer to Kigali. However, at midline, these correlations were much reduced or removed altogether. This shows that the program has been effective in providing much-needed support to those teachers and schools that needed it the most. 17 Teacher absenteeism was computed as the percent of teachers absent on the day of the assessment and on the day previous to that day, averaged. National Fluency Assessment of Rwandan Schools: Midline Report 2015 26 TABLE 8. CORRELATIONS BETWEEN DISTANCE TO KIGALI AND FARS RESULTS P1 P2 P3 P4 KR P4 English fluency comp. fluency comp. fluency comp. fluency comp. fluency comp. B Distance to Kigali -.098* N/S -.158** -.102* -.155** -.156** Not tested M Distance to Kigali N/S N/S N/S -.096* -.097** -.125** N/S N/S N/S N/S B = baseline; M = midline; N/S = not significant * Correlations are significant at the >0.05 level (2-tailed) **Correlations are significant at the >0.01 level (2-tailed) Shaded correlations are those with r > .1 In addition to exploring how distance to Kigali might affect school performance, at midline, the school form included a question about distance to the local District Office. The reported range was between zero and 96 kilometers, with an average of 22 kilometers. Distance to the District office was found to be an important predictor of how well learners performed on the oral reading fluency test, with school farther from the District Office having learners with lower FARS scores. All correlations were statistically significant at p < .001 level. TABLE 9. CORRELATIONS BETWEEN DISTANCE TO DISTRICT OFFICE AND FARS RESULTS, AT MIDLINE P1 P2 P3 P4 KR P4 English fluency comp. fluency comp. fluency comp. fluency comp. fluency comp. Distance to District Office -.125** -.136** -.137** -.130** -.181** -.150** -.106** -.142** -.157** -.115** **Correlations are significant at the >0.001 level (2-tailed) Shaded correlations are those with r > .1 Distance to the District Office was also found to be very highly positively correlated with teacher absenteeism (r = .498, significant at p < .001). It is possible that more remotely located schools are challenging to access for both learners and teachers. In fact, data analysis shows statistically significant relationship between teacher absenteeism and learner absenteeism, both also correlated highly with the distance to the District Office. SUMMARY Overall, oral reading fluency assessment results show that the proportion of learners reading on grade level increased very substantially over the course of the first year of L3 implementation nation-wide. Most of those learners who were able to read the text were also able to answer some or all comprehension questions. Since literacy instruction is conducted in the mother tongue of the vast majority of learners, it is probable that the major obstacle to reading is decoding. Comparisons in comprehension rates between timed and untimed reading show a significant improvement in comprehension rates when learners are allowed to finish reading the text and refer to the text when answering comprehension questions. National Fluency Assessment of Rwandan Schools: Midline Report 2015 27 MATHEMATICS ASSESSMENT OF RWANDAN SCHOOLS (MARS) SUMMARY FINDINGS Mathematics assessment was developed by EDC mathematics experts based on the review of the Rwandan mathematics curriculum in early grades and on the international standards of mathematics instruction. All tasks are supposed to test grade-level procedural fluency in basic mathematical concepts. The mathematics test included three subtests with 10 items each for Primary 1 and 2, four subtests for P3 and five subtests for P4. The tasks were developed to reflect grade-level competencies (see Methodology section in Appendix A); hence tasks for each grade were more difficult than tasks for the previous grade. Two of the types of tasks were the same for all grades (addition and subtraction); however they varied in difficulty by grade. Similarly, additional subtests of increased difficulty were added to each grade, including division for Primary 3 and division and comparing numbers for Primary 4. All tasks were timed at 120 seconds for P1, P2 and P3, and 60 seconds for P4. The table below shows the tasks included in the test by grade. TABLE 10. MATHEMATICS COMPETENCIES INCLUDED IN MARS The same tests were used for P1, P2 and P3 at both baseline and midline, so no equating of the results was necessary. Analysis of baseline and midline MARS results found that learners in P1, P2, and P3 showed an improvement in mathematics after one year of L3 implementation. Figure 14 shows the GRADE LEVEL SUBTEST TASK Primary 1 Subtest 1 Adding numbers within 10 Subtest 2 Subtracting numbers within 10 Subtest 3 Comparing magnitude of numbers (up to 2 digits) Primary 2 Subtest 1 Adding numbers within 100 Subtest 2 Subtracting numbers within 20 Subtest 3 Multiplying numbers within 10 Primary 3 Subtest 1 Multiplying numbers up to 10 Subtest 2 Dividing numbers within 10 Subtest 3 Adding numbers within 100 Subtest 4 Subtracting numbers within 100 Primary 4 Subtest 1 Adding numbers within 200 Subtest 2 Subtracting numbers within 100 Subtest 3 Multiplying number within 20 Subtest 4 Dividing numbers within 200 Subtest 5 Comparing magnitude of numbers (fractions, decimals and 2 digit numbers) National Fluency Assessment of Rwandan Schools: Midline Report 2015 28 average percent of MARS tasks solved correctly at baseline and midline, by grade. All grades showed statistically significant gains (p<.001). P1 showed the largest gains from baseline to midline with an average increase of 22.2% (± 3.1%) in the percent of MARS tasks solved correctly. P2 and P3 also demonstrated significant gains from baseline to midline with an average increase of 8.4% (±2.9%) and 7.4% (± 2.7%), respectively. FIGURE 14. AVERAGE PERCENT CORRECT ON MARS SUBTESTS, BY GRADE18 Further analysis of performance of learners by MARS subtests showed the results varied by subtest and by grade. Analysis of average learner achievement on the different MARS subtests at midline showed interesting results. P2 learners did better in subtracting numbers than adding numbers, while P1 and P3 learners did better in adding numbers than subtracting numbers. P1 learners performed nearly as well as P2 learners in Addition. Overall, analysis of midline results showed that P2 and P3 learners demonstrated the most knowledge and skills in the most elementary and procedural of MARS subtests—addition and subtraction. By contrast, the subtests in which learners particularly struggled were ones focused on more advanced mathematical operations—multiplication and division. The same assessment instrument for each grade was used in baseline and midline assessment although, as described above, the 18 Average MARS Percent Correct is calculated by averaging the percent correct for each MARS subtest. Error bars show 95% confidence interval of means. 25.7% 33.9% 39.0% 51.9% 47.9% 42.2% 46.4% P1 P2 P3 P4* Baseline Midline * Baseline data for P4 was collected in 2015; midline data not available National Fluency Assessment of Rwandan Schools: Midline Report 2015 29 assessment instruments varied from grade to grade, ensuring that they had tasks of appropriate difficulty for each grade. Complete results are found in Appendix C. Detailed analysis of improvement from baseline to midline by subtest showed significant gains (p<.01) on all MARS subtests for all grades (P1, P2 and P3). The largest gains were seen in P1 with average gains of more than 20% on all subtests. Effect sizes were also calculated to analyze improvement on the MARS assessment between baseline and midline. Overall, analysis showed a very large effect size (d=0.81) for P1 learners, which suggests that, at midline, 79% of P1 learners scored higher on the MARS assessment than P1 baseline learners. Analysis of gains for P2 and P3 showed effect sizes of 0.33 and 0.31 respectively, meaning that, at midline, 62% of P2 and P3 learners scored higher on the MARS than at baseline. The table below shows the average MARS subtest gains and effect size values for each MARS subtest by grade. For P1, the subtraction subtest showed the largest gains from baseline to midline with an effect size of .81; the other two subtests also showed large effect sizes. P2 and P3 showed smaller effect sizes on MARS subtests compared to P3. For both P2 and P3, the largest gains were found in the Addition subtests with effect sizes of 0.37. TABLE 11. AVERAGE PERCENT CORRECT ON MARS SUBTESTS AT BASELINE AND MIDLINE, BY GRADE Grade Subtest BASELINE (pct)19 MIDLINE (pct) GAIN (pct) EFFECT SIZE (Cohen’s d) P1 Addition 22.4% (± 2.2%) 43.6% (± 2.7%) 21.2% (± 3.5%) 0.69 (± .12%) Subtraction 15.1% (± 1.9%) 39.2% (± 2.8%) 24.1% (± 3.4%) 0.81 (± .12%) Comparing Numbers 39.6% (± 2.6%) 60.8% (± 2.6%) 21.2% (± 3.6%) 0.66 (± .12%) P2 Addition 31.5% (± 2.4%) 43.7% (± 2.9%) 12.2% (± 3.7%) 0.37 (± .11%) Subtraction 45.3% (± 2.7%) 50.9% (± 2.7%) 5.5% (± 3.9%) 0.16(± .11%) Multiplication 24.8% (± 1.6%) 32.1% (± 1.9%) 7.3% (± 2.5%) 0.34(± .11%) P3 Multiplication 45.5% (± 2.4%) 52.2% (± 2.4%) 6.7% (± 3.4%) 0.22(± .11%) Division 26.8% (± 2.3%) 32.2% (± 2.5%) 5.3% (± 3.3%) 0.18(± .11%) Addition 45.5% (± 2.1%) 55.5% (± 2.3%) 10.0% (± 3.1%) 0.37(± .11%) Subtraction 38.3% (± 2.2%) 45.9% (± 2.2%) 7.6% (± 3.1%) 0.27(± .11%) P420 Addition 77.0% (± 2.0%) -- -- -- Subtraction 62.9% (± 2.4%) -- -- -- Multiplication 51.8% (± 2.2%) -- -- -- Division 33.1% (± 2.4%) -- -- -- Comparing Numbers 34.7% (± 1.9%) -- -- -- 19 Table shows the average percent correct by MARS subtest by grade. A 95% confidence interval is also reported, which indicates that the point estimate of average percent correct has a margin of error of 5.0%. 20 P4 baseline date was collected in 2015. Endline data will be collected along with P1-P3 in 2016. National Fluency Assessment of Rwandan Schools: Midline Report 2015 30 Analysis of MARS assessment results showed that the percent of P1, P2 and P3 learners with zero scores decreased from baseline to midline. About 14 percent of P1 learners and 7 percent of P2 learners could not solve a single mathematics problem at baseline. By midline, the percent of P1 learners with zero scores had decreased significantly (p<.01) to 8.3%. P2 and P3 learners did not showed statistically significant decreases in zero scores on the MARS assessment. Figure 16 shows the average procedural fluency (speed of mathematics problem solving) at midline, by grade level and type of problem. On average, P1 learners performed the best on the comparing numbers subtest in terms of accuracy and speed. In fact, the average procedural fluency score for comparing numbers was significantly (p<.001) higher than for the other subtests. P2 learners were able to solve subtraction problems the quickest, in which they were able to, on average, solve one more subtraction problem correctly per minute than addition problem. P3 learners had the highest fluency measures in multiplication, and addition. Analysis of fluency in solving mathematics problems at baseline and midline showed that for P1, the number of mathematics problems learners could answer correctly per minute significantly (p<.001) increased for all mathematics subtests. Conversely, P2 and P3 learners showed decreases in the number of mathematics problems they could answer in one minute; however, only decreases were only statistically significant (p<.001) for P3. FIGURE 16. AVERAGE AUTOMATICITY/ FLUENCY IN SOLVING MATHEMATICS PROBLEMS AT MIDLINE, BY GRADE Primary 1 Primary 2 Average number of problems solved in 60 seconds Average number of problems solved in 60 seconds 4.5 4.0 7.0 Addition Subtraction Comparing Numbers 4.5 5.4 3.4 Addition Subtraction Multiplication FIGURE 15. PERCENT OF LEARNERS WITH ZERO SCORES ON ALL MARS TASKS 14.2% 6.9% 3.5% 0.6% 8.3% 5.2% 1.7% P1 P2 P3 P4* Baseline Midline * Baseline data for P4 was collected in 2015 National Fluency Assessment of Rwandan Schools: Midline Report 2015 31 Primary 3 Primary 4 Average number of problems solved in 60 seconds Average number of problems solved in 60 seconds Comparisons by sex did not show statistically significant differences in overall MARS performances at midline for any grade. Notably, although P1 girls performed worse than P1 boys at baseline, by midline, girls had caught up to boys. The contribution of L3 program is in making the overall learning environment more gender-balanced and sensitive to how school environment and teaching practice impacts learning among girls and boys. The L3 program recognizes that the teaching and learning materials that learners encounter every day have a powerful impact on how they learn to view themselves, each other, and the world. The L3 teaching and learning materials are made up of audio programs and text, which includes stories, descriptions of situations and information, as well as pictures in stories used to illustrate a concept. When a learner hears or reads a story or tries to think about a problem, he or she relates it to his/her own experience and tries to see him/herself ‘in the story/picture’. Visual images often act as socializing agents because they convey clear messages to boys and girls about the gender roles they are expected to adopt. The L3 materials have been carefully constructed to reinforce positive messages re boys and girls gender roles, including but not limited to:  Equal number of girls heard/portrayed.  Equal number of boys heard/portrayed.  Equal number of women heard/portrayed.  Equal number of men heard/portrayed.  Equal number of males/females featured in stories and exercises.  Both female and male characters depicted as having equal intelligence (e.g. problem solving abilities)  Both females and males show aptitude in language and mathematics  Both female and males are portrayed as competent in what they do  Both female and males express opinions  Both female and males are both portrayed as confident and assertive 3.3 2.7 3.6 2.2 Addition Subtraction Multiplication Division 10.0 8.2 7.1 4.9 6.3 Addition Subtraction Multiplication Division Comparing Numbers National Fluency Assessment of Rwandan Schools: Midline Report 2015 32  Both female and males express emotions  Both female and males have equal freedom of movement and activity  Both female and males play the same games ( e.g. football)  Both female and males are both capable to perform the same tasks While L3 materials did not specifically target mathematics in erasing harmful gender stereotypes and misconceptions, it appears that it succeeded in doing so. FIGURE 17. AVERAGE PERCENT CORRECT ON MARS TASKS, BY GRADE AND SEX As seen in Table 12, on average, girls in P1, P2 and P3 showed larger gains than boys from baseline to midline on the MARS assessment. Girls in P1 demonstrated largest gains between the baseline and midline, more than doubling the percent of problems answered correctly. The effect sizes of the change between baseline and midline by sex was large across the board, ranging from d=.29 for P3 boys to d=1.03 for P1 girls. 23% 33% 37% 28% 34% 41% 49% 42% 45% 47% 43% 48% Boys Girls Boys Girls P1 P2 P3 Boys Girls BASELINE MIDLINE National Fluency Assessment of Rwandan Schools: Midline Report 2015 33 TABLE 12. AVERAGE PERCENT CORRECT ON MARS ASSESSMENT, BY SEX Grade Sex BASELINE (pct)21 MIDLINE (pct) GAIN (pct) EFFECT SIZE (Cohen’s d) P1 Boys 28.4% (±2.9%) 46.5% (±3.5%) 18.1% (±4.5%) 0.66 (± 0.16) Girls 22.8% (±2.5%) 49.3% (±3.3%) 26.5% (± 4.2%) 1.03 (±0.17) P2 Boys 34.4% (±2.8%) 42.8% (±3.1%) 8.4% (±4.1%) 0.33 (±0.16) Girls 33.4% (±2.6%) 41.6% (±3.0%) 8.2% (± 3.9%) 0.33 (±0.16) P3 Boys 40.8% (±2.7%) 47.7% (±2.8%) 6.9% (±3.9%) 0.29 (±0.16) Girls 37.3% (±2.5%) 45.2% (±2.7%) 7.9% (± 3.7%) 0.35 (±0.16) P422 Boys 53.4% (±2.1%) -- -- -- Girls 50.3% (±2.2%) -- -- -- DETAILED P1 MARS FINDINGS The P1 MARS assessment tests learners’ procedural fluency in key foundational mathematics skills: addition, subtraction and comparing numbers. Each subtest consists of 10 questions and is timed at 120 seconds. P1 learners demonstrated significant gains (p<.001) from baseline to midline. Analysis by sex showed that although boys performed significantly better than girls at baseline, by midline, there was no significant difference in performance on the P1 MARS test by sex. The figure below shows the average MARS percent correct for P1 learners as well as by sex. FIGURE 18. OVERALL P1 MARS SCORES AT BASELINE AND MIDLINE FIGURE 19. AVERAGE PERCENT CORRECT ON P1 MARS SUBTESTS AT BASELINE AND MIDLINE 21 Table shows the average percent correct by MARS subtest by grade. A 95% confidence interval is also reported, which indicates that the point estimate of average percent correct has a margin of error of 5.0%. 22 P4 baseline date was collected in 2015; midline data not available Baseline, 25.7% Baseline, 28.4% Baseline, 22.8% Midline, 47.9% Midline, 46.5% Midline, 49.3% Overall Boys Girls 22.4% 15.1% 39.6% 43.6% 39.2% 60.8% Addition Subtraction Comparing Numbers National Fluency Assessment of Rwandan Schools: Midline Report 2015 34 The analysis of data showed that P1 learners tended to perform best on the comparing numbers subtest in which at midline, learners on average answered 60.8% of questions correctly. P1 learners demonstrated significant gains (p<.001) from baseline to midline on all three MARS subtests: Addition, Subtraction and Comparing Numbers. The largest gain (d=.81) was seen in the subtraction subtests in which P1 learners on average scored 24.1% higher at midline on this subtest. Analysis of zero scores across subtest showed that at midline the percent of learners who could not solve any problems had decreased from baseline. Figure 20 shows the number of problems solved by P1 learners at baseline and midline. At baseline, a large proportion of P1 learners could not solve any subtraction problems (59%), any addition problems (41%) and any number comparison problems (19%). At midline, the percent of P1 learners with zero scores had dropped significantly (p<.001), in which, 23, 33 and 10% of learners could not solve any addition, subtraction and number comparison problems, respectively. FIGURE 20. NUMBER OF PROBLEMS SOLVED ON P1 MARS SUBTESTS DETAILED P2 MARS FINDINGS The P2 MARS assesses learners’ procedural fluency in three grade-level elementary mathematics skills: addition, subtraction and multiplication. Each subtest consists of 10 questions and is timed at 120 seconds. P2 learners demonstrated significant gains (p<.001) from baseline to midline, however gains were not as substantial as seen for P1 learners. Analysis by sex showed that there were no statistically significant differences between the performance of P2 girls and P2 boys. Both male and female learners showed significant gains (p<.001 gains) from baseline to midline (Fig.21). The analysis of baseline and midline P2 MARS 0% 10% 20% 30% 40% 50% 60% zero 1 2 3 4 5 6 7 8 9 10 Percent of learners BASELINE Addition Subtraction Number comparison 0% 10% 20% 30% 40% 50% 60% zero 1 2 3 4 5 6 7 8 9 10 MIDLINE National Fluency Assessment of Rwandan Schools: Midline Report 2015 35 data showed higher achievement in some subtests than others (Fig. 22). Overall, P2 learners tended to perform best on the subtraction subtest, followed by addition. P2 learners demonstrated significant gains (p<.01) from baseline to midline on all three MARS subtests. The largest gain was seen in the addition subtests in which P2 learners on average scored 12.2% higher at midline on this subtest. Overall, analysis of the magnitude of gains from baseline to midline on P2 subtests showed moderate to large effect sizes (d=.16 to d=.37). FIGURE 21. OVERALL P2 MARS SCORES AT BASELINE AND MIDLINE FIGURE 22. AVERAGE PERCENT CORRECT ON P2 MARS SUBTESTS AT BASELINE AND MIDLINE The analysis of assessment results found a substantial reduction in a proportion of learners with zero scores on MARS subtests. The addition and subtraction subtests had the largest percentage of learners with zero scores at baseline. The assessment results showed a notable decrease in zero scores for these subtests at midline (p<.05). Figure 23 shows the number of problems solved by P2 learners at baseline and midline. At baseline, nearly a third (28%) of P2 learners could not solve any subtraction problems, roughly a quarter (22%) could not solve any addition problems and 16% could not solve any multiplication problems. At midline, the percent of P2 learners with zero scores had dropped significantly (p<.05), in which, 21, 17 and 11% of learners could not solve any addition, subtraction and multiplication problems, respectively. Baseline 33.9% Baseline 34.4% Baseline 33.4% Midline 42.2% Midline 42.8% Midline 41.6% Overall Boys Girls Baseline 31.5% Baseline 45.3% Baseline 24.8% Midline 43.7% Midline 50.9% Midline 32.1% Addition Subtraction Multiplication National Fluency Assessment of Rwandan Schools: Midline Report 2015 36 FIGURE 23. NUMBER OF PROBLEMS SOLVED ON P2 MARS SUBTESTS DETAILED P3 MARS FINDINGS The P3 MARS assesses learners’ procedural fluency in the four grade-level elementary mathematics skills: addition, subtraction, multiplication and division. Each subtest consists of 10 questions and is timed at 120 seconds. P3 learners demonstrated significant gains (p<.001) from baseline to midline with average MARS scores of 39.0% at baseline and 46.4% at midline. Analysis by sex showed no significant differences between male and female learners on the P3 MARS assessment. Both boys and girls learners showed significant (p<.001 gains) from baseline to midline as seen in the figure below. The figure below shows the overall average MARS percent correct for P3 learners as well as average performance disaggregated by sex of learner. Analysis of performance on the four MARS subtests showed the P3 learners performed better on certain subtests than others. Overall, P3 learners tended to perform best on the addition and multiplication subtests in which learners were able to answer 55.5% and 52.2% of questions, respectively. P3 learners struggled the most with the division subtest in which learners on average were only able to answer roughly a third (32.2%) of division problems at midline. Overall, P3 learners demonstrated significant gains (p<.01) from baseline to midline on all four MARS subtests. The largest gains were seen in the addition and subtraction subtests in which P3 learners on average scored 10.0% and 7.6% higher at midline on these subtests, respectively. Overall, analysis of the magnitude of gains from baseline to midline on P3 subtests showed moderate to large effect sizes (d=.18 to .27). 0% 5% 10% 15% 20% 25% 30% zero 1 2 3 4 5 6 7 8 9 10 Percent of learners BASELINE Addition Subtraction Multiplication 0% 5% 10% 15% 20% 25% 30% zero 1 2 3 4 5 6 7 8 9 10 MIDLINE National Fluency Assessment of Rwandan Schools: Midline Report 2015 37 FIGURE 24. OVERALL P3 MARS SCORES AT BASELINE AND MIDLINE FIGURE 25. AVERAGE PERCENT CORRECT ON P3 MARS SUBTESTS AT BASELINE AND MIDLINE For P3 learners, the hardest tasks at baseline were subtraction and division, with 18% and 26% of learners failing to solve a single problem on those subtests, respectively. About 10% of P3 learners had zero scores on addition and multiplication subtests at baseline. At midline, division remains the hardest task for P3 learners in which 60% of learners were unable to answer four or more questions correctly. P3 learners performed the best on multiplication and addition subtests. Roughly 40% of P3 learners were able to answer 7 or more multiplication and addition questions correctly at midline which is a substantial increase from baseline in which P3 learners were able to only answer 31% of multiplication and 23% of addition questions correctly. Figure 26 shows the number of problems solved by P2 learners at baseline and midline. FIGURE 26. NUMBER OF PROBLEMS SOLVED ON P3 MARS SUBTESTS AT BASELINE AND MIDLINE Baseline, 39.0% Baseline, 40.8% Baseline, 37.3% Midline, 46.4% Midline, 47.7% Midline, 45.2% Overall Boys Girls 45.5% 26.8% 45.5% 38.3% 52.2% 32.2% 55.5% 45.9% Multiplication Division Addition Subtraction 0% 5% 10% 15% 20% 25% 30% zero 1 2 3 4 5 6 7 8 9 10 Percent of learners Baseline Multiplication Division Addition Subtraction 0% 5% 10% 15% 20% 25% 30% zero 1 2 3 4 5 6 7 8 9 10 Midline National Fluency Assessment of Rwandan Schools: Midline Report 2015 38 Analysis of zero scores showed significant decreases in the percent of P3 learners with zero scores on the subtraction and addition subtests. The percent of learners who were unable to answer any multiplication and division questions changed little between baseline and midline. DETAILED P4 MARS FINDINGS Given that L3 is not scheduled to roll out P4 intervention until 2016, the P4 MARS baseline assessment was conducted in the end of 2015, at the same time as the midline assessment of P1, P2 and P3. Figure 27 shows the average MARS scores for P4 learners at baseline. The P4 MARS assesses learners’ procedural fluency in the four grade-level mathematics operations (addition, subtraction, multiplication and division) as well as their conceptual knowledge (comparing numbers with decimals and fractions). Each subtest consisted of 10 questions and was timed at 60 seconds. At baseline, P4 learners, on average, scored 51.9 percent correct on the overall MARS assessment. On average, boys scored 53.4% on the MARS assessment compared to 50.3% for females; however, this difference is not statistically significant and could be due to chance variation across samples. P4 learners demonstrated strong performance on the Addition subtest: learners on average were able to answer 77 percent of the addition problems. Learners were able to, on average, answer 63% of the subtraction questions. As with P3 learners, the most challenging subtest for P4 learners was the division subtest, in which learners, on average, were only able to answer a third (33.1%) of division problems. FIGURE 27. OVERALL P4 MARS SCORES AT BASELINE FIGURE 28. AVERAGE PERCENT CORRECT ON P4 MARS SUBTESTS AT BASELINE Figure 29 shows the number of problems P4 learners answered correctly at baseline. As seen in the figure, the majority (73%) of P4 learners answered seven or more addition problems correctly at baseline. A substantial proportion of P4 learners were also able to answer seven or more subtraction problems. Learners struggled the most with the division and comparing 51.9% 53.4% 50.3% Overall Boys Girls 77.0% 62.9% 51.8% 33.1% 34.6% Addition Subtraction Multiplication Division Comparing Numbers National Fluency Assessment of Rwandan Schools: Midline Report 2015 39 numbers subtests, in which 20% and 28% of learners were unable to solve a single problem correctly, respectively. FIGURE 29. NUMBER OF PROBLEMS SOLVED ON P4 MARS SUBTESTS AT BASELINE Since P4 is the first primary grade in which the language of instruction is English and not Kinyarwanda, the assessment attempted to find out to what extent learners feel comfortable performing mathematics operations in English. At the end of the assessment, P4 learners were asked “Do you prefer to count and do mathematics problems in English or in Kinyarwanda?” The overwhelming majority (84%) said they preferred to count and do mathematics in Kinyarwanda. Learners who said they preferred to count and do mathematics in English scored significantly higher on MARS: the average MARS score among learners who indicated preference for English was 59%, compared to 50.5% among learners who said they preferred to do mathematics in Kinyarwanda. FIGURE 30. MARS RESULTS AND LANGUAGE PREFERENCE FOR DOING MATH, P4 These findings suggest that some learners might be falling behind in mathematics because they are not sufficiently fluent in the language of instruction. 3% 8% 6% 20% 28% 4% 13% 23% 42% 6% 20% 21% 43% 21% 60% 73% 58% 28% 18% 6% Addition Subtraction Multiplication Division Comparing Numbers Zero 1-3 4-6 7-10 English 16.4% Kinyarwanda 83.6% Language preference for doing math, P4 AVG. MARS SCORE=, 59.0% AVG. MARS SCORE=, 50.5% English Kinyarwanda MARS results by language preference National Fluency Assessment of Rwandan Schools: Midline Report 2015 40 IMPACT OF CONTEXTUAL FACTORS ON MATHEMATICS ACHIEVEMENT LEARNER CHARACTERISTICS AND MARS RESULTS The impact of age was not as straightforward as with literacy: older learners in P1 tended to have better mathematics performance (Pearson’s r=.334 bivariate correlation between average MARS results and age at baseline and .245 at midline, statistically significant at p<.001 level), while at P4 level younger learners performed better than older learners (Pearson’s r=-.154, statistically significant at p<.001 level). The impact of age at P2 was smaller than at P1, and by P3 the age was not found to impact results. This finding is likely due to the fact that P1 mathematics tasks were simple mathematics calculations that children are expected to perform using money when sent to the market for family needs. P1 learners who are older have the advantage of having practiced these tasks longer, so their performance on the test is higher. At P4 level, however, the tasks are much more complex, involving multiplication and division that are not practiced in everyday transactions. FIGURE 31. CORRELATION BETWEEN LEARNER AGE AND MARS RESULTS Bivariate statistical analysis found few statistically significant correlations between learner results in MARS and context interview composites. The correlations were more pronounced in P4, compared to earlier grades. TABLE 13. CORRELATIONS BETWEEN CONTEXT INTERVIEW COMPOSITES AND MARS RESULTS P1 MARS P2 MARS P3 MARS P4 MARS Are you repeating this grade? (yes=1) -.245** Learner's agу .245** .145** -.154** Mother literate (1=yes) How often do you miss school? (4=a lot, 3= sometimes, 2=rarely, 1=never) -.123** -.087* -.125** -.181** How often are you late for school? (4=a lot, 3= sometimes, 2=rarely, 1=never) -.114** P1 0.245 P2 0.145 P3 0 P4 -0.154 National Fluency Assessment of Rwandan Schools: Midline Report 2015 41 P1 MARS P2 MARS P3 MARS P4 MARS Have you or any of your siblings ever repeated a grade? (1=yes) -.131** Do your parents/caregivers check your homework? (1=yes) .111** When not understanding a lesson, do you ask questions? (1=yes) -.167** -.220** Did you have something to drink today (like water, tea, milk or juice)? (1=yes) .134** .160** Did you have something to eat today, like potatoes, rice, bread or beans? (1=yes) .092* .114** Do you have radio or cell phone at home? (1=yes) .090* Does anyone at your house have a bicycle/motorcycle or a car? (1=yes) .184** *Correlations are significant at the >0.05 level (2-tailed) **Correlations are significant at the >0.01 level (2-tailed) Weak positive association (.1 to .199) Stronger positive association (.2 and higher) Weak negative association (-.1 to -.199) Stronger negative association (-.2 and higher) Literacy-related questions, such as reading stories in the class and taking books home were not included in the correlation analysis with mathematics results. Additionally, questions that had no or very little variance were also not included in the analysis23 . SCHOOL CHARACTERISTICS To help us better understand the variation in learner scores, we looked at the differences in school characteristics, such as distance to Kigali and District Office, teacher absenteeism, and other factors. A variety of factors were found to be associated with learner performance on MARS. Similar to the findings presented in the FARS section, such factors as overcrowding (teacher to learner ratio), availability of a school or a community library, school size, and the nursery attached to school were not found to be associated with learner results. However, similar to FARS results, teacher absenteeism was found to be negatively associated with learner performance on MARS, although the relationship was not as strong as with FARS results. Table below shows that the higher the proportion of teachers absent, the lower the learner results are. 23 Nearly all learners said yes to the following questions: Do your parents/caregivers want you to go to school every day?, Does your mathematics teacher check your work that you do in class?, Does your mathematics teacher check/mark your homework?, Does your Kinyarwanda teacher check your work that you do in class?, Does your Kinyarwanda teacher check/mark your homework? Additionally, nearly all learners said they speak Kinyarwanda at home. These questions were excluded from correlation analysis. National Fluency Assessment of Rwandan Schools: Midline Report 2015 42 TABLE 14. CORRELATIONS BETWEEN TEACHER ABSENTEEISM AND MARS RESULTS, AT MIDLINE P1 P2 P3 P4 Teacher absenteeism24 -.091* Not significant -.167** -.181** * Correlations are significant at the >0.05 level (2-tailed) **Correlations are significant at the >0.01 level (2-tailed) Shaded correlations are those with r > .1 Differently from FARS, MARS results were not found to have significant negative correlations with distance to Kigali. However, similar to FARS results, distance to the local District Office was found to be negatively associated with learners’ results on MARS, although the coefficients were not as large as with FARS results. Distance to the District office was found to be an important predictor of how well learners performed on the MARS, with school farther from the District Office having learners with lower MARS scores. TABLE 15. CORRELATIONS BETWEEN DISTANCE TO DISTRICT OFFICE AND FARS RESULTS, AT MIDLINE P1 P2 P3 P4 Distance to the District Office -.098* -.082* -.146** -.096** SUMMARY Overall, mathematics assessment results show that, at midline, a large proportion of primary grade learners are still developing basic mathematics skills that would enable them to perform grade-level number operations with accuracy and speed. The majority of tested learners were able to work out some problems correctly at midline, but very few learners demonstrated grade-level procedural fluency on elementary mathematics operations. This suggests that many learners do not yet possess grade-level procedural mathematics fluency which they need to have to be able to advance to more complex mathematical problems. Analysis of MARS results from baseline to midline that learners in P1, P2, and P3 showed statistically significant gains (p<.001) in average MARS scores from baseline to midline. These findings suggest significant improvement in mathematic achievement of P1, P2 and P3 learners since the roll-out of the L3 nationwide intervention. 24 Teacher absenteeism was computed as the percent of teachers absent on the day of the assessment and on the day previous to that day, averaged. National Fluency Assessment of Rwandan Schools: Midline Report 2015 43 READING AND MATHEMATICS SCHOOL-LEVEL RESULTS An analysis of school averages in each grade25 showed an improvement of average results across schools, in each grade on both tests. Figures 32 through 34 show scatterplots of average percent of words in a grade-level text read correctly by tested learners, by grade, in each tested school at midline and baseline. Each cross on the graphs represents average results in a study school. FIGURE 32. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P1 LEARNERS ON ORAL READING FLUENCY TEST IN STUDY SCHOOLS FIGURE 33. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P2 LEARNERS ON ORAL READING FLUENCY TEST IN STUDY SCHOOLS 25 The averages were computed using unweighted percent correct read by sampled learners in each grade, for each school. The mean and standard deviation are calculated at the school-level. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=51%, SD=21% mean=19%, SD=15% P1 midline P1 baseline 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=51%, SD=21% mean=44%, SD=22% P2 midline P2 baseline National Fluency Assessment of Rwandan Schools: Midline Report 2015 44 FIGURE 34. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P3 LEARNERS ON ORAL READING FLUENCY TEST IN STUDY SCHOOLS As seen in the figures above, FARS P2 scores are the most dispersed at both baseline and midline. For each grade, groups of outlier schools are seen that have extreme FARS results: either performing substantially worse than other schools in the sample or performing substantially better. P2 had five schools scoring on average 20% or lower on the FARS at midline; at the other extreme there are five schools scoring on average 80% or higher. Assessor interviews indicated that schools on the lower end of achievement are located in the areas with higher levels of poverty, while schools that perform better, on average, have an excellent leadership from their head teachers. Some schools that do not perform well are located either near the border where there is a high proportion of transient population, or in very sparsely populated areas where learners are obliged to travel far to school and the attendance suffers as a result. Similar analysis found an improvement in school averages in mathematics scores across all schools in the sample. P1 MARS results are the most dispersed, with the majority of schools with average scores between 36% and 66%. A few outlying cases in P1 have extreme results: one school with an average of 1%; six schools had lower average results (below 30%), and two schools had very high average results (over 80%). A study of these outliers could shed more light on the reasons behind these anomalies. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=46%, SD=14% mean=38%, SD=15% P3 midline P3 baseline National Fluency Assessment of Rwandan Schools: Midline Report 2015 45 FIGURE 35. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P1 LEARNERS ON MARS IN STUDY SCHOOLS FIGURE 36. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P2 LEARNERS ON MARS IN STUDY SCHOOLS FIGURE 37. AVERAGE PERCENT CORRECT AT BASELINE (SY 2014) AND MIDLINE (SY 2015) AMONG TESTED P3 LEARNERS ON MARS IN STUDY SCHOOLS 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=49%, SD=16% mean=27%, SD=13% P1 midline P1 baseline 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=43%, SD=13% mean=35%, SD=13% P2 midline P2 baseline 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Average percent of words in a grade level text read correctly average within the grade across all study schools mean=47%, SD=14% mean=39%, SD=12% P3 midline P3 baseline National Fluency Assessment of Rwandan Schools: Midline Report 2015 46 Comparisons of school-level average results on oral reading fluency and mathematics assessment tasks found that learners in the same schools do below average on both tests, and do above average on both tests. In other words, many of the same dozen schools showed low average results among their P1, P2, P3 and P4 learners in both oral reading fluency and mathematics assessment. On the other end of the spectrum, in a handful of schools all tested learners performed well above average. In the vast majority of schools, however, the average learner results varied greatly. The analysis of average learner performance in study schools showed that, on average, the same schools did well at baseline and at midline. Statistical comparisons of background characteristics of the top performing and bottom performing schools26 demonstrated a few significant differences in these schools. The most notable differences were found in two variables: the distance from school to District Office, and the average percent of absent teachers on the day of the test and the preceding day. The average distance to District Office among top performing schools was 12.8 kilometers, while the average distance to District Office among bottom performing schools was 44.7 kilometers. The average percent of absent teachers over two days was only 3.7% in the top performing schools, compared to 18.9% in the bottom performing schools. The correlation between the distance to District Office and teacher absenteeism is very strong (r = .511) in the general dataset. FIGURE 38. FACTORS CORRELATED WITH SCHOOL PERFORMANCE (N=16) Other variables did not show a significant difference between the top and bottom performing schools. On average, the top performing schools were slightly larger, had higher teacher to learner ratio and had similar repeater rate. 26 To determine the top performing and bottom performing schools, results across the tested grades were averaged for each school by test (FARS/MARS) and the schools with highest and lowest average result were selected for further analysis. 12.9 KM 3.7% 44.7 KM 18.9% Distance to District Office Percent of teachers absent TOP PERFORMING BOTTOM PERFORMING SCHOOLS National Fluency Assessment of Rwandan Schools: Midline Report 2015 47 STUDY OF GRADE REPEATERS DESCRIPTION OF THE SAMPLE OF GRADE REPEATERS Baseline Study Repeaters. During the baseline assessment in 2014, the assessment team collected personal information on the random sample of tested learners to allow us to track those learners in subsequent years. Consequently, in the October 2015 midline assessment, it was possible to establish how many learners from the original sample progressed to the next grade, how many were retained in the previous year’s grade, and how many dropped out. The assessment team was able to locate and test 1,130 learners from the baseline sample of 1,799 learners (62.8%). There were no substantial differences between the male and female learners and across grades. Of the tracked learners who were present on the day of midline testing, 175 learners (6.8% of the sample) were found to be repeating the same grade where they were at baseline testing a year previously. This section presents detailed analysis of repeaters versus non-repeaters. It is important to note that many repeating learners show poor attendance which, in part, is the reason why they are identified by the school to repeat the grade. Therefore, it is likely that many of the repeating students were absent on the day of the midline assessment. This assumption is also supported by the fact that while teachers noted that 15.6% of their learners were repeaters, on average, the longitudinal study was able to locate only 6.8% of the sample among the repeaters. Among 37.2% of the original baseline sample that the study team was not able to find on the day of the midline testing, there were undoubtedly some repeaters. Figure 39 shows the proportion of longitudinally tracked repeaters tested during the midline vis￾à-vis the overall proportion of the repeaters. As clearly seen in this figure, the study was able to track and test fewer than a half of all repeating learners. Characteristics of All Repeaters. Most of learners repeating the grade were found in P1. The figure below shows the grade distribution, both for the repeaters tracked from the baseline, and for those newly sampled learners who were discovered to be repeating the grade. As seen from Figure 40, the percent of learners who repeat the grade decreases as they progress from grade to grade. FIGURE 39. LONGITUDINALLY TRACKED LEARNERS AND REPEATERS 15.6% 6.8% Average % repeaters Tracked repeaters National Fluency Assessment of Rwandan Schools: Midline Report 2015 48 FIGURE 40. PERCENT OF REPEATERS, BY GRADE Data analysis by sex showed that slightly more boys were repeating P1 and slightly more girls were repeating P3. Because of the small sample size of the repeaters tracked from the baseline, it is unknown to what extent the difference is due to a random sample error. The test of significance did not find the relationship between sex and grade repetition to be statistically significant. TABLE 16. SAMPLE OF REPEATING LEARNERS TRACKED FROM BASELINE, BY SEX Province P1 P2 P3 TOTAL Boys 43 21 20 84 51.2% 25.0% 23.8% 100.0% Girls 38 21 32 91 41.8% 23.1% 35.2% 100.0% TOTAL 81 42 52 175 46.3% 24.0% 29.7% 100.0% Who are the repeating learners? They are, on average, older than their non-repeating peers. The proportion of boys and girls is similar to that of non-repeaters. Repeating learners had slightly lower scores on home environment, school and teacher, and socio-economic composites, compared to non-repeaters, although the difference was not statistically significant. The difference on risk factors composite was found to be statistically significant. National Fluency Assessment of Rwandan Schools: Midline Report 2015 49 FIGURE 41. DEMOGRAPHIC CHARACTERISTICS OF REPEATERS COMPARED TO NON-REPEATERS PERFORMANCE OF REPEATERS IN READING An analysis of learners who are repeating the grade and those who are attending the current grade for the first time revealed striking differences in their oral reading skills. While a higher proportion of P1 learners who were repeating the grade had zero scores on the reading passage, the average fluency rate among repeaters was not statistically different from those who were attending P1 for the first time. In P2, P3 and P4, by contrast, repeaters demonstrated both slower average reading speed in Kinyarwanda and a higher percent of learners with zero scores (p <.01 level). There was no statistically significant difference in average reading speed in English between repeaters and non-repeaters. The figure below shows differences between these two groups of learners in average fluency as well as in a percent of learners with zero scores on a grade-level reading test. 8.6 10.6 11.8 12.5 7.3 9.1 10.6 11.6 P1 P2 P3 P4 Age of repeating and non-repeating learners repeaters non-repeaters 2.7 6.5 2.1 5.4 2.9 6.9 2.3 3.9 home factors composite school factors composite socio-economic status risk factors composite Learner interview composite averages repeaters non-repeaters National Fluency Assessment of Rwandan Schools: Midline Report 2015 50 FIGURE 42. AVERAGE ORAL READING FLUENCY RESULTS AMONG REPEATERS AND NON-REPEATERS, AT MIDLINE Learning Gains over a Year. When learners are retained to repeat a grade, how much do they learn, on average, during the year they repeat? While the sample of learners that the study team was able to track and re-test was less than a half of all repeaters, as shows in the opening part of this section of the report, it is helpful to understand whether these learners improved over the course of the year. The figure below shows changes in oral fluency of repeating learners who were tracked from baseline. The figure demonstrates that over the course of one academic year repeating learners experienced gains in oral reading fluency. P1 repeaters in the sample actually caught up to their non-repeating peers, while P2 and P3 repeaters closed more than half of the gap between them and non-repeaters. Blue circles denote the average 9.1 15.0 13.8 34.6 25.5 6.8 22.3 25.9 41.2 26.3 P1 P2 P3 P4 KR P4 English Average Kinyarwanda fluency results, 2015, in words correct per min repeaters non-repeaters 42% 34% 42% 16% 26% 54% 24% 11% 7% 22% P1 P2 P3 P4 KR P4 English Zero scores on Kinyarwanda fluency results, 2015 repeaters non-repeaters National Fluency Assessment of Rwandan Schools: Midline Report 2015 51 fluency of non-repeaters in the sample. It is important to note that these findings represent results from less than a half of repeating learners. FIGURE 43. CHANGES IN FLUENCY OVER THE COURSE OF 2015 ACADEMIC YEAR AMONG REPEATERS PERFORMANCE OF REPEATERS IN MATH Midline measurement of learner performance in mathematics did not find statistically significant differences in the performance on MARS between P2, P3 and P4 repeaters and non￾repeaters. However, P1 repeaters did statistically significantly better than non-repeaters, and also had significantly fewer zero scores. This can possibly be explained by the fact that repeaters in P1 in particular are older than non-repeaters and had more practice with basic mathematics operations. 0.2 6.8 6.9 4.2 14.5 22.3 12.0 23.0 25.9 0 5 10 15 20 25 30 words correct per minute P3 repeaters gained 11 wcpm, on average P2 repeaters gained 10.4 wcpm, on average P3 P2 P1 repeaters gained 6.7 wcpm, on average P1 average fluency of non-repeaters 52 FIGURE 44. AVERAGE MATHEMATICS RESULTS AMONG REPEATERS AND NON-REPEATERS, AT MIDLINE Learning Gains over a Year. How much mathematics did repeaters learn in one year? The figure below shows changes in average percent correct across all MARS subtests of learners who repeated the grade, over the course of one acadmic year. The figure demonstrates that over the course of one academic year repeating learners experienced gains in procedureal fluency and accuracy of solving foundational mathematics problems. Study repeaters by and large caught up to their non-repeating peers. Purple circles denote the average MARS scores of non-repeaters in the sample. It is important to note that these findings represent results from less than a half of repeating learners. 59.5% 41.8% 40.4% 49.5% 43.2% 42.3% 46.9% 52.1% P1 P2 P3 P4 Average MARS results, 2015, in average percent correct across all tasks repeaters non-repeaters 2.8% 4.2% 5.2% 1.9% 10.5% 5.4% 1.5% 0.5% P1 P2 P3 P4 Zero scores on MARS fluency results, 2015 repeaters non-repeaters National Fluency Assessment of Rwandan Schools: Midline Report 2015 53 FIGURE 45. CHANGES IN MARS OVER THE COURSE OF 2015 ACADEMIC YEAR AMONG REPEATERS TEACHERS’ REPORTS ABOUT REPEATERS To better understand the reasons why learners are held back a year, the study team asked teachers familiar with repeating learners. The most common reason teachers cited that learners were retained in the current grade was low academic performance (57.6%) and poor attendance (23.5%). Reasons for grade repetition were consistent across sex and grade. FIGURE 46. REASONS WHY LEARNER WAS RETAINED IN CURRENT GRADE (N=170) The majority of repeaters were not orphans; in fact, only 16.5% were orphans. Of the baseline repeaters, very few (3.8%) were identified by teachers as having learning barriers such as poor vision, hearing, disabilities or chronic diseases. 6.3 43.2 49.8 19.1 38.3 42.3 24.4 44.0 46.9 0 10 20 30 40 50 60 percent correct on MARS P3 repeaters gained 19.7%, on average P2 repeaters gained 19.3%, on average P3 P2 P1 repeaters gained 43.4%, on average P1 average MARS percent correct of non-repeaters 57.6% 23.5% 6.5% 4.7% 3.5% 2.9% 1.2% Low grades Poor attendance Problems with health Parent requested student repeat grade Student was too young Family problems Behavioral/social problems National Fluency Assessment of Rwandan Schools: Midline Report 2015 54 FIGURE 47. BACKGROUND CHARACTERITICS OF REPEATERS (N=170) Teachers reported that the large majority of repeaters attended school regularly during the year they repeated and that the majority had improved sufficiently to be promoted to the next grade next year. Roughly a quarter of repeaters were at risk for having to repeat the current grade again the following year. Analysis by sex showed that female repeaters were slightly more likely to attend school regularly than male repeaters, however, the likelihood of being promoted to the next grade, was similar for both boys and girls. Teachers remarked that common risk factors that might affect children’s academic performance and attendance in school was poverty: many of the children came from poor families and as a result came to school hungry and could not concentrate on their lessons. Additionally, several children were needed to help out at home with household chores such as looking after the cattle, cooking and taking care of the children, which caused learners to miss school. Teachers also mentioned lack of involvement of parents in their child’s education was a risk factor citing several parents who were not involved in their child’s education. According to interviewed teachers, roughly one fifth (17.7%) of repeaters from the baseline sample are at risk of dropping out. Slightly more male repeaters (20.7%) were identified by their teachers as having potential for dropping out than female repeaters (14.6%); however, the difference was not statistically significant. EVIDENCE FROM OTHER STUDIES The study of repeaters found that repeating a grade was academically beneficial for most of the learners that the study was able to track and test twice: at baseline and a year later. However, this finding has one essential caveat: less than a half of repeaters were tracked and re-tested a year later. As absenteeism is one of the key reasons why learners are identified to repeat the grade, it is likely that repeating learners absent on the day of the assessment were 80.0% 93.5% 16.5% 3.8% Is this child an orphan? Does this student have learning barriers? (e.g. poor vision or hearing, disability, chronic diseases) NO YES National Fluency Assessment of Rwandan Schools: Midline Report 2015 55 also generally more absent and consequently would be likely not to show substantial improvements in their test results. The long-term issues associated with repeating the grade have been extensively studied in other countries. For example, a meta-analysis of 19 studies of grade retention in the USA27 found that initial achievement gains may occur during the year the student is retained. However, the consistent trend across many research studies is that achievement gains decline within 2-3 years of retention, such that retained children either do no better or perform more poorly than similar groups of promoted children. Another meta-study of 20 rigorous studies of long-term effects of grade repetition28 found similar trends. Retention does not appear to have a positive impact on self-esteem or overall school adjustment; however, retention is associated with significant increases in behavior problems as measured by behavior rating scales completed by teachers and parents, with problems becoming more pronounced as the child reaches adolescence. Results indicate that grade retention had a negative impact on all areas of achievement (reading, math and language) and socio-emotional adjustment (peer relationships, self-esteem, problem behaviors, and attendance). The effects of grade repetition in Rwandan schools would warrant a separate study. Bearing in mind that large class sizes are one of the causes of poor academic results overall, the fact that many of the pupils are over age might be a contributing factor to poor academic performance of all learners. Schools need to provide support such as remediation for learners who fall behind to help advance them together with their age cohort. If Rwanda wants pupils to be workforce-ready, retaining learners above the average age in early primary grades may not be an effective policy in the long run. 27 National Association of School Psychologists. (2011). Grade retention and social promotion (White Paper). Bethesda, MD. Accessed from https://www.nasponline.org/ in March, 2016. 28 Jimmerson, S.R. (2001). Meta-analysis of Grade Retention Research: Implications for Practice in the 21st Century. School Psychology Review. 2001, Volume 30, No. 3, pp. 420-437. National Fluency Assessment of Rwandan Schools: Midline Report 2015 56 SCHOOL, TEACHER AND LEARNER-LEVEL FINDINGS The school’s environment and management is critical to understanding the teaching and learning that is taking place in the school. Concurrently with the learner assessment, the data collection team gathered data on school context, grade-level resources and practices related to L3 activities. In all, 60 head teachers and 433 P1, P2, P3 and P4 teachers were surveyed. Data was collected to provide an overall picture of the school and learning environment; particularly data was collected on: 1) the school environment, 2) the teaching and learning demographic, 3) school leadership, 4) parent and community involvement, and 5) teacher practices and beliefs. This information was used as covariates in oral reading fluency and mathematics assessment data analysis. SCHOOL ENVIRONMENT Data were collected on the school infrastructure of 60 sampled schools. Observations of sampled schools showed varying conditions in the school infrastructure and learning environment in these schools. The majority of schools were scored by assessors as having either “good” or “adequate” school buildings, roofs and separate latrines for boys and girls. Slightly more than half of schools had adequate or good electricity and a safe space for children to run and play outside. One in five observed schools did not have any electricity; nearly a third (31%) did not have a suitable space for play for children. Very few schools (10%) had drinking water available for learners. Libraries in good conditions were also not found to be common. In fact, 40% of schools did not have drinking water; 43% did not have a school library. The observed state of the school library varied from school to school, in which some schools had well-equipped and well-managed libraries; other school libraries were small and contained only a few books. Very few head teachers (25.0%) reported access to a community library or similar place in the community where learners could borrow books to read in their communities. National Fluency Assessment of Rwandan Schools: Midline Report 2015 57 FIGURE 48. OBSERVED CONDITION OF SCHOOL INFRASTRUCTURE IN SAMPLED SCHOOLS (N=58) Head teachers were also asked whether they received support from other organizations/NGOs. About a third (36.7%) of schools indicated that they received support from other organizations, largely in the form of provision of teaching and learning materials, teacher training and construction. Schools reporting holding school assemblies frequently. Of the sampled schools nearly two￾thirds (61.7%) indicated that they held daily school assemblies. Roughly a third (30.0%) held assemblies at least once a week. A few schools indicated that they held assemblies less frequently (1.7% said they held assemblies once a month; 6.7% said they held assemblies once a term). TEACHERS Overall, the majority of schools reported that they had one to four P1 classrooms (although in some schools this number was as high as six or seven), and one to three P2, P3 and P4 classrooms. The table below shows the average number of learners enrolled by grade. On average, near gender parity can be seen in enrolment for all four grades, with roughly equal numbers of male and female learners enrolled. Further analysis by grade shows that the average number of enrolled learners decreases as they transition into higher grades. As can be seen in the table below, on average, P4 has nearly 50% fewer learners than P1. The decrease in enrolment is similar for both males and female learners. 1 2 3 4 School Building Roof Electricity Drinking Water Separate Latrines Library Safe space to run and play outside NONE POOR ADEQUATE GOOD National Fluency Assessment of Rwandan Schools: Midline Report 2015 58 Sampled schools reported relatively low dropout rates during the 2015 school year. Overall the majority of schools reported total dropout rates by grade ranging between zero and four percent. The table below shows the average reported dropout rate by grade. As seen in the table below, on average, boys had slightly higher dropout rates than girls across all grades (P1-P4). TABLE 17. SCHOOL ENROLMENT AND DROPOUT STATISTICS, BY GRADE Grade № of learners enrolled range Avg. № Learners Enrolled Avg. Dropout Rate29 Male Female TOTAL Male Female TOTAL P1 45-606 125 116 241 1.7% 1.3% 1.5% P2 24-519 94 94 188 1.5% 1.3% 1.4% P3 25-417 76 77 153 2.5% 1.5% 2.0% P4 20-366 63 67 130 3.1% 2.1% 2.4% On average, teachers reported that 15.6% of learners in their classrooms were repeaters or have repeated the same grade. The percentage of repeaters by grade was fairly consistent with the exception of P1, which had, on average, the largest reported percentage of repeaters per classroom – 18.2%. The average percent of learners repeating the grade across the first three primary grades was 17.4%. There was a significant variation across schools with regard to the proportion of learners repeating grade, ranging from reported zero percent to the reported 62 percent average across the first three primary grades. The figure below summarizes the differences in the percent of P1 through P3 learners who were repeating the grade in the school year 2015. 29 The dropout rate was calculated by dividing the total number of learners by grade who dropout during the current school year by the total number of enrolled learners by grade. National Fluency Assessment of Rwandan Schools: Midline Report 2015 59 FIGURE 49. PERCENT OF REPEATERS IN P1-P3, ON AVERAGE, IN THE STUDY SCHOOLS (N=59) Teachers also reported high absenteeism of their learners: teachers reported that nearly 30% of their learners in their classroom were absent on the day of data collection. Analysis by grade showed that significantly more learners in the higher grades (P3 and P4) were absent than in the lower grades (P1 and P2); this was statistically significant at the p<.01 level. The majority of head teachers reported that their schools had between 4 and 9 teachers teaching P1, P2 and P3 learners. Only 8% had more than 15 teachers. On average, in the sampled schools, two-thirds (67%) of P1-P3 teachers are female. The most common subject taught by male teachers was math, while the most common subject for female teachers was Kinyarwanda. TABLE 18. LEARNER/TEACHER RATIO STATISTICS, BY GRADE Grade № of classes № of teachers Learner/ Teacher ratio P1 2.5 3.9 66 : 1 P2 2.2 3.4 58 : 1 P3 1.8 3.2 54 : 1 P4* 1.6 Not collected * Note: Data on the number of P4 teachers was not collected during data collection An analysis of learner/teacher ratio showed that, on average, a P1 classroom can be expected to have between 11 and 121 learners per one teacher; a P2 classroom can be expected to have between 8 and 158 learners enrolled per one teacher; a P3 classroom can be expected to have 10.2% 6.8% 22.0% 16.9% 11.9% 10.2% 6.8% 15.3% zero 1 to 5% 5 to 10% 10 to 15% 15 to 20% 20 to 25% 25 to 30% over 35% “The classrooms are overcrowded and this is the barrier to learn effectively.” -Teacher in Kigali province National Fluency Assessment of Rwandan Schools: Midline Report 2015 60 between 8 and 139 learners per one teacher, with respective averages of 66, 58 and 54 learners per teacher. Teaching experience (in years) of sampled teacher ranged very significantly. The average number of years of teaching experience was 12 years; the median was 9 years. Overall, nearly two-thirds of teachers (66.1%) had attended TTC in preparation for teaching; 32.8% had attended General Secondary School (GSS); and 1.2% reported having no professional preparation for teaching. Only 23.0% of teachers overall had received training on literacy or numeracy by an NGO30 . Most teachers (89.9%) had been with the same class since the beginning of the year. SCHOOL LEADERSHIP (POLICIES, PRACTICES, AND MONITORING) All sampled head teachers reported that their school has a system for tracking teacher attendance, in which the majority (98.3%) of head teachers collected teacher attendance data daily. Teacher attendance records showed that on average, on a given day, 7.5% of all P1, P2, P3 teachers were absent.31 These results were consistent with self-reported absenteeism by teachers in which the majority of teachers (90.5%) reported they were not absent at all the previous week; 7.2% of teachers said they were absent one time; 2.3% were absent more than once. All head teachers reported that they observe teachers teaching in the classroom. More than three-quarters (78.3%) of head teachers reported that they observe teachers on a weekly basis, while 18.3% observed teachers monthly. Surveyed teachers corroborated these responses, in which all surveyed teachers reported that school administration observed them teaching in the classroom. More than half (53.6%) of surveyed teachers indicated that they are observed one a week; while more than a third (37.9%) reported being observed once a month. The frequency in which teachers reported being observed teaching was largely consistent across subject (Kinyarwanda, English and Math) as well as by the number of years teaching. The majority (96.7%) of sampled head teachers also reported that they monitor reading progress of learners. Various methods of monitoring progress were reported; the most common methods being classroom observation, monitoring learners’ results on tests given by teachers and evaluating children orally. 30 L3 is a national literacy and numeracy program serving all government support primary schools. The purpose of this question was to establish if teachers were receiving additional literacy and numeracy training from other NGOs. 31 Teacher attendance records were reviewed. Data was collected for teacher absences on the day of the data collection and the day before. The average percent of absent teachers was calculated by averaging the daily teacher absenteeism rate for the two data points. Data was not collected for P4 teachers. National Fluency Assessment of Rwandan Schools: Midline Report 2015 61 FIGURE 50. METHODS USED BY PRINICPALS/HEAD TEACHERS TO MONITOR THE READING PROGRESS OF LEARNERS IN SCHOOLS (N=60; MULTIPLE RESPONSE) On average, teachers reported that 14.1% of their learners in their class have learning barriers. Interviews with head teachers showed that slightly more than two-thirds (68.3%) of head teachers tracked children with learning barriers; the remaining third did not record this information. Of those head teachers who did track learners with learning barriers, a large majority (78%) reported having some remedial measures to support children with learning barriers. Almost every teacher (99.3%) takes attendance daily and used a daily attendance register (98.2%). The majority (61.9%) of teachers responded that most of their learners come to school on time or early and that only a few learner arrive to school late. Nearly all schools in the sample (89.9%) have discipline measures for children who come late to school. The most common form of discipline is tasking the learner with helping to clean (75.5%). Several schools reported using corporal punishment (13.2%) to punish learners who were late to school. Similarly, the majority (76.3%) of schools also have discipline measures for learners who miss school. Head teachers were also asked about their methods of encouraging learners to come to school. The most prevalent methods of encouraging learners to attend school were parent meetings/PTA, provision of incentives for good academic performance, ensuring separate toilets for girls and boys and through competitions. The figure below shows the distribution of head teachers’ responses. 76.7% 21.7% 18.3% 10.0% 8.3% 1.0% Classroom Observation Testing (tests given by teachers) Evaluate children orally End of term evaluations Check assignments/homework Progress Reports Learners who are late are tasked with helping to clean in 75.5% of schools; 13.2% of schools reported using corporal punishment to discipline learners who were late to school. National Fluency Assessment of Rwandan Schools: Midline Report 2015 62 FIGURE 51. METHODS TO ENCOURAGE LEARNERS TO ATTEND SCHOOL (N=60; MULTIPLE RESPONSE) Teachers were also asked how they decide when to hold a learner back a year. Almost all teachers said that low grades could be a reason that a learner is held back. Half of teachers would hold back a learner because a parent requested it. Poor attendance would be taken into account by about a third of sampled teachers. Head teachers reported that similarly low grades were the most important factor for holding a learner back a year; however, head teachers reported that poor attendance was a more important factor to holding children back than parent requests. FIGURE 52. TEACHERS’ REASONS THAT A LEARNER CAN BE HELD BACK A YEAR (N=433) 91.7% 58.3% 40.0% 33.3% 30.0% 25.0% 8.3% 3.3% Parent Meetings/PTA Incentives for good academic performance Separate toilets for girls and boys Competitions Special rooms for girls Punishment if student does not come to school Provide shoes to learners Provide milk to learners 93.3% 49.4% 35.1% 8.3% 7.9% Low grades Parent requests that the student be held back Poor attendance Student is an inappropriate age for the grade Behavioral problems National Fluency Assessment of Rwandan Schools: Midline Report 2015 63 LEARNING ENVIRONMENT Head teachers were asked how many learners share one desk in the classroom by grade. On average, head teachers reported that in the majority of P1, P2, P3 and P4 classrooms between two to four learners share one desk; responses varied from one to six learners per desk. Teachers reported large ranges in age of learners in their classes, in which learners could be up 6 years older, on average, than the expected enrolment age for their grade. Observed conditions in sampled classrooms showed that the majority of classrooms were in adequate or good condition with respect to blackboards, clean classroom space, good lighting, desks for learners, and reading and writing materials for learners. In more than half of observed schools print materials (posters, signs, etc.) were observed on school or classroom walls. Almost all sampled teachers reported receiving chalk, books for learners and posters for the classroom from the school administration. Around two-thirds of teachers reported receiving instructional technology 32 . Additional materials teachers mentioned receiving, included: pens/pencils, attendance registers, class diaries, and notebooks. FIGURE 53. MATERIALS THE SCHOOL ADMINISTRATION PROVIDES TEACHERS WITH (N=433, MULTIPLE REPSONSE) 32 L3 provides technology for teachers’ use to all government supported primary schools, however, it is possible that this technology may not be assigned exclusively to individual teachers. Technology includes mobile phones with SD cards that contain the interactive audio instruction, speakers, and, where needed, solar panels for power. 97.0% 94.0% 80.6% 62.8% 12.9% 9.9% 7.4% 3.7% Chalk Books for students Posters for classrooms Instructional technology Paper for students Pencils/pens Notebooks Laptops for learners TABLE 19. AVERAGE AGE RANGE OF LEARNERS, BY GRADE Grade Average age range P1 6 to 11 years P2 7 to 12 years P3 8 to 14 years P4 9 to 15 years National Fluency Assessment of Rwandan Schools: Midline Report 2015 64 Head teachers were asked about challenges they face in their schools that inhibit teaching and learning. The most common challenges that head teachers reported were lack of support of parents/caregivers for their child’s education; lack of help at home for learners with their homework; overcrowding in classrooms; and low literacy levels of parents/caregivers. The figure below shows the percent of schools that identified the following challenges as moderate or severe problems in their school. FIGURE 54. COMMON CHALLENGES TO TEACHING AND LEARNING IN SCHOOLS (N=60, MUTLIPLE RESPONSE) What are moderate or severe problems that inhibit teaching and learning for your school? L3 MATERIALS AND TECHNOLOGY Head teachers were asked the number of materials they received from L3 over the last three school terms. Table 20 shows the average number of materials received for P1, P2 and P3. TABLE 20. AVERAGE NUMBER OF L3 MATERIALS SCHOOLS RECEIVED, BY GRADE Material P1 P2 P3 Kinyarwanda guide 2-3 2-3 3-4 Kinyarwanda read-aloud 1-2 1-2 2-3 Mathematics guide 1-2 1-2 2-3 Kinyarwanda reader term 1 293 194 152* Kinyarwanda reader term 2 296 188 --- Kinyarwanda reader term 3 296 200 --- English guide 1-2 2-3 2-3 English reader 279 192-204 159 English read-aloud** --- --- 8-9 *P3 received a comprehensive Kinyarwanda Daily reader instead of Kinyarwanda readers by term. 86.7% 70.0% 68.3% 68.3% 61.7% 53.3% Parents/Caregivers do not support their childrens' education Students receive little help with school work at home/do not complete homework Too many students in a class Parents/Caregivers are not literate Students are hungry Students do not attend class consistently or arrive late National Fluency Assessment of Rwandan Schools: Midline Report 2015 65 **Only P3 classrooms received English read-alouds L3-provided printed teachers’ guides were observed in sampled schools. Primarily they were observed in use by teachers; in 10 schools they were found in the library. L3-provided learner books were also observed in schools. In the majority (76%) of observed schools learner books were observed in use by learners or on the classroom shelves. In a few schools, learner books were found in the library or the headmaster’s office. In nearly all schools, learner books looked used. Head teachers were also asked the quantity of technological equipment received from L3, such as speakers, cell phones, SD cards, and solar panels. All sampled schools reported receiving at least one speaker, cell phone or SD card. On average schools received around five speakers, cell phones and SD cards from L3. Nearly half of the schools (45.0%) in the sample received from L3 a solar panel. Nearly nine out of ten P1, P2, and P3 teachers received technology (cell phone, speakers, and/or SD card) from L3. As expected, given that the roll out of the L3 nationwide intervention for P4 is not scheduled until the 2016 school year, only 8.6% of P4 teachers reported receiving L3 technology. Teachers received cell phones, speakers, and SD cards at almost equal rates. Out of the teachers who received technology, the majority (82.0%) use technology at least once a week. Most teachers reported using technology two to four times a week. Results were generally consistent across grade and subject. Most of the teachers who never use technology in the classroom said this was a result of damage to the technology. A few teachers indicated that they had not been trained, and as such did not use the technology in the classroom. Type of Materials Range of number received* Avg. Number received Speakers 2 to 16 5 Cell phones 1 to 16 5 SD cards 1 to 16 5 *The number received was based on the number of teachers in the school teaching primary grades. TABLE 21. AVERAGE NUMBER OF TECHNOLOGY MATERIALS RECEIVED FROM L3 National Fluency Assessment of Rwandan Schools: Midline Report 2015 66 FIGURE 55. PERCENT OF TEACHERS WHO RECEIVED TECHNOLOGY FROM L3, BY GRADE (N=433) FIGURE 56. HOW OFTEN DO YOU USE THE TECHNOLOGY YOU RECEIVED FROM L3 TO TEACH LEARNERS THIS SUBJECT? (N=433) TEACHING PRACTICES AND BELIEFS When preparing for lessons, overall, three-quarters of sampled teachers report that they use L3 teacher guides. Curriculum documents and schemes of work provided by REB are used by almost half of teachers when developing lesson plans. Only a quarter of teachers utilize textbooks to prepare lesson plans. The use of curriculum documents and L3 guides varied, by grade. While P1, P2, and P3 teachers were similar in their use of materials to prepare lessons, as expected given that the L3 intervention has yet to be rolled out to P4, teachers in P4 classrooms differed quite a bit on how much they used curriculum documents and L3 guides. In P1, P2 and P3 classrooms, the majority (91.2%) of teachers largely utilized L3 teacher guides in preparing for lessons. P1, P2, and P3 teachers also used curriculum documents to prepare lessons, but to a lesser extent, in which 28.6% of P1, 45.0% of P2, and 29.5% of P3 teachers reported using curriculum documents. Conversely, the majority (83.8%) of P4 teachers use curriculum documents in preparing lessons plans; only 24.8% of P4 teachers used L3 teachers’ guides. Teachers’ use of schemes of work and textbooks were similar across all grade levels. FIGURE 57. DOCUMENTS TEACHERS USE WHEN PREPARING LESSON PLANS (N=433, MULTIPLE RESPONSE) 93.3% 89.2% 87.5% 8.6% P1 P2 P3 P4 8.6% 67.1% 6.3% 4.9% 13.2% Every day 2-4 times a week Once a week Less than once a week Never 75.1% 46.4% 43.9% 23.6% 0.5% L3 teachers' guides Curriculum documents Schemes of work provided by REB Textbooks None National Fluency Assessment of Rwandan Schools: Midline Report 2015 67 Most (57.4%) Kinyarwanda and English teachers said that it was sometimes not easy to teach reading. Roughly a third (33.5%) felt that teaching reading was “mostly easy” or “very easy.” Only 9.4% felt that teaching reading was “not easy at all.” Reading teachers in earlier grades (P1 and P2) reported feeling that it was slightly harder to teach reading than teachers in P3 and P4, largely because learners came into P3 and P4 with previous knowledge and skills in reading from earlier grades. For teachers that reported teaching reading was easy, teachers largely cited having enough learning materials (teacher’s guides, daily readers) as a key resource that made teaching reading easier. Several Kinyarwanda teachers remarked that they felt it was easier to teach reading in Kinyarwanda given that it was the mother tongue of their learners. Common reasons for reading being a difficult subject to teach included: When asked whether it was easier to teach boys or girls how to read, the vast majority (90.8%) said that there was no difference. A few teachers (6.5%) said it was easier to teach girls than boys, and even fewer (2.8%) said boys were easier to teach. These views were consistent when comparing the answers of male and female teachers. When asked to elaborate on their beliefs on teaching boys and girls to read, the majority of teachers said that they believed that boys and girls have the same level of ability to learn reading and that “all children are capable of learning to read.” A few teachers revealed gender biases. For instance, a few teachers remarked that boys were “not interested,” “not concentrated” and that girls were “more concentrated in learning” and “more willing.” Additional responses from teachers indicated that some learners faced gender barriers CHALLENGES TO TEACHING READING IN PRIMARY GRADES:  Learner absence  Learner lateness  Learners come to school hungry  Different levels of reading ability of learners in the classroom  Overcrowded classrooms/large class size  Lack of electricity  Learners’ parents are not engaged and do not participate in reading development of their children  Age differences between learners  Not enough books/materials  Learners did not attend pre-primary school “Little kids have the same capacity of understanding regardless of whether they are boys or girls.” -Teacher in Southern province National Fluency Assessment of Rwandan Schools: Midline Report 2015 68 to reading, not in the classroom, but at home; in which several teachers remarked that boys would drop-out to work for money and that some girls would miss school because they were needed to help out at home. L3 TRAINING AND SCHOOL-BASED MENTORS While 94.8% of P1, P2 and P3 teachers used L3 TLMs with their learners, only 56.0% had received L3 training. Teachers were asked whether they had any comments on L3 materials, training, or school-based mentors. The most common comments/suggestions from surveyed teachers include:  Additional trainings are needed for teachers on L3 program, materials and technologies.  Stories and reading texts are too long for their learners.  An increased number of L3 materials are needed, especially mathematics books.  Additional phones and speakers are required so that teachers do not have to share.  Damaged technology needs to be replaced.  Teachers suggested that a school-based mentor would be helpful in those schools that do not have them.  Teachers need support to improve their own English.  There is a need for sensistizing parents on their child’s education. Overall, teachers reported satisfaction with L3 program, including materials and training that they received. PARENTAL AND COMMUNITY INVOLVEMENT Overall, sampled schools reported that the majority of learners came from families that were “somewhat poor” (45.0%) or “of moderate means” (28.3%). The majority of schools indicated that parents were required to purchase various school supplies for their child to attend school, primarily pens/pencils, notebooks, uniforms and school bags. Only a quarter of surveyed schools said that parents were required to pay school fees, purchase books or pay for food for their child in order for their child to attend school. Head teachers also indicated that paying for these items was somewhat difficult for many parents. Of the sampled schools, 65% reported that families in their school found it “somewhat difficult” to pay for these school supplies; 11.7% reported that it was “extremely difficult” for families to afford these school “The L3 program is better in the process of teaching learning because it involves all learners in learning activities. It makes learners feel at ease in front of their teacher.” -Teacher in Eastern province National Fluency Assessment of Rwandan Schools: Midline Report 2015 69 supplies. Only about one in five schools (23.3%) felt that it was “easy” or “somewhat easy” for parents to afford the required school supplies/items. FIGURE 58. WHICH ITEMS MUST PARENTS PURCHASE FOR A LEARNER TO ATTEND SCHOOL? (N=60) All schools in the sample had a parent teacher association (PTA/PTC). Schools that are within Concern Worldwide operation area (Southern Province and Two Districts of Northern Province) reported that members from the PTA had been trained by Concern. On average, for schools that had PTA members trained by Concern, between 3 and 4 PTA members that attended the training. Three quarters of PTA members who had received training trained other PTA members. FIGURE 59. PTA MEMBERS TRAINED BY CONCERN WORLDWIDE Has the PTA been trained by Concern Worldwide? (n=60) Did the PTA members (who attended the training facilitated by Concern) train others PTA members? (n=17) Nearly three-quarters (73.3%) of schools reported that their PTA/PTC had an action plan at midline, compared to roughly half (51.7%) of schools at baseline. Four out of ten schools 98.3% 98.3% 95.0% 83.3% 25.0% 23.3% 21.7% 6.7% Pens/pencils Notebooks Uniform School bag School Fees Food to eat at school Books Tuition Yes, 28.3% No, 71.7% 76.5% 23.5% Yes No National Fluency Assessment of Rwandan Schools: Midline Report 2015 70 (41.7%) reported that PTA/PTCs have undertaken initiatives to support teacher motivation in their schools. The major ways PTA/PTCs support teacher motivation include:  Providing lunch and tea break for teachers at school; and  Providing financial incentives and awards to best performing teachers. Slightly less PTAs/PTCs (26.7%) have undertaken initiatives to support literacy and equity in education in their schools. Examples of initiatives include:  Reading or writing competitions;  Creating reading clubs to assist children in reading;  Encouraging learners to read and go to the library;  Sensitizing parents on gender equality;  Mobilizing parents to read with their children at home;  Providing school materials to children (boys and girls). On average, head teachers reported inviting parents/caregivers to the school between two to three times a year. When asked how many parent or caregivers come to the school when invited, head teachers resoundingly reported high participation of parents/caregivers. In fact, half of surveyed head teachers indicated that “most” parents/caregivers come to the school when they are invited; another 44% reported that a “moderate amount” of parents/caregivers come. Only three of the sampled schools reported that a “few” parents/caregivers attend. Similarly, nearly two-thirds (65.5%) of surveyed teachers reported that learners’ parents or caregivers usually came to talk to them at least once per semester. Almost a third (28.4%) of parents in some provinces, however, never came to talk to their child’s teacher. “We need collaboration with parent of learners.” -Teacher in Southern province National Fluency Assessment of Rwandan Schools: Midline Report 2015 71 LEARNER CONTEXT INTERVIEW FINDINGS It is widely recognized in the field of education that such contextual factors as home environment that supports learning, adequate nutrition, and early exposure to literacy play a prominent role in helping children succeed academically.33 Additionally, such school factors as teachers assigning homework or teachers reading to children have also been found to be associated with improved performance. To better understand which of these potential moderators seems to be particularly influential in explaining variance in learner performance in Rwanda, L3 assessment team developed a learner interview questionnaire. The intent behind the questionnaire was to gather background information about the child’s life and experiences that have direct relevance to his or her competencies in literacy and math. The following questions were included in the interview: Home Environment 1. What language do you speak at home? 2. At home, does someone read stories to you? 3. Do you see your mother (or main caregiver) reading books or newspapers? 4. How often do you miss school? 5. How often are you late for school? 5a. Why are you missing school or late for school? 6. Have you or any of your siblings ever repeated a grade? 7. At home, which of the following do your parents expect you to do regularly? (Help with household chores, go to market, study, etc.) 8. Do your parents/caregivers want you to go to school every day? 9. Do your parents/caregivers check your homework? School/Teacher 10. What do you like about school? 11. What do you NOT like about school? 12. Is this your first year in this grade? 13. Does your mathematics teacher check your work that you do in class? 33 Park, H. (2008). Home literacy environments and children's reading performance: A comparative study of 25 countries. Educational Research and Evaluation, 14(6), 489–505. 2008. “Reading Achievement: International Perspectives from IEA's Progress in International Reading Literacy Studies (PIRLS)”, Special Issue in Educational Research and Evaluation: An International Journal on Theory and Practice, Vol. 14, Issue 6, 2008. Fan, Xitao and Chen, Michael. 2001. “Parental Involvement and Learners' Academic Achievement: A Meta-Analysis” in Education Psychology Review. March 2001, Volume 13, Issue 1, pp 1-22. Bus, Adriana G., Van IJzendoorn, Marinus H. and Pellegrini, Anthony D. 1995. “Joint Book Reading Makes for Success in Learning to Read: A Meta-Analysis on Intergenerational Transmission of Literacy”. Review of Educational Research, Spring 1995 vol. 65 no. 1 1-21. National Fluency Assessment of Rwandan Schools: Midline Report 2015 72 14. Does your mathematics teacher check/mark your homework? 15. Does your Kinyarwanda teacher check your work that you do in class? 16. Does your Kinyarwanda teacher check/mark your homework? 17. Do you ask questions when you do not understand something? 18. At school, can you choose which stories to read? 19. Are you allowed to take books home from school? 20. Do you ever take books from school to read at home? Socio-Economic Factors 21. Did you have something to drink today, like water, tea, milk or juice? 22. Did you have something to eat today, like potatoes, rice, bread or beans? 23. Do you have radio or cell phone at home? 24. What light do you have at home? (Candles, electric lamp, solar panel lamp, etc.) 25. Does anyone at your house have a bicycle, a motorcycle or a car? During the analysis, a composite variable was created for each subsection of the interview and used in multivariate analyses to help explain variations in learner performance on literacy and mathematics tasks. Both the composites and the individual items were also used in the multivariate analyses in the relevant sections of the report. Home Environment. The majority of learners reported that they speak only Kinyarwanda at home. Only a few learners spoke French, English or Kiswahili at home. Some learners in the Northern province spoke Urukiga, a northern variant of Kinyarwanda. However, the numbers of learners speaking those languages was very small in the sample and did not allow for any meaningful comparisons. More than two thirds of learners said that a caregiver at home reads stories to them (70.4%). P2, P3, and P4 learners had about equal numbers of learners being read to at home: 72.9%, 71.9% and 71.0%, respectively. Only 65.6% of P1 learners said they were read to at home. In P2 and P3, more girls reported being read to at home than boys did, while grades P1 and P4 little difference was seen between boys and girls. However, accuracy of this self-reported data could not be confirmed with parents since parent interviews were not part of the study. National Fluency Assessment of Rwandan Schools: Midline Report 2015 73 FIGURE 60. HOME ENVIRONMENT INTERVIEW RESULTS (N=2,405) Learners’ self-reported frequency of being late or absent were consistent across sex, but showed slight variation across grades, with P3 learners reporting more lateness and absences than the other grades. The learners’ reasons for being late or absent were also consistent among boys and girls. There was a strong relationship between the reported frequency of being late and the reported frequency of missing school, Pearson’s correlation .293, statistically significant at p<.001. FIGURE 61. HOW OFTEN DO YOU MISS SCHOOL OR ARE LATE? (N=2,405) For those learners who mentioned that they were late for school frequently, the most cited reasons for being late or absent were needing to do chores (68.4%) and waiting to eat (25.7%). The majority (85.9%) of learners reported that they were expected by their parents to help with household chores regularly. Nearly two-thirds (60.2%) said that their parents expected them to study regularly. Several learners also mentioned that they were expected to clean, collect firewood, fetch water, look after livestock and prepare food. Results were fairly consistent for both boys and girls with the exception of helping with the other children in the family in which significantly (p<.001) more girls were expected to do this than boys. Yes, 70.4% Yes, 72.4% Yes, 97.7% Yes, 78.3% Yes, 55.6% At home, does someone read stories to you? Do you see your mother/caregiver reading books or newspapers? Do your parents/caregivers want you to go to school every day? Do your parents/caregivers check your homework some or all the time? Have you or any of your siblings ever repeated a grade? A lot, 1% A lot, 3% Some￾times 11% Sometimes 26% Rarely 40% Rarely 26% Never, 48% Never, 38% How often do you miss school? How often are you late? a lot or sometimes rarely or never National Fluency Assessment of Rwandan Schools: Midline Report 2015 74 FIGURE 62. WHICH OF THE FOLLOWING DO YOUR PARENTS EXPECT YOU TO DO REGULARLY? (N=2,405, MULTIPLE RESPONSE) School and Teacher. Learners overwhelmingly reported that mathematics and Kinyarwanda teachers check their homework and in-class work. The study did not collect data on the content of the work or what “checking homework” entails. Overall, 70.7% of learners report that when they do not understand something, they ask a teacher; 22.8% ask a peer. Only 6.4% do not ask anyone for help. Nearly two thirds (64.4%) of learners are able to choose which stories to read at school. The majority (80.9%) of children are allowed to take books from school to read at home, and out of these learners, 95.4% take the opportunity to do so. FIGURE 63. SCHOOL/TEACHER FACTORS INTERVIEW RESULTS (N = 2,405) Learners were asked what they liked about school. The most common responses from learners was that they enjoyed studying Kinyarwanda, math, and English and playing. Only two sampled learners said that they did not like anything about school. 85.9% 60.2% 15.3% 12.2% 11.9% 11.3% Help with household chores Study Help with other children in the family Go work in the field Other Go to market Yes, 64.4% Yes, 80.9% Yes, 95.4% At school, can you choose which stories to read? Are you allowed to take books home from school? If you are allowed to, do you ever take books from school to read at home? National Fluency Assessment of Rwandan Schools: Midline Report 2015 75 FIGURE 64. WHAT DO YOU LIKE ABOUT SCHOOL? (N=2,405, MULTIPLE RESPONSE) In terms of what learners did not like about school, the most common response was that they did not like fighting and abuse by other learners. Nearly a third of sampled learners responded that there wasn’t anything that they didn’t like about school. Other common reasons why learners didn’t like school were disturbances in class by learners and indiscipline of some learners. FIGURE 65. WHAT DO YOU NOT LIKE ABOUT SCHOOL? (N=2,405, MULTIPLE RESPONSE) 71% 67% 55% 50% 30% 17% 12% 12% 11% 9% 1% 1% 10% Studying Kinyarwanda Studying Math Playing Studying English Studying science Reading books I like how we are taught by our teachers Studying French Writing Being with other students School environment Everything Other 36% 34% 23% 20% 14% 9% 8% 7% 6% 4% 4% 4% Fighting and abuse by other students Nothing Disturbances in class by students Indiscipline of some students Disputes among children Do not like studying English Corporal punishment Other Do not like studying some lessons Do not like studying Math Unclean school environment Do not like studying Kinyarwanda National Fluency Assessment of Rwandan Schools: Midline Report 2015 76 Socio-Economic Status. To better understand the socio-economic context of learners, we asked five simple questions that together serve as a proxy for a learners’ family wealth. Most learners (89.1%) reported having a radio or cell phone at home. The majority of learners also reported having eaten before they came to school that day (72.3%) and more than half (57.4%) reported having something to drink, but of concern is that almost a third of learners who came to school hungry. Poor nutrition is one of the key factors impeding learning, as international research shows. Learners were also asked whether someone in their family owns a means of transportation such as a bike, a motorcycle or a car, 35.1% of learners reported that they did. FIGURE 66. SOCIO-ECONOMIC STATUS INTERVIEW RESULTS (N = 2,405) To better understand lighting conditions of learners’ home that might impact their ability to do homework, learners were asked what type of light they used at home. Few learners (11.7%) said they used candles at home, and a third of learners (31.4%) used electric lights. Another third of learners (33.0%) answered they used some other type of light than was provided on the interview, and when writing in responses, most of this group reported using flashlights (including cell phone flashlights). FIGURE 67. WHAT LIGHT DO YOU HAVE AT HOME? (N=2,405) Yes, 89.1% Yes, 72.3% Yes, 57.4% Yes, 35.1% Do you have radio or cell phone at home? Did you have something to eat today, like potatoes, rice, bread or beans? Did you have something to drink today (like water, tea, milk or juice)? Does anyone at your house have a bicycle/motocycle or a car? 33.7% 30.4% 18.5% 11.7% 5.2% 0.5% Electric lamp/Amashanyarazi Flashlight/itoroshi Paraffin lamp/Itara rya peterore Candles/buji Solar panel lamp/Ingufu z’izuba Other/Ikindi National Fluency Assessment of Rwandan Schools: Midline Report 2015 77 Composite 34 variables were created for each of the section of the questionnaire (home environment, school and teacher, and socio-economic status). Additionally, a composite variable was created for all the risk factors that could impact learning. These risk factors included the following: child reporting he/she does not see mother read; missing school often; late for school often; self or sibling repeating grade; parents not expecting child to go to school every day; parents not checking homework; disliking school; and not having anything to eat or drink before coming to school; All four composite variables correlate35 with each other, as shown in next table. TABLE 22. CORRELATIONS BETWEEN SECTIONS OF SURVEY RESULTS Context interview composite Home environment School and teacher Socio-economic Home environment 1 School and teacher .281*** 1 Socio-economic .185*** .134*** 1 Risk factors -.406*** -.092*** -.205*** *** Correlations are significant at the .01 level (2-tailed) The composite variables as well as specific measures from the context interview were included in bivariate and multivariate analyses in the relevant sections of this report to better understand variation in learner scores. Summary. The study found that most schools fall within the range between poor and adequate infrastructure. About a third of study schools reported receiving support from local or international organizations/NGOs, mostly in the form of provision of teaching and learning materials, teacher training or infrastructure. Schools were found to have large classrooms, particularly in earlier grades: average learner to teacher ratio in P1 was found to be 66 learners to one teacher, 58 learners to a teacher in P2, and 54 learners to a teacher in P3. Many learners were found not to attend regularly; on the day of the assessment, about a third of a class was not found to be present, on average. Teachers reported that overcrowding was an important barrier to effective teaching. On average, teachers reported that 15.6% of learners in their classrooms were repeaters. 34 “Composite” is a score created by adding data across multiple variables, when each of the variables is expressed in binary terms (e.g., “yes” = 1 and “no” = 0). 35 “Correlation” is a statistical term that describes a degree of relationship between two variables. Two variables are considered correlated when a change in one is associated with a change in another. Correlation does not presume causation since a change in both variables might be caused by the third variable. Correlation coefficient ranges between 0 and 1, with higher value denoting stronger relationship. Correlations in a range between .1 and .3 are considered rather weak, in a range between .4 and .6 are considered moderate, and above .6 are considered strong. National Fluency Assessment of Rwandan Schools: Midline Report 2015 78 Teachers reported that learner lateness and absenteeism were the main challenges to teaching both reading and math. In many schools, teachers reported that learners come to school hungry. This is confirmed by the learner interview which found that 28% of learners did not have anything to eat before coming to school on the day of the assessment. The review of teacher attendance records also revealed a high level of teacher absenteeism: on average, on a given day, 7.5% of all P1, P2, P3 teachers were absent. The percent of absent teachers was found to be much higher in more remotely located schools. In school with higher teacher absenteeism learner achievement was found to be lower. Observed conditions in sampled classrooms showed that the majority of classrooms were in adequate or good condition with respect to blackboards, clean classroom space, good lighting, desks for learners, and reading and writing materials for learners. In more than half of observed schools print materials (posters, signs, etc.) were observed on school or classroom walls. Teachers reported having received materials and instructional technology from L3. Teaching and learning materials were observed in use by teachers; in 18% of schools they were found in the library. L3-provided learner books were also observed in schools. In the majority (76%) of observed schools learner books were observed in use by learners or on the classroom shelves. In a few schools, learner books were found in the library or the headmaster’s office. In nearly all schools, learner books looked used. Teachers reported having received cell phones, speakers, and SD cards from L3; the majority said they use technology at least once a week. Most teachers reported using technology two to four times a week. National Fluency Assessment of Rwandan Schools: Midline Report 2015 79 RECOMMENDATIONS Review and revision of system-level policies that impact the process of instruction, learner and teacher attendance, learner repetition, teacher class assignments, among other, could have a positive impact on overall learner achievement. Specifically, the review/revision of policies relating to the following issues is recommended: - At the heart of the Rwandan curriculum lies an explicit emphasis on the development of skills and competences for lifelong learning and for operating effectively in society. The curriculum also promotes formative assessment to monitor learner progress and make appropriate instructional decisions. Currently early grades do not have class teachers, only subject teachers. During data colleaction, teachers and data collectors observed that there is little or no time to gather, analyse and use assessment information to improve learning and inform planning. This inhibits teachers’ ability to get to know their pupils personally, differentiate appropriately, as well as the effectiveness of instructional practice. An approach where a single teacher teaches all subjects to an assigned class is recommended. This would enable teachers to provide a broad and balanced curriculum for all learners and allow them to frequently integrate formative assessments during typical daily activities. - Teacher and learner absenteeism was found to be associated with poor academic results. An important finding of the assessment is that the farther the school is from the District Office, the more likely the teachers and the learners to not come to school. It is recommended that policies be put in place to counteract this trend. - Grade repetition contributes to overcrowding early grade classrooms. Providing class teachers with professional development programmes to effectively implement continuous assessments and remediation activities is recommended. This would enable teachers to differentiate instruction effectively, given large class sizes, provide remediation for struggling learners and support these learners to catch up and progress with the rest of their peers. - Annual competence-based assessments would provide consistent information on learner performance on key indicators, such as grade-level reading procedural mathematics fluency. Data on learner achievement, class size, repetition and other central issues in education should be used to inform policy. Finally, the proposed Teacher Development and Management Policy, November 2015 prioritizes the development and implementation of a well-structured programme of Continuous Professional Development for teachers to improve all aspects of the quality of education, especially in literacy, numeracy and essential life skills. Ensuring that learners achieve the foundations of learning in the early grades is a vital way of overcoming early disadvantage; well-trained teachers are key to improving pupils’ early learning. In order to National Fluency Assessment of Rwandan Schools: Midline Report 2015 80 achieve these objectives an explicit focus on early grade instruction and recognition of its importance as a foundation for life-long learning should be incorporated into pre-service and inservice teacher training programmes. Public awareness campaigns promoting the importance of parental and community involvement in the development of early grade literacy and numeracy competences should also be considered. Page A81 APPENDIX A: METHODOLOGY ASSESSMENT QUESTIONS The assessments were developed by a team of experts from the REB and L3 and are based on a) international standards for testing and measuring learners’ oral reading fluency in the early grades, and b) on existing grade level standards in literacy and mathematics. The assessments were extensively piloted through a number of pilot activities. The first pilot activity took place in March of 2014 with a sample of 1,237 learners randomly selected from 62 schools from all districts in Rwanda. The results were documented in a detailed report36; after the initial pilot the assessment team made appropriate adjustments and revised tools which were piloted again in July of 2014. All revisions were made in close collaboration with REB literacy and mathematics experts. The finalized assessment tools were used in the baseline assessment that took place in October of 2014. Assessment items for the midline and endline assessments of P2 and P3 were piloted in October of 2014 with a sample of 664 learners from 12 schools. Assessment items for P4 were piloted in of 2015 with a sample of 240 learners randomly drawn from four schools. All assessment items for school-level longitudinal comparisons have been equated using linear equating method. Since 2012, the REB and L3 worked closely to create national reading performance standards for primary grades 3 and 5. A national assessment of P3 and P5 to validate those standards was conducted at the end of the 2012 school year. In 2014, this work continued with proposing reading standards for Primary 2 (P2) and validating them through national sample-based testing. The present report presents learner achievement data collected at the end of 2015 school year utilizing these recently established national reading performance standards. The assessment had the following main objectives: 4. Impact evaluation of L3 project37: a. Document changes in P1, P2 and P3 learner achievement in oral reading fluency against established benchmarks, and in mathematics on grade-level procedural fluency tasks after one year of national implementation of L3 intervention. b. Collect baseline data in oral reading fluency in Kinyarwanda, English and in procedural fluency in mathematics for P4 learners. 36 National Fluency and Mathematics Assessment Report, September 2014. Prepared by EDC for USAID. 37 The impact evaluation is designed based on the principles of the impact attribution articulated in USAID Evaluation Policy (2011), and recommendations in the Technical Notes of the Education Strategy (2012, 2015). The counterfactual for the project impact is captured through the baseline conducted with the nationally representative sample of primary schools in October of 2014. Methodology Page A-82 5. Investigation of factors impacting learner achievement: a. Analyze variance in learner achievement using school-level data such as Parent Teacher Association (PTA) involvement, and classroom-level data such as teacher background characteristics, using instructional technology, teaching experience, etc. b. Analyze variance in learner achievement using learner background characteristics, such as parental literacy, support with homework at home, etc. c. Analyze learner achievement among learners who repeated the grade to establish to what extent repeating a grade positively impacted learner achievement. 6. Development of recommendations for L3 and REB with regard to support systems needed to accelerate improvements in learner achievement. Recommendations will inform L3 activities in the final year of its implementation. In addition to these L3-related objectives, the assessment also provides an opportunity to begin conversations about how nation-wide periodic assessments based on international standards can inform Education Sector Planning. L3 is adhering to the following assessment schedule: TABLE 23.SCHEDULE OF NATIONAL ASSESSMENTS USING FARS/MARS Grade Schedule of testing 2014 Sept/Oct 2015 Sept/Oct 2016 Sept/Oct P1 Baseline Midline Endline P2 Baseline Midline Endline P3 Baseline Midline Endline P4 n/a Baseline Endline *P4 baseline does not occur till the end of 2015 since L3 is scheduled to roll out P4 intervention in SY2016. FLUENCY ASSESSMENT OF RWANDA SCHOOLS (FARS) TOOLS The REB and L3 experts worked collaboratively to develop reading tests for P1, P2, P3 and P4 reading assessment. The developed tests reflected emerging national standards for reading in the first three primary grades, since the testing was taking place at the end of the school year. The criteria used for the test development included the text genre, text structure, vocabulary, sentence structure, letter-sound combinations, length, content, and the type of comprehension questions. The development process involved three stages: Methodology Page A-83 1. Convening of the expert group to develop three passages appropriate for the end of the Primary 1, Primary 2 and Primary 3 with 5 comprehension questions each (July 2014) 2. Pilot test the three passages with a sample of learners (July-August 2014) 3. Based on the results of the pilot test, select the final text (August 2014). Summary of the text development criteria is provided in Table 24, and the full sets of criteria can be found in Table 28 at the end of the Methodology section. The end of Primary 1 was considered to correspond to Level 4, and the end of Primary 2 was considered to correspond to Level 9. Criteria for these levels were used to develop the reading tests. TABLE 24. CRITERIA FOR DEVELOPING TEXTS IN KINYARWANDA Criteria P1 (Level 4) P2 (Level 9) Genre Very simple narrative, familiar single theme; simple topic Simple narrative; familiar themes; single idea or simple topic Vocabulary Familiar words Familiar vocabulary; 1-3 syllables; nearly all high frequency words Sentence structure Simple sentence structure, short sentences, present tense. Simple sentence structure; short and long sentences; present tense Length (target) 15 words 35 words Content Simple structure; literal information Simple structure; literal information Comprehension questions 5 questions; literal questions, i.e., who, what, when, where, why 5 questions; literal questions, i.e., who, what, when, where, why The timed portion of the FARS was timed at 60 seconds for the reading portion which was followed by five comprehension questions. This was followed by the extended FARS which was capped at 3 minutes. MATHEMATICS ASSESSMENT OF RWANDA SCHOOLS (MARS) TOOLS MARS was developed and pilot-tested by L3 staff with the technical support from EDC mathematics experts prior to the national baseline conducted in 2014. The tests were based on the results of an analysis of the Rwanda curriculum framework, mathematics teacher guides, and learning materials for the primary grades, to determine which mathematical concepts were pivotal for each grade. Selected concepts were then cross-referenced with the research-based international standards of teaching mathematics in early grades. Each subtest included 10 tasks; each subtest was timed at 120 seconds at P1, P2 and P3, and 60 seconds at P4, for the entire MARS not to exceed 10 minutes in administration, per child, including introduction and conclusion. A reliability analysis of the MARS showed a strong reliability for all four MARS subtests. The subtest with the lowest item-total correlation was P4 Subtest 5 (“Comparing Numbers”) which Methodology Page A-84 had the correlation coefficient of .115. This result shows that students who are proficient in adding, subtracting, dividing and multiplying do not have the same level of proficiency when it comes to comparing fractions and decimal numbers, which is a competency they are supposed to acquire in Primary 4. However, other P4 MARS subtests as well as subtests in other grades relate strongly to each other. TABLE 25. MARS RELIABILITY ANALYSIS Subtests MARS P1 MARS P2 Item-Total Correlation Alpha if Item is Deleted Item-Total Correlation Alpha if Item is Deleted Subtest 1 .798 .799 .742 .618 Subtest 2 .814 .783 .766 .583 Subtest 3 .690 .892 .496 .871 Cronbach’s alpha .879 .799 Subtests MARS P3 MARS P4 Item-Total Correlation Alpha if Item is Deleted Item-Total Correlation Alpha if Item is Deleted Subtest 1 .688 .796 .614 .685 Subtest 2 .586 .842 .646 .665 Subtest 3 .721 .783 .665 .660 Subtest 4 .726 .781 .613 .678 Subtest 5* Not included .115 .829 Cronbach’s alpha .842 .757 * Grades P1 and P2 MARS test included only three subtests; MARS P3 test included four and P4 included five subtests. Table 26 summarizes the subtests and how they are presented in the report. TABLE 26. FARS AND MARS SUBTESTS P1 Test Tasks # Description (Instrument) Tasks Max. Pts. Timed FARS 1A Oral Reading Fluency 27-word passage 27 Yes (60 sec.) 1B Reading Comprehension 5 questions 5 No (without text) 1C Reading Comprehension 5 questions 5 No (with text, after extended time reading) MARS 1 Adding Numbers 10 equations 10 Yes (120 sec.) 2 Subtracting Numbers 10 equations 10 Yes (120 sec.) 3 Comparing Numbers 10 equations 10 Yes (120 sec.) P2 Test Tasks # Description (Instrument) Tasks Max. Pts. Timed FARS Methodology Page A-85 1A Oral Reading Fluency 41-word passage 41 (equated to baseline 42) Yes (60 sec.) 1B Reading Comprehension 5 questions 5 No (without text) 1C Reading Comprehension 5 questions 5 No (with text, after extended time reading) MARS 1 Adding Numbers 10 equations 10 Yes (120 sec.) 2 Subtracting Numbers 10 equations 10 Yes (120 sec.) 3 Multiplying Numbers 10 equations 10 Yes (120 sec.) P3 Test Tasks # Description (Instrument) Tasks Max. Pts. Timed FARS 1A Oral Reading Fluency 57-word passage 57 (equated to baseline 58) Yes (60 sec.) 1B Reading Comprehension 6 questions 6 No (without text) 1C Reading Comprehension 5 questions 5 No (with text, after extended time reading) MARS 1 Multiplying Numbers 10 equations 10 Yes (120 sec.) 2 Dividing Numbers 10 equations 10 Yes (120 sec.) 3 Adding Numbers 10 equations 10 Yes (120 sec.) 4 Subtracting Numbers 10 equations 10 Yes (120 sec.) P4 Test Tasks # Description (Instrument) Tasks Max. Pts. Timed FARS Kinyarwanda 1A Oral Reading Fluency 66-word passage 66 Yes (60 sec.) 1B Reading Comprehension 5 questions 5 No (without text) 1C Reading Comprehension 5 questions 5 No (with text, after extended time reading) FARS English 1A Oral Reading Fluency 56-word passage 56 Yes (60 sec.) 1B Reading Comprehension 5 questions 5 No (without text) 1C Reading Comprehension 5 questions 5 No (with text, after extended time reading) MARS 1 Adding Numbers 10 equations 10 Yes (60 sec.) 2 Subtracting Numbers 10 equations 10 Yes (60 sec.) 3 Multiplying Numbers 10 equations 10 Yes (60 sec.) 4 Dividing Numbers 10 equations 10 Yes (60 sec.) 5 Comparing Numbers 10 numbers 10 Yes (60 sec.) Methodology Page A-86 DATA ANALYSIS Collecting data electronically eliminated the need for data entry. L3 M&E staff cleaned the data sets and conducted appropriate statistical analyses, including frequency distributions, comparisons of means and multivariate statistical analysis (regression). For the analysis of the FARS data, we used the words correct per minute (wcpm) score as the main fluency measure which was calculated as follows: WCPM = (Words Read Correctly / Number of Seconds Used) x 60 For instance, if a student read 10 words correctly from the text and used 30 out of the 60 seconds, their rate would be 20 correct words per minute: WCPM = (10 / 30) x 60 = 20. MARS data analysis is presented both by section and by total scores across the sections. Results from the student context survey and the demographics section were used for the multivariate analysis of student-level results. Composite variables were constructed for each of the three sections of the context survey and used in the multivariate analysis. SAMPLING PARAMETERS The sampling approach followed random clustered sampling method to obtain a nationally representative sample of non-private schools (public or government-aided schools only). The sample was determined based on the following assumptions:  Type of analysis: logistic regression  Alpha (probability of Type I error): .05/2 = .025. Alpha is divided by two because two separate measures are used by the test (fluency and comprehension)  Power (probability of Type II error): 0.9, or 90 percent  Expected effect size: 0.3 (moderate)  Expected inter-class correlation (ICC, or roh): 0.1 Using Optimal Design cluster sampling software, the following sample size was computed:  Number of clusters (schools) = 60  Cluster size (number of students in a school, per each grade, per each gender) = 5 randomly selected boys and 5 randomly selected girls, 10 students in each grade, 40 students in each school. Total sample size for each grade: 600 students. Total number of students: 2,400 in four grades. SAMPLING PROCESS The list of all government schools in Rwanda (“sampling frame”) was obtained by L3 from REB in 2012. Early in 2014 the decision was made in collaboration with REB and USAID that the sample would be stratified by school district. Therefore, the key parameter used in drawing Methodology Page A-87 the sample was the school district as the stratification variable. To compensate for the difference in the number of schools in each district, weights were applied during the data analysis. The sampling of schools was conducted randomly (not targeted) and is fully representative of the universe of Rwandan schools. Sampling was conducted by L3 M&E advisor in July of 2014 using the following inputs: 1. Sampling frame: list of all government schools in Rwanda. The list was obtained from REB by L3 staff in Excel format and was imported by Dr. Vinogradova into SPSS. 2. Sampling stratification: school district. Two schools per district were sample. The sample was drawn by software without any human interference using the sampling frame provided by REB. The sample was drawn using the Complex Samples module of Statistical Package for the Social Sciences (SPSS). The selection of schools was conducted by the software from the sampling frame, based on the specified criteria (stratification levels). The L3 assessment team did not know the condition of the schools that were randomly selected by the software, nor were any technical staff involved in the selection process. The sample by province and district is shown below. TABLE 27. NUMBER OF SAMPLED LEARNERS BY DISTRICT (2 SCHOOLS PER DISTRICT) Province District Learners P1 P2 P3 P4 TOTAL Eastern Bugesera 20 20 20 20 80 Gatsibo 20 19 20 20 79 Kayonza 20 21 20 21 82 Kirehe 20 19 20 19 78 Ngoma 20 20 20 20 80 Nyagatare 20 20 20 20 80 Rwamagana 20 20 20 20 80 Kigali City Gasabo 20 21 20 20 81 Kicukiro 20 20 20 20 80 Nyarugenge 21 21 21 20 83 Northern Burera 21 20 21 20 82 Gakenke 20 20 21 21 82 Gicumbi 20 20 20 20 80 Musanze 20 20 20 20 80 Rulindo 20 19 20 20 79 Southern Gisagara 20 20 20 20 80 Huye 20 20 20 20 80 Kamonyi 20 20 20 20 80 Muhanga 20 20 20 20 80 Nyamagabe 20 20 19 20 79 Nyanza 20 20 20 20 80 Methodology Page A-88 Nyaruguru 22 20 20 20 82 Ruhango 20 20 20 20 80 Western Karongi 20 20 20 20 80 Ngororero 20 20 20 20 80 Nyabihu 20 20 20 20 80 Nyamasheke 20 20 20 21 81 Rubavu 20 20 20 20 80 Rusizi 20 20 18 19 77 Rutsiro 20 20 20 20 80 TOTAL 604 600 600 601 2405 LIMITATIONS OF DESIGN The assessment had some limitations. In cross-sectional designs, major threats to validity38 involve selection-history (when other events occur between cohorts that may impact one group but not the other), selection-instrumentation (when the test used with cohorts is slightly different), and selection-mortality (when there is a different rate of dropout in different tested cohorts, for whatever reason). While it is possible to control for the selection-instrumentation bias by extensive pilot testing, other two threats relate to the passage of time and external events outside of control or knowledge of the study team. It is therefore unknown to what extent external factors may impact different cohorts. Other limitations originate from the assessment’s sampling strategy. First, the sample size was designed to provide national estimates of literacy and mathematics achievement of P1, P2 and P3 students. While the sample was stratified by district to ensure adequate representation of students from all regions of the country, the province-level or district-level sub-samples are not large enough to be treated as separate samples. A much larger sample size would be required to enable such analyses. Finally, a limitation was the timing of the assessment. While the baseline assessment was conducted in September of 2014, the midline assessment was conducted in October, almost a full month later. It is likely that the results are slightly skewed toward higher scores due to this difference. 38 W. Trochim, Research Methods Knowledge Base. Cornell University, 2006. Page A89 TABLE 28. CRITERIA FOR FARS DEVELOPMENT: CHARACTERISTICS OF TEXTS ACCORDING TO THE LEVEL OF DIFFICULTY Level Presentation of the text Illustrations Length Choice of words Syntax Style 0  Simple presentation  Writing is separate from illustration  Text on 1 page, illustration on the other  Text is always in the same place on the page  Direct link between the text and the illustration  Each idea is illustrated  There is more space for the illustration than for the text  1 to 3 words per line  1 line per page  16 to 24 words  About 8 pages  Only familiar and frequent words  Simple vocabulary that is frequently used orally  Word or group of words  Predictable structure  Repetitive structure (pattern book)  A single idea is presented  Lists of things or actions 1  Simple presentation  Writing is separate from illustration  Text is always in the same place on the page  Direct link between the text and the illustration  Each idea is illustrated  About 8 pages  2 to 5 words per line  1 to 2 lines per page  21 to 40 words  About 8 pages  Only familiar and frequent words  Simple vocabulary that is frequently used orally  Declarative sentences  Simple sentences (S-V-C)  Verbs are in the simple present tense  Predictable structure  Repetitive structure (pattern book)  A single idea is presented 2  Simple presentation  Writing and illustrations are sometimes on the same page  Text is always in the same place on the page  Direct link between the text and the illustration  Each idea is illustrated  There is more space for the illustration than for the text  3 to 8 words per line  1 to 2 lines per page  30 to 55 words  About 8 pages  Only familiar and frequent words  Simple vocabulary that is frequently used orally  Declarative and/or exclamatory sentences  Simple sentences (S-V-C)  Verbs are in the simple present tense of the indicative or the imperative  Predictable structure  Repetitive structure (pattern book)  Story structure (beginning, middle and end)  Narrative or informative text  Some dialogues 3  Simple presentation  Writing and illustrations are sometimes on the same page  Text is always in the same place on the page  Direct link between the text and the illustration  Each idea is illustrated  There is more space for the illustration than for the text  5 to 8 words per sentence  1 to 2 lines per page  50 to 80 words  About 8 pages  Only familiar and frequent words  Simple vocabulary that is frequently used orally  Declarative and/or exlamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the simple present tense of the indicative or the imperative  Predictable structure  Repetitive structure (pattern book)  Story structure (beginning, middle and end)  Narrative or informative text  Dialogues 4  Simple presentation  Writing and illustrations are sometimes on the same page  Text location may vary  Sentences continue one more than one line  Each new sentence begins on a new line  Direct link between the text and the illustration  More than one idea or action is contained in the illustration  There is more space for the illustration than for the text  5 to 8 words per sentence  2 to 3 lines per page  75to 100 words  About 8 to12 pages  Mostly familiar and frequent words  Simple vocabulary  Text includes 1 to 3 new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the simple present tense of the  Story structure (beginning, middle and end)  Only one theme is presented  Narrative or informative text  Dialogues Data Collection Page A-90 Level Presentation of the text Illustrations Length Choice of words Syntax Style indicative or the imperative  There may be verbs on the present continuous 5  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Sentences continue one more than one line  Each new sentence begins on a new line  Direct link between the text and the illustration  More than one idea or action is contained in the illustration  There is more space for the illustration than for the text  5 to 11 words per sentence  2 to 5 lines per page  75to 130 words  About 8 to12 pages  Mostly familiar and frequent words  Simple vocabulary  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the simple present or continuous present tense of the indicative or the imperative  There can be verbs in the past or the future tense  Story structure (beginning, middle and end)  Only one theme is presented with several events  Narrative or informative text  Dialogues 6  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  The illustration supports the text  The illustration takes up several ideas in the text  The meaning of the story is more in the text than in the illustration  2 to 12 words per sentence  3 to 5 lines per page  130 to 180 words  About 8 to16 pages  Mostly familiar and frequent words  Simple vocabulary  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the simple present, present continuous, past and/or future tense of the indicative or present of the imperative   Story structure (beginning, middle and end)  Only one theme is presented with several events  Narrative or informative text  Dialogues 7  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  The illustration supports the text  The illustration takes up several ideas in the text  The meaning of the story is more in the text than in the illustration  4 to 12 words per sentence  3 to 8 lines per page  120 to 200 words  About 8 to16 pages  Mostly familiar and frequent words  Simple vocabulary  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative   Story structure (beginning, middle and end)  Only one theme is presented with several events  Narrative or informative text  Dialogues Data Collection Page A-91 Level Presentation of the text Illustrations Length Choice of words Syntax Style 8  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  The illustration supports the text but only in part  The illustration takes up several ideas in the text  The meaning of the story is more in the text than in the illustration  Average 7to 8 words per sentence  4 to 9 lines per page  180 to 270 words  About 8 to16 pages  Some familiar and frequent words  Some vocabulary is a little more complex  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative   Story structure (beginning, middle and end)  Only one theme is presented with several events  Narrative or informative text  Dialogues 9  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  There are some pages that contain only text  The illustration offers a weak to moderate support to the text  The illustration takes up several ideas in the text  The illustration lengthen the text by adding detail  The illustration promote an interpretation of the story  The meaning of the story is more in the text than in the illustration  Average 9 words per sentence  4 to 10 lines per page  250 to 320 words  About 8 to16 pages  Some familiar and frequent words  Some vocabulary is a little more complex  New specific vocabulary linked to the context  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative   Story structure (beginning, middle and end)  Only one theme is presented with several events  Narrative or informative text  Dialogues 10  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  There are short paragraphs  The illustration offers a weak to moderate support to the text  The illustration takes up several ideas in the text  The illustration lengthen the text by adding detail  The illustration promote an interpretation of the story  The meaning of the story is more in the text than in the illustration  Average 9 words per sentence  4 to 12 lines per page  300 to 400 words  About 14 to16 pages  Some familiar and frequent words  Some vocabulary is a little more complex  New specific vocabulary linked to the context  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative   Story structure (beginning, middle and end)  Story with multiple episodes links to a single plot line  Narrative or informative text  Dialogues Data Collection Page A-92 Level Presentation of the text Illustrations Length Choice of words Syntax Style 11  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  There are short paragraphs  The illustration offers a weak to moderate support to the text  The illustration summarize the main idea of the text  The illustration promote an interpretation of the story  The meaning of the story is more in the text than in the illustration  Average 8 to 10 words per sentence  4 to 14 lines per page  350 to 460 words  About 14 to16 pages  Some familiar and frequent words  Some vocabulary is a little more complex  New specific vocabulary linked to the context  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative  Story structure (beginning, middle and end)  Story with multiple episodes links to a single plot line  Narrative or informative text  Dialogues 12  Simple presentation  Writing may be presented in “talking bubbles”  Text is separate from illustrations (except for “talking bubbles”)  Text location may vary  Font reduced and easy to see  Sentences continue one more than one line  Each new sentence begins on a new line  There are short paragraphs  The illustration offers a weak to moderate support to the text  The illustration summarize the main idea of the text  The illustration promote an interpretation of the story  The meaning of the story is more in the text than in the illustration  4 to 14 lines per page  420 to 600 words  About 16 to 20 pages  Some familiar and frequent words  Some vocabulary is a little more complex  New specific vocabulary linked to the context  Text includes some new words not present in child’s oral vocabulary  Declarative and/or exclamatory sentences  There can be some interrogative and negative sentences  Simple sentences (S-V-C)  Verbs are in the present, present continuous, past and/or future tense of the indicative or present of the imperative  Story structure (beginning, middle and end)  Story with multiple episodes links to a single plot line  Narrative or informative text  Dialogues  Longer text with simple sentence structures to facilitate extended reading Page A93 APPENDIX B: DATA COLLECTION Teams of REB staff with support from L3 M&E specialists administered the FARS/MARS to the sample of students. Data collectors were identified by REB and trained by L3 staff in September of 2015 in Kigali. The training was designed to standardize the administration of the tools and increase the reliability of the assessment. It began with orienting the data collectors to the study and reviewing the fluency and mathematics instruments. Because the data were collected electronically, data collectors were trained how to use tablets. A significant portion of the training time was devoted to practice using the tools, both in the training environment and in schools. During actual data collection, teams of four REB-appointed and L3-trained data collectors traveled to five provinces; three out of five teams were accompanied by an L3 M&E specialist traveling with them to supervise data collection. The other two teams had regular communication with L3 M&E Manager who supervised the entire assessment to address any potential issues or concerns. Since the data capture was done electronically, daily data checks were conducted by L3 M&E Advisor to ensure high quality of data. Data checks included timer data, duration of administration, time of start and time of finish of each assessment, and GPS coordinates of the places of assessments. Completeness of the data and the accuracy of timers were ensured by the software used for the assessment. Data were then collected from the same 60 sampled schools that participated in the baseline. Teams of data collectors collected data at the same time, between October 5 and October 21 of 2015. All data were collected electronically, using tablets with SurveyToGo software in which FARS and MARS were programmed. All timed tasks were implemented automatically to reduce the possibility of an error. All testing was implemented in Kinyarwanda in P1-P3, and in Kinyarwanda and English in P4. The entire assessment took between 4 and 40 minutes, with the average time of 15 minutes per child. Assessors were asked whether schools and teachers were supportive of data collection. The majority of assessors reported that the school administration and teachers were supportive of data collection. When asked about school administration specifically, 85% of assessors reported that administrators were very supportive, with 15% indicating administrators were somewhat supportive. School teachers were rated as very supportive of data collection by 88% of assessors, with 11.5% saying that teachers were somewhat supportive and only one individual (0.5%) reporting that teachers were not supportive. Data Collection Page A-94 FIGURE 68. SCHOOL/TEACHER ARE SUPPORTIVE OF DATA COLLECTION (N=223) The majority of assessors reported that they were able to conduct interviews without interruptions by teachers or other learners walking into the room where the assessment was being conducted. While 75.5% indicated experiencing no interruptions, 24% experienced a few interruptions, and one assessor (0.5% of respondents) reported experiencing several interruptions. FIGURE 69. ASSESSMENTS WERE CONDUCTED WITH FEW INTERRUPTIONS (N=183) Assessors were asked whether the students they assessed were able to understand the language they were speaking. While 85% of assessors said that all or most of the students were able to understand the language they were speaking, 12% indicated that only some students were able to understand the language, and 3% stated that the students were not able to understand the language in which they were speaking. Yes, very, 85.2% Supportive school administration? Yes, very, 88.0% Yes, somewhat 11.5% Not supportive, 0.5% Supportive school teachers? Yes, somewhat 14.8% 75.5% 24.0% 0.5% There were no interruptions Yes, there were a few interruptions Yes, there were a lot of interruptions Data Collection Page A-95 Assessors were also asked whether they experienced problems during data collection. The majority (72.7%) of assessors reported that they did not experience any problems during data collection at the sample schools. The most common problems assessors faced were poor student knowledge including difficulty reading, disruptions during assessments, delays due to the rain or examinations being given in schools, and the absence of key teachers or administrators. A few assessors mentioned that students left before completing the assessment. Others mentioned a variety of isolated incidents including mis-entering data, having problems with a tablet, or the difficulty of assessing a student with a visual impairment. FIGURE 71. PROBLEMS DURING DATA COLLECTION (N=183) 72.7% 9.8% 4.4% 3.3% 3.3% 2.7% No problems Student Knowledge and Abilities Interruptions Rain delayed data collection Absent teachers or administrators Delayed start due to school testing 76.0% 9.3% 12.0% 2.7% All Students Understood Most Students Understood Some Students Understood Students did NOT Understand FIGURE 70: PERCENT OF STUDENTS WHO UNDERSTOOD THE LANGUAGE THE ASSESSOR USED Page A96 APPENDIX C. DATA COLLECTION TOOLS STUDENT CONTEXT SURVEY 1.What language(s) do you speak at home? (select all that apply) Mu rugo iwanyu muvuga uruhe rurimi? a. Kinyarwanda b. Kirundi c. Urukiga d. Amashi e. French f. English g. Swahili h. Arabic i. Other 2.At home, does someone read a stories to you? Mu rugo bajya bagusomera inkuru cyangwa bagucira umugani? a. Yes/Yego b. No/Oya c. Don’t know/Simbizi 3. Do you see your mother (or main caregiver) reading books or newspapers? Ujya ubona mama wawe cyangwa undi ukurera asoma igitabo cyngwa ikinyamakuru? a. Yes/yego b. No/oya c. My mother does not know how to read (Mama wanjye/undera ntazi gusoma) d. No response/Nta gisubizo 4. How often do you miss school? Ni kangahe ujya usiba ishuli? a. A lot (Kenshi) b. Sometimes (Rimwe na rimwe) c. Rarely (Gacye) d. Never (Ntanarimwe) 5. How often are you late for school? Ni kangahe ukererwa ishuli? a. A lot (Kenshi) b. Sometimes (Rimwe na rimwe) c. Rarely (Gacye) d. Never (Ntanarimwe) 5b. Why are you missing school or late for school? Vuga impamvu usiba/ucyererwa ishuli? a. Need to do chores (Gukora imirimo yo mu rugo) b. Go to market (Kuntuma ku isoko) c. Go work in the field (gukora mu murima) d. Waiting to eat (Gutegereza kurya) e. Long distance to school (Urugendo rurerure kugera ku ishuli) f. Want to play with my friends (Mba nkina n’inshuti zanjye) g. Help care for other children (Kurera barumuna banjye) h. Sick/not feeling well (Ntabwo meze neza. Ndarwaye) i. Sleep (kuryama) j. Other (Ikindi) Student Context Interview Page A-97 k. No response (Nta gisubizo) 6. Have you or any of your siblings ever repeated a grade? Wowe se cyangwa muri bakuru bawe hari uwigeze asibira mu mwaka? a. Yes/yego b. No/oya c. I don't know/Simbizi d. No response/Nta gisubizo 7. At home, which of the following do your parents expect you to do regularly? (tick all that apply) Mu rugo iwanyu, ni iki muri ibi bikurikira ababyebi bawe bagusaba gukora kenshi? (Hitamo igisubizo/ibisubizo) a. Help with household chores/Gufasha mu mirimo yo mu rugo b. Go to market/Kuntuma ku isoko c. Go work in the field/Gukora mu murima d. Study/Kwiga e. Help with other children in the family/Kurera barumuna banjye f. Other/Ikindi g. No response/Nta gisubizo 8. Do your parents/caregivers want you to go to school every day? Ese ababyeyi bawe/abakurera bifuza ko ujya ku ishuli buri munsi? a. Yes/yego b. No/oya c. No response/Nta gisubizo 9. Do your parents/caregivers check your homework? Ababyeyi bawe/abakurera bajya bagenzura umukoro wawe? a. Yes, every time/ Yego, buri gihe b. Yes, sometimes/ Yego rimwe na rimwe c. No, they do not check/ Oya, ntabwo bawugenzura 10. What do you like about school? (ask without reading the list) Ni iki ukunda kigendanye n’ishuli? a. Being with other students/Kuba hamwe nabandi banyeshuri b. I like how we are taught by our teachers/Nkunda uko abarimu batwigisha c. Playing/Gukina d. Studying Kinyarwanda/Isomo ry’ikinyarwanda e. Studying Math/Isomo ry’imibare f. Studying English/Isomo ry’ icyongereza g. Studying French/Isomo ry’igifaransa h. Studying science/Isomo ry’ubumenyi i. Reading books/Gusoma ibitabo j. Writing/Kwandika k. School environment/Imiterere y’ishuli l. Other/Ikindi m. Everything n. Nothing o. No response/Nta gisubizo 11. What do you NOT like about school? Ni iki udakunda ku bigendanye n’ishuli? a. Dirty school environment/Umwanda ku ishuli Student Context Interview Page A-98 b. Disputes among children/Impaka/guharira kw’abandi bana c. Disturbances in class by students/Gusakuza kw’abanyeshuri d. Corporal punishments given by teachers/Ibihano mpabwa n’abarimu e. Fighting and abuse by other students/Abana barwana f. Do not like studying Math/Kwiga imibare g. Do not like studying English/Kwiga icyongereza h. Do not like studying Kinyarwanda/Kwiga ikinyarwanda i. Do not like studying some lessons/Kwiga amasomo amwe namwe j. Indiscipline of some students/Abana bagira ikinyabupfura gicye k. Other/ikindi l. Everything m. Nothing n. No response/Nta gisubizo 12. Is this your first year in this grade? Ni ubwa mbere wiga muri uyu mwaka? a. Yes/Yego b. No, I am repeating this grade/ Oya, nasibiye muri uyu mwaka c. No response/Nta gisubizo 13. Does your math teacher check your work that you do in class? Mwarimu w’imibare ajya areba imyitozo ukorera mu ishuli? a. Yes/yego b. No/oya c. No response/Nta gisubizo 14. Does your math teacher check/mark your homework? Mwarimu w’imibare ajya areba/akosora umukoro wawe? a. Yes/yego b. No/oya c. No response/Nta gisubizo 15. Does your Kinyarwanda teacher check your work that you do in class? Mwarimu w’ikinyarwanda ajya areba imyitozo ukorera mu ishuli? a. Yes/yego b. No/oya c. No response/Nta gisubizo 16. Does your Kinyarwanda teacher check/mark your homework? Mwarimu w’icyinyarwanda ajya areba/akosora umukoro wawe? a. Yes/yego b. No/oya c. No response/Nta gisubizo 17. Do you ask questions when you don't understand something? Ese mu ishuli iyo hari ibyo utumvise, urabaza? a. Yes, I ask the teacher/ Yego mbaza mwalimu b. Yes, I ask other students/ Yego, mbaza abandi banyeshuli c. No, I don’t ask/ Oya, ntabwo mbaza Student Context Interview Page A-99 d. No response/Nta gisubizo 18. At school, can you choose which stories to read? Ese ujya uhabwa amahirwe yo kwihitiramo inkuru usoma uri mu ishuli? a. Yes/yego b. No/oya c. No response/Nta gisubizo 19. Are you allowed to take books home from school? Wemerewe se gutahana ibitabo mu rugo uvanye ku ishuli? a. Yes/yego b. No/oya c. No response/Nta gisubizo 20. Do you ever take books from school to read at home? Ujya utahana ibitabo ubivanye ku ishuli byo gusomera mu rugo? a. Yes/yego b. No/oya c. No response/Nta gisubizo 21. Did you have something to drink today (like water, tea, milk or juice)? Waje ku ishuli hari icyo unyweye? a. Yes/yego b. No/oya c. No response/Nta gisubizo 22. Did you have something to eat today, like potatoes, rice, bread or beans? Waje ku ishuli hari icyo uriye? a. Yes/yego b. No/oya c. No response/Nta gisubizo 23. Do you have radio or cell phone at home? Mu rugo iwanyu mufite iradiyo cyangwa terefoni? a. Yes/Yego b. No/Oya c. Don’t know/Simbizi 24. What light do you have at home? Mu rugo iwanyu mucana iki? a. Candles/buji b. Electric lamp/Amashanyarazi c. Paraffin lamp/Itara rya peterore d. Solar panel lamp/Ingufu z’izuba e. Biogas lamp/biyogaze f. Other/Ikindi 25. Does anyone at your house have a bicycle/motocycle or a car? Ese mu rugo iwanyu hari uwaba atunze igare/ipikipiki/imodoka? a. Yes/Yego b. No/Oya c. Don’t know/Simbizi P1 Assessment Tool Page A100 P1 ASSESSMENT ASSESSMENT INFORMATION SHEET A. Assessor’s Name B. Date of Assessment C. Province: D. School District: E. School Name: F. Student’s Name: Family name__________________ Other names___________________ G. Student’s Oldest Sibling’s First Name39: H. Student’s Age: [number of full years] I. Student’s Gender o Boy o Girl J. Consent o Yes o No K. Student’s Class o P1 o P2 o P3 o P4 Please enter this student’s teachers’ names: Kinyarwanda teacher’s name: ________________________________ Math teacher’s name: ________________________________ 39 If the student IS the oldest child in the family, write down “self”. P1 Assessment Tool Page A-101 FARS Task 1a: Oral Reading Fluency TEXT A  60 seconds Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Mahoro yagiye ku isoko guhaha ariko arayoba. Nuko asubira mu rugo arira. Yahuye na Kagabo amusaba kumuyobora. Kagabo aramuherekeza amugeza ku isoko. Nuko Mahoro ataha anezerewe pe! Task 1b: Reading Comprehension Noneho ngiye kukubaza ibibazo kuri iyi nkuru umaze gusoma Remove the text from the child before asking comprehension questions. Questions (Correct Answer) Correct Incorrect No answer Not Attempt ed 1. Mahoro yari agiye he? 2. Ni iki cyarijije Mahoro? 3. Mahoro yahuye na nde ubwo yari amaze kuyoba? 4. Kagabo yakoreye iki Mahoro? (Yaramuyoboye; Yaramuherekeje amugeza ku isoko; Yaramuherekeje; Yamujyanye ku isoko) 5. Mahoro yatashye ameze ate? Number of correct answers P1 Assessment Tool Page A-102 FARS Task 1c: SECOND READING Oral Reading Fluency TEXT A  180 seconds (3 MIN) Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Mahoro yagiye ku isoko guhaha ariko arayoba. Nuko asubira mu rugo arira. Yahuye na Kagabo amusaba kumuyobora. Kagabo aramuherekeza amugeza ku isoko. Nuko Mahoro ataha anezerewe pe! Task 1d: Reading Comprehension Noneho ngiye kukubaza ibibazo kuri iyi nkuru umaze gusoma Leave the text in front of the student. Questions (Correct Answer) Correct Incorrect No answer Not Attempted 1. Mahoro yari agiye he? 2. Ni iki cyarijije Mahoro? 3. Mahoro yahuye na nde ubwo yari amaze kuyoba? 4. Kagabo yakoreye iki Mahoro? (Yaramuyoboye; Yaramuherekeje amugeza ku isoko; Yaramuherekeje; Yamujyanye ku isoko) 5. Mahoro yatashye ameze ate? Number of correct answers P1 Assessment Tool Page A-103 MARS Task 1: Adding Numbers Sheet A 120 seconds Dore indi myitozo yo guteranya turi bukore [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Write: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 2 + 7 = (9) 2. 1 + 3 = (4) 3. 3 + 2 = (5) 4. 4 + 5 = (9) 5. 2 + 4 = (6) 6. 1 + 2 = (3) 7. 3 + 4 = (7) 8. 7 + 3 = (10) 9. 1 + 6 = (7) 10. 6 + 4 = (10) Total correct: _______________/10 P1 Assessment Tool Page A-104 Task 2: Subtracting Numbers Sheet B 120 seconds Dore indi myitozo yo guteranya turi bukore [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Write: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 7 - 4 = (3) 2. 9 - 5 = (4) 3. 5 - 2 = (3) 4. 3 - 2 = (1) 5. 8 - 4 = (4) 6. 6 - 5 = (1) 7. 9 - 7 = (2) 8. 10 - 3 = (7) 9. 8 - 3 = (5) 10. 9 - 4 = (5) Total correct: _______________/10 P1 Assessment Tool Page A-105 Task 3: Comparing numbers Sheet C 120 seconds Reba kuri buri tsinda ry’ imibare ikurikira. Muri buri tsinda, umubare munini ni uwuhe? Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 6 8 10 18 53 44 82 91 79 80 63 56 25 16 54 62 61 59 24 13 Total correct: _______________/10 P2 Assessment Tool Page A106 P2 ASSESSMENT ASSESSMENT INFORMATION SHEET A. Assessor’s Name B. Date of Assessment C. Province: D. School District: E. School Name: F. Student’s Name: Family name__________________ Other names___________________ G. Student’s Oldest Sibling’s First Name40: H. Student’s Age: [number of full years] I. Student’s Gender o Boy o Girl J. Consent o Yes o No K. Student’s Class o P1 o P2 o P3 o P4 Please enter this student’s teachers’ names: Kinyarwanda teacher’s name: ________________________________ Math teacher’s name: ________________________________ 40 If the student IS the oldest child in the family, write down “self”. P2 Assessment Tool Page A-107 FARS Task 1a: Oral Reading Fluency TEXT A  60 seconds  Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Ibidukikije bidufitiye akamaro kanini cyane. Ni byo dukesha ubuzima bwiza. Bigizwe n'umwuka mwiza duhumeka, ubutaka duhingamo ibidutunga, imisozi, imigezi, inzuzi n'ibiyaga. Bigizwe kandi n’indabo, ibiti by'amoko yose, amatungo n'inyamaswa by'amoko yose. Twese hamwe, nimuze turwane ku bidukikije, dutera ibiti, kandi turwanya isuri. Task 1b: Reading Comprehension Noneho ngiye kukubaza ibibazo kuri iyi nkuru umaze gusoma Remove the text from the student before asking comprehension questions. Questions Correct Incorrect No answer Not Attempted 1. Iyi nkuru iravuga ku biki? (Ibidukikije, ) 2. Kuki dukwiye kurengera ibidukikije? 3. Vuga ibintu bigize ibidukikije bivugwa mu mwandiko ? (kimwe muri ibi bikurikira: umwuka mwiza duhumeka, ubutaka duhingamo ibidutunga, imisozi, imigezi, inzuzi n'ibiyaga) 4. Vuga ibintu bibiri wakora ngo urengere ibidukikije? 5. Uyu mwandiko ukwigishije iki? Number of correct answers P2 Assessment Tool Page A-108 FARS Task 1c: SECOND READING Oral Reading Fluency TEXT A  180 seconds (3 MIN)  Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Ibidukikije bidufitiye akamaro kanini cyane. Ni byo dukesha ubuzima bwiza. Bigizwe n'umwuka mwiza duhumeka, ubutaka duhingamo ibidutunga, imisozi, imigezi, inzuzi n'ibiyaga. Bigizwe kandi n’indabo, ibiti by'amoko yose, amatungo n'inyamaswa by'amoko yose. Twese hamwe, nimuze turwane ku bidukikije, dutera ibiti, kandi turwanya isuri. Task 1d: Reading Comprehension Noneho ngiye kukubaza ibibazo kuri iyi nkuru umaze gusoma Leave the text in front of the student. Questions (Correct Answer) Correct Incorrec t No answer Not Attempted 6. Iyi nkuru iravuga ku biki? (Ibidukikije, ) 7. Kuki dukwiye kurengera ibidukikije? 8. Vuga ibintu bigize ibidukikije bivugwa mu mwandiko ? (kimwe muri ibi bikurikira: umwuka mwiza duhumeka, ubutaka duhingamo ibidutunga, imisozi, imigezi, inzuzi n'ibiyaga) 9. Vuga ibintu bibiri wakora ngo urengere ibidukikije? 10. Uyu mwandiko ukwigishije iki? Number of correct answers P2 Assessment Tool Page A-109 MARS Task 1: Adding Numbers Sheet A 120 seconds Dore indi myitozo yoguteranya turibukore [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Write: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 13 + 3 = (16) 2. 16 + 4 = (20) 3. 45 + 5 = (50) 4. 11 + 7 = (18) 5. 15 + 4 = (19) 6. 13 + 10 = (23) 7. 63 + 2 = (65) 8. 7 + 13 = (20) 9. 21 + 6 = (27) 10. 13 + 7 = (20) Total correct: _______________/10 P2 Assessment Tool Page A-110 MARS Task 2: Subtracting Numbers Sheet B 120 seconds Dore indi myitozo yo gukuramo turibukore [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Write: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 11. 12 - 4 = (8) 12. 6 - 3 = (3) 13. 10 - 5 = (5) 14. 13 – 1 = (12) 15. 10 – 1 = (9) 16. 20 – 10 = (10) 17. 15 – 5 = (10) 18. 17 – 6 = (11) 19. 15 – 10 = (5) 20. 20 – 17 = (3) Total correct: _______________/10 P2 Assessment Tool Page A-111 Task 3: Multiplying numbers Sheet C 120 seconds  Dore indi myitozo yo gukuba tugiye gukora [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora ibibazo byinshi bishoboka. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 2 x 2 = (4) 2. 3 x 5 = (15) 3. 4 x 5 = (20) 4. 2 x 6 = (12) 5. 6 x 3 = (18) 6. 7 x 4 = (28) 7. 9 x 1 = (9) 8. 5 x 6 = (30) 9. 7 x 7 = (49) 10. 8 x 9 = (72) Total correct: _______________/10 P3 Assessment Tool Page A112 P3 ASSESSMENT ASSESSMENT INFORMATION SHEET A. Assessor’s Name B. Date of Assessment C. Province: D. School District: E. School Name: F. Student’s Name: Family name__________________ Other names___________________ G. Student’s Oldest Sibling’s First Name41: H. Student’s Age: [number of full years] I. Student’s Gender o Boy o Girl J. Consent o Yes o No K. Student’s Class o P1 o P2 o P3 o P4 Please enter this student’s teachers’ names: Kinyarwanda teacher’s name: ________________________________ Math teacher’s name: ________________________________ 41 If the student IS the oldest child in the family, write down “self”. P3 Assessment Tool Page A-113 FARS Task 1a: Oral Reading Fluency TEXT A  60 seconds Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Nyanjwenge yigaga ku kigo cy’amashuri cya Mbyo. Yifuzaga kubona inyamaswa z’ishyamba. Ubwo yari mu kiruhuko, yagiye gutashya mu ishyamba yitwaje inshyimbo n’indyankwi mu ntoki. Akinjira ishyamba yabonye imbwebwe arashwiragira, ati reka “nshwekure”! Yigiye imbere ahura n’impyisi aramanjirirwa. Nuko yiruka kibuno mpa amaguru! Impyisi iramwirukankana ahita yurira igiti agira ati “kuva ubu sinzongera kwifuza kubona inyamaswa z’ishyamba”! Task 1b: Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Remove the text from the child before asking comprehension questions. Questions Correct Incorrect No answer Not Attempted 1. Ni nde uvugwa muri uyu mwandiko? (Nyanjwenge) 2. Yigaga ku kihe kigo cy’amashuri? (Icyigo cya mbyo) 3. Ni iki yifuzaga kubona? (Inyamaswa Z Ishyamba) 4. Ni iki yari afite mu ntoki ubwo yajyaga gutashya mu ishyamba? (Inshyimbo n’indyankwi) 5. Ahuye n’impyisi yabyifashemo ate? (Y’aramanjiriwe y’iruka kibuno mpamaguru) 6. Nyanjwenge yafashe uwuhe mwanzuro mu mpera z’uyu mwandiko? (Kutazongera kwifuza kubona inyamaswa z’ishyamba) Number of correct answers P3 Assessment Tool Page A-114 FARS Task 1c: SECOND READING Oral Reading Fluency TEXT A  180 seconds (3 MIN) Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira Nyanjwenge yigaga ku kigo cy’amashuri cya Mbyo. Yifuzaga kubona inyamaswa z’ishyamba. Ubwo yari mu kiruhuko, yagiye gutashya mu ishyamba yitwaje inshyimbo n’indyankwi mu ntoki. Akinjira ishyamba yabonye imbwebwe arashwiragira, ati reka “nshwekure”! Yigiye imbere ahura n’impyisi aramanjirirwa. Nuko yiruka kibuno mpa amaguru! Impyisi iramwirukankana ahita yurira igiti agira ati “kuva ubu sinzongera kwifuza kubona inyamaswa z’ishyamba”! Task 1d: Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Leave the text in front of the student. Questions Correct Incorrect No answer Not Attempted 1. Ni nde uvugwa muri uyu mwandiko? (Nyanjwenge) 2. Yigaga ku kihe kigo cy’amashuri? (Icyigo cya mbyo) 3. Ni iki yifuzaga kubona? (Inyamaswa Z Ishyamba) 4. Ni iki yari afite mu ntoki ubwo yajyaga gutashya mu ishyamba? (Inshyimbo n’indyankwi) 5. Ahuye n’impyisi yabyifashemo ate? (Y’aramanjiriwe y’iruka kibuno mpamaguru) 6. Nyanjwenge yafashe uwuhe mwanzuro mu mpera z’uyu mwandiko? (Kutazongera kwifuza kubona inyamaswa z’ishyamba) Number of correct answers P3 Assessment Tool Page A-115 Task 1: Multiplying Numbers Sheet A 120 seconds Dore indi myitozo yo gukuba tugiye gukora [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora ibibazo byinshi bishoboka. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 2 x 3 = (6) 2. 3 x 4 = (12) 3. 4 x 2 = (8) 4. 10 x 2 = (20) 5. 3 x 6 = (18) 6. 6 x 2 = (12) 7. 5 x 5 = (25) 8. 2 x 8 = (16) 9. 5 x 4 = (20) 10 5 x 40 = (200) Total correct: _______________/10 P3 Assessment Tool Page A-116 Task 2: Dividing Numbers Sheet B 120 seconds Dore indi myitozo yo gukuba tugiye gukora [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora ibibazo byinshi bishoboka. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 4 : 2 = (2) 6 : 3 = (2) 8 : 2 = (4) 6 : 2 = (3) 10 : 5 = (2) 8 : 4 = (2) 10 : 2 = (5) 2 : 2 = (1) 9 : 3 = (3) 12 : 6 = (2) Total correct: _______________/10 P3 Assessment Tool Page A-117 Task 3: Adding Numbers Sheet C 120 seconds Dore indi myitozo yoguteranya turibukore [glide hand from left to right]. Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 7 + 2 = (9) 2. 13 + 3 = (16) 3. 16 + 4 = (20) 4. 45 + 5 = (50) 5. 11 + 17 = (28) 6. 15 + 40 = (55) 7. 13 + 23 = (36) 8. 17 + 13 = (30) 9. 21 + 6 = (27) 10. 130 + 12 = (142) Total correct: _______________/10 P3 Assessment Tool Page A-118 Task 4: Subtracting Numbers Sheet D 120 seconds Dore indi myitozo yo gukuramo turibukore [glide hand from left to right]. Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 21. 7 - 4 = (3) 22. 13 - 3 = (10) 23. 18 - 1 = (17) 24. 23 – 3 = (20) 25. 17 – 5 = (12) 26. 40 – 15 = (25) 27. 100 – 50 = (50) 28. 38 – 6 = (32) 29. 25 – 9 = (16) 30. 40 – 37 = (3) Total correct: _______________/10 P4 Assessment Tool Page A119 P4 ASSESSMENT ASSESSMENT INFORMATION SHEET A. Assessor’s Name B. Date of Assessment C. Province: D. School District: E. School Name: F. Student’s Name: Family name__________________ Other names___________________ G. Student’s Oldest Sibling’s First Name42: H. Student’s Age: [number of full years] I. Student’s Gender o Boy o Girl J. Consent o Yes o No K. Student’s Class o P1 o P2 o P3 o P4 Please enter this student’s teachers’ names: Kinyarwanda teacher’s name: ________________________________ English teacher’s name: ________________________________ Math teacher’s name: ________________________________ 42 If the student IS the oldest child in the family, write down “self”. P4 Assessment Tool Page A-120 FARS Kinyarwanda Task 1a. Oral Reading Fluency TEXT A  60 seconds Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira usome Umunsi umwe, impyisi yagiye gutembera irayoba. Igerageza gusoma ibyapa biyobora abagenzi yabonaga birayinanira. Ikomeza kugenda iyobagurika. Mu nzira ihura n'imbwa irayiyoboza. Imbwa yo yari intyoza, irayibaza iti: "Ese uzi gusoma ibyapa biyobora abagenzi?" Impyisi iti:"Ashwi da! " Imbwa irayiyobora, ariko iyishishikariza kujya kwiga gusoma no kwandika. Impyisi iribwira iti: "Ni byo koko, kutamenya gusoma ni ikibazo gikomeye." Nyuma y’iminsi mike, impyisi ijya kwiga gusoma no kwandika Kinyarwanda Text 1b: Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Remove the text from the child before asking comprehension questions. Questions Correct Incorrect No answer Not Attempted 1. Ni izihe nyamaswa zivugwa muri iyi nkuru? (Impyisi n’imbwa.) 2. Ni ukubera iki impyisi yayobye? (Yayobye kubera kutamenya gusoma ibyapa biyobora abagenzi.) 3. Ni iki imbwa yabajije impyisi? (Yayibajije niba izi gusoma ibyapa biyobora abagenzi.) 4. Ni iyihe nama imbwa yagiriye impyisi? (Yayishishikarije kujya kwiga gusoma no kwandika.) 5. Impyisi yafashe uwuhe mwanzuro? (Yafashe umwanzuro wo jujya kwiga gusoma no kwandika.) Number of correct answers P4 Assessment Tool Page A-121 FARS Kinyarwanda Task 1C. Oral Reading Fluency – SECOND READING TEXT A  180 seconds (3 MIN) Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira usome Umunsi umwe, impyisi yagiye gutembera irayoba. Igerageza gusoma ibyapa biyobora abagenzi yabonaga birayinanira. Ikomeza kugenda iyobagurika. Mu nzira ihura n'imbwa irayiyoboza. Imbwa yo yari intyoza, irayibaza iti: "Ese uzi gusoma ibyapa biyobora abagenzi?" Impyisi iti:"Ashwi da! " Imbwa irayiyobora, ariko iyishishikariza kujya kwiga gusoma no kwandika. Impyisi iribwira iti: "Ni byo koko, kutamenya gusoma ni ikibazo gikomeye." Nyuma y’iminsi mike, impyisi ijya kwiga gusoma no kwandika Kinyarwanda Text 1D: Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Leave the text in front of the student. Questions Correct Incorrect No answer Not Attempted 6. Ni izihe nyamaswa zivugwa muri iyi nkuru? (Impyisi n’imbwa.) 7. Ni ukubera iki impyisi yayobye? (Yayobye kubera kutamenya gusoma ibyapa biyobora abagenzi.) 8. Ni iki imbwa yabajije impyisi? (Yayibajije niba izi gusoma ibyapa biyobora abagenzi.) 9. Ni iyihe nama imbwa yagiriye impyisi? (Yayishishikarije kujya kwiga gusoma no kwandika.) 10. Impyisi yafashe uwuhe mwanzuro? (Yafashe umwanzuro wo jujya kwiga gusoma no kwandika.) Number of correct answers P4 Assessment Tool Page A-122 FARS English Task 1a: Oral Reading Fluency TEXT B  60 seconds Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira usome My name is Kalisa. I like to take care of my body. I drink clean water and eat healthy food. I like to eat fresh fruit and vegetables. It is important to wash your hands before you eat. I like to play games and read books. Sleeping is good for you. It helps your body rest. English Task 1b. Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Remove the text from the child before asking comprehension questions. Questions Correct Incorrect No answer Not Attempted 1. Who is talking in the story? (Kalisa) 2. What does Kalisa do to take care of his body? (Drinks clean water and eats healthy food) 3. According to Kalisa, what should you do before eating? (Wash your hands) 4. What does Kalisa like to do? (Play games and read books) 5. Why is sleeping good? (It helps you rest) Number of correct answers P4 Assessment Tool Page A-123 FARS English Task 1C: Oral Reading Fluency: SECOND READING TEXT B  180 seconds (3 MIN) Ngiye kugusaba gusoma inkuru. Ugerageze gusoma cyane. Uriteguye dutangire? Ngaho tangira usome My name is Kalisa. I like to take care of my body. I drink clean water and eat healthy food. I like to eat fresh fruit and vegetables. It is important to wash your hands before you eat. I like to play games and read books. Sleeping is good for you. It helps your body rest. English Task 1D. Reading Comprehension Noneho ngiye kukubaza ibibazo kur’iyi nkuru umaze gusoma Leave the text in front of the student. Questions (Correct Answer) Correct Incorrect No answer Not Attempted 6. Who is talking in the story? (Kalisa) 7. What does Kalisa do to take care of his body? (Drinks clean water and eats healthy food) 8. According to Kalisa, what should you do before eating? (Wash your hands) 9. What does Kalisa like to do? (Play games and read books) 10. Why is sleeping good? (It helps you rest) Number of correct answers P4 Assessment Tool Page A124 Task 1: Addition Sheet A 60 seconds  Dore indi myitozo yoguteranya turibukore [glide hand from left to right]. Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 4 + 5 = (9) 2. 3 + 9 = (12) 3. 7 + 2 = (9) 4. 5 + 15 = (20) 5. 20 + 20 = (40) 6. 5 + 6 = (11) 7. 32 + 3 = (35) 8. 25 + 25 = (50) 9. 19 + 6 = (25) 10. 300 + 200 = (500)  Total correct: _______________Total time: ____________________ P4 Assessment Tool Page A-125 Task 2: Subtraction Sheet B 60 seconds Dore indi myitozo yo gukuramo turibukore [glide hand from left to right]. Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 7 - 2 = (5) 2. 10 - 6 = (4) 3. 25 - 5 = (20) 4. 18 – 2 = (16) 5. 50 – 10 = (40) 6. 16 – 4 = (12) 7. 9 – 5 = (4) 8. 200 – 100 = (100) 9. 50 - 60 = (-10) 10. 100 – 100 = (0)  Total correct: _______________Total time: ____________________ P4 Assessment Tool Page A-126 Task 3: Multiplication Sheet C 60 seconds Dore indi myitozo yo gukuba tugiye gukora [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora ibibazo byinshi bishoboka. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 2 x 4 = (8) 2. 3 x 3 = (9) 3. 5 x 2 = (10) 4. 5 x 10 = (50) 5. 7 x 3 = (21) 6. 4 x 6 = (24) 7. 15 x 2 = (30) 8. 20 x 10 = (200) 9. 6 x 5 = (30) 10 12 x 10 = (120)  Total correct: _______________Total time: ____________________ P4 Assessment Tool Page A-127 Task 4: Division Sheet D 60 seconds Dore indi myitozo yo gukuba tugiye gukora [glide hand from left to right]. Ngiye kwifashisha iyi saha ibara. Ngaho kora ibibazo byinshi bishoboka. Niba utazi igisubizo, jya ku kibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. 0 = Incorrect or no response [ ] After last problem attempted 1. 9 ÷ 3 = (3) 2. 4 ÷ 2 = (2) 3. 24 ÷ 6 = (4) 4. 10 ÷ 2 = (5) 5. 15 ÷ 3 = (5) 6. 50 ÷ 10 = (5) 7. 20 ÷ 5 = (4) 8. 55 ÷ 11 = (5) 9. 70 ÷ 1 = (70) 10. 200 ÷ 2 = (100)  Total correct: _______________Total time: ____________________ P4 Assessment Tool Page A-128 Task 5: Number comparison Sheet E 60 seconds Reba kuri buri tsinda ry’ imibare ikurikira. Muri buri tsinda, umubare munini ni uwuhe? Ngiye kwifashisha iyisaha ibara. Ngaho kora imibare myinshi uko ushoboye. Niba utazi igisubizo, jya kukibazo gikurikiyeho. Uriteguye? . . . - Tangirira aha [point to first problem]. Circle: 1 = Correct. (Bolded and underlined number is the correct answer) 0 = Incorrect or no response [ ] After last problem attempted 1. 2/4 or 2/6 2. 0.01 or 0.1 3. 55 or -60 4. 4/2 or 3 5. 3.7 or 3.77 6. -2 or 0.5 7. 4/5 or 1/2 8. 7.5 or 70 9. -20 or -25 10. 1/4 or 4/2  Total correct: _______________Total time: ____________________ School Form Page A129 SCHOOL MONITORING FORM A. L3observer /Izina ry’Umukozi wa L3: B. Date of visit/Itariki y’isuzuma: C. Province: D. School District: E. School Name: Name of Head-teacher: _________________________ Phone No: _____________________ 1. Materials checklist: Did the school receive from L3 the following (indicate quantity of each) Ishuri muyobora ryabonye ibitabo bivuye muri L3 ( Andika umubare ): Material check Item Quantity/umubare Item Quantity/ umubare T1 T2 T3 P1 Kinyarwanda guide/ Imfashanyigisho Kinyarwanda P1 P1 Kinyarwanda reader/ Igitabo cy’umunyeshuri P1 Kinyarwanda P1 Kinyarwanda Read aloud/ Igitabo cy’inkuru P1 Kinyarwanda P2 Kinyarwanda reader/ Igitabo cy’umunyeshuri P2 Kinyarwanda P2 Kinyarwanda guide/ Imfashanyigisho Kinyarwanda P2 P1 English reader/ Igitabo cy’umunyeshuri P1 Icyongereza P2 Kinyarwanda Read aloud/ Igitabo cy’inkuru P2 Kinya P2 English reader/ Igitabo cy’umunyeshuri P2 Icyongereza P3 Kinyarwanda guide/ Imfashanyigisho Kinyarwanda P3 P2 Math guide/ Imfashanyigisho P2 Imibare School Form Page A-130 P3 Kinyarwanda Read aloud/ Igitabo cy’inkuru P3 Kinya P3 English guide/ Imfashanyigisho P3 Icyongereza P1 English guide/ Imfashanyigisho P1 Icyongereza P3 Kinyarwanda reader/ Igitabo cy’umunyeshuri P3 Kinyarwanda P2 English guide/ Imfashanyigisho P2 Icyongereza P3 English reader/ Igitabo cy’umunyeshuri P3 Icyongereza P3 English guide/ Imfashanyigisho P3 Icyongereza Solar Panel/Icyuma gitanga amashanyarazi P3 English Read aloud/Igitabo cy’inkuru P3 Kinyarwanda Cellphones/telefone P1 Math guide/ Imfashanyigisho P1 Imibare Speakers/indangururamaj wi P3 Math guide/ Imfashanyigisho P3 Imibare SD cards/memori kadi 2. PTA/PTC information Answers 2.1 Does the school have a functioning PTA/PTC? / Mwabamugira PTA/PTC ikora? Yes /yego No/oya 2.2.Has the PTA been trained by Concern Worldwide? /Niba ihari yaba yarahuguwe na Concern Worldwide? Yes/yego No/oya 2.3 How many PTA/PTC members attended the training? /Ni bangahe bitabiriye amahugurwa? _______(number/umubare) 2.4 Did the PTC members (who attended the training facilitated by Concern) train other PTA/PTC members?/ Abitabiriye amahugurwa (yateguwe na ConcernWoldwide) bahuguye abandi bagize PTA/PTC? Yes/yego No/oya 2.5 Does the PTA/PTC have an action plan? / PTA/PTC ifite iteganyabikorwa? Yes/yego No/oya 2.6 Has the PTC/PTA undertaken initiatives to support teacher motivation? /PTA/PTC yaba yaratangije gahunda zafasha mwarimu gukora umurimo we awishimiye? Yes/yego No/oya (if YES specify how)/Niba zihari ,zivuge? - School Form Page A-131 - 2.7 Has the PTC/PTA undertaken initiatives to support literacy and equity in education/ PTA/PTC yaba yaratangije gahunda ziteza imbere umuco wo gusoma, ubudasumbana no guha abana bose amahirwe angina? Yes/yego No/oya (if YES specify how) Niba zihari, zivuge? - - 3. School-Based Mentors Answers/ibisubizo 3.1 Does the School have a Mentor? /Mufite mentor? Yes/yego No/oya 3.2 Is there a weekly plan detailing school based mentor’s activities in the school? Yes/yego No/oya 3.3 Does the Mentor train the teachers/head teachers on the use of L3 materials/?/Mentor ajya ahugura abarimu/Umuyobozi? Yes/yego No/oya 3.4 If yes, how many P3 math, English and Kinyarwanda teachers trained this month by the school-based mentor? /Ni abarimu bangahe bigisha imibare, ikinyarwanda,icyongereza bo muri P3 bahuguwe na Mentor muri uku kwezi? __________(number of male teachers) Umubare w’ Abagabo __________(number of female teachers) Umubare w’ Abagore 3.5. How many lessons has the School Based Mentor observed in this term? ____________lessons 3.6. Does the SBM use videos modules to train teachers? Yes/yego No/oya 4. 4. Enrolment (Indicate the total number of students enrolled, not just those present during the visit)/Vuga umubare w’abanyeshuri bose banditse: Students/ abanyeshuri Repeaters/Abasi bire Drop out students in this year /Abaretse ishuri No of classrooms Umubare w’ibyumba by’amashuri Shift/Isim burana 1: Single 2: Double Male/ Gabo Female / Gore Male/ Gabo Female/ Gore Male/ Gabo Female /Gore P 1 P 2 P 3 P 4 School Form Page A-132 5. 5. Number of Teachers/Umubare w’abarimu Subject/isomo Number of Teachers GRADE Male/Gabo Female/Gore P1 Kinya P1 Math P1 English P1 Total actual number of P1 teachers*:/ Umubare w’abarimu bigisha P1 P2 Kinya P2 Math P2 English P2 Total actual number of P2 teachers: / Umubare w’abarimu bigisha muri P2 P3 Kinya P3 Math P3 English P3 Total number of P3 teachers: Umubare w’abarimu bigisha muri P3 TOTAL actual NUMBER OF P1- P3** teachers Igiteranyo cya P1 –P3 Name (s) of P1 teacher(s) Name (s) of P2 teacher (s) Name (s) of P3 teacher (s) *In some schools a teacher may team more than one subject. Please indicate here the total number of teachers in this grade. /Hari amwe mumashuri afite umuwarimu wigisha amasomo arenze rimwe. Vuga umubare w’abarimu bigisha muri uyu mwaka. ** In some schools a teacher may team more than one grade. Please indicate here the actual total number of teachers teaching P1, P2 and P3 in this grade. 6. School leadership Answers/ibisubizo 6.1 Does your school have a system for tracking teacher attendance? If yes, ask head teacher to show you their teacher attendance records. No Yes, Attendance records completed daily Yes, Attendance records completed weekly Yes, Attendance records completed bi￾weekly School Form Page A-133 Yes, Attendance records completed monthly 6.2 How many teachers who teach P1, P2, or P3 were absent yesterday?/Ni abarimu bangahe bigisha P1, P2, P3 basibye ejo? ______ teachers out of __________ teachers 6.3 How many teachers who teach P1, P2, or P3 are absent today? Ni abarimu bangahe bigisha P1, P2, P3 basibye none? ______ teachers out of __________ teachers KR 6.4 How often do you observe teachers teaching in their classrooms?/Ni kangahe ugenzura imyigishirize y’abarimu mu mashuri?  At least once a week/Nibura rimwe mu cyumweru  At least once a month/Nibura rimwe mu kwezi  At least once a term/Nibura rimwe mu gihembwe  At least once a year/Nibura rimwe mu mwaka  Not at all/Nta narimwe 6.5 Do you monitor the reading progress of students in the school? Yes/yego No/oya 6.6 If yes, how do you monitor the reading progress of students in the school? Classroom Observation Monitor students’ results on tests given by the teacher Evaluate children orally myself Review students’ assignments or homework Teachers provide me progress reports End of term evaluations 6.7 Are there records of children with learning barriers? Yes/yego No/oya 6.8 If yes, are there some remedial measures to support children with learning barriers? Yes/yego No/oya 7 School environment Answers/ibisubizo 7.1 Does the School have a library? / Ishuri ryanyu rifite Isomero? (If yes, ask to see the library) Yes/yego No/oya Observation: 7.2 Is there a nursery school attached to the school? Yes/yego No/oya Observation: 7.3 Does your school get support from other organization/ NGO (s)? (Ishuri ryanyu rihabwa inkunga nindi Miryango) (other than L3) Yes/Yego No/Oya If yes, specify/Inkunga mu biki?: Teaching and learning materials School Form Page A-134 Teacher training Other……………………………… 8 Literacy resources in the community Answers/ibisubizo 8.1 Is there a community library or place in the community where students can borrow books to read? Yes/yego No/oya 9 Other School Features Answers/ibisubizo 9.1 How far is the school located from the District’s Office? __________kms 9.2 How often does this school have school assemblies?/Inteko/rassemblement n’abanyeshuri iba kangahe?  Every Day/Buri munsi  At least once a week/Nibura rimwe mu cyumweru  At least once a month/Nibura rimwe mu kwezi  At least once a term/Nibura rimwe mu gihembwe  At least once a year/Nibura rimwe mu mwaka  Never/Nta narimwe 9.3 How many times are parents/caregivers invited to come to the school each year?/Inama y’ababyeyi iba kangahe mu mwaka? Number/Umubare : _________ 9.3b (if the answer to #2 is greater than 0) When invited to come to the school, how many parents/caregivers come ?/(Niba igisubizo ku kibazo cya 2 ari hejuru ya kabiri) Iyo bayumiwe mu nama, Haza abangana iki?  All/Bose  Most /Hafi ya bose  A Moderate Amount/Abagereranyije  Few/Bacye 9.4 What do you do to encourage students to come to school ?/Ni iki ukora ngo ushishikarize abanyeshuri kuza kwiga?  Parent meetings/PTA/Inama z’ababyeyi  Provide milk to students/Kubaha amata yo kunywa  Provide shoes to students/Kubaha inkweto zo kwambara  Playground for students to enjoy/Kubategurira ibibuga byo gukiniraho  Separate toilets for boys and girls/Ubwiherero butandukanye kubahungu n’abakibwa  Special rooms for girls/Imyumba byihariye byisuku by’abakobwa  Incentives for good academic performance/Ibihembo kubanyeshuli batsinda kurusha abandi  Punishment if not come/Ibihano kubasiba School Form Page A-135  Competitions/Amarushanwa atandukanye  Other/Ikindi_______________  Nothing/Ntacyo 9.5 Are there discipline measures for children who come late to school?/Hari ibihano bihabwa bana baza bakererewe?  Yes /Yego  No/Oya  Depends on a teacher, it’s up to them/Biva k’umwarimu 9.6 If yes, what are they?/Niba Bihari, bivuge  Corporal punishment/Ibihanobibabaza (gukubita, gupfukama, etc)  Student who is late is not admitted to class/Uwacyererewe ntiyemererwa kwinjira mu ishuli  Student who is late is sent for detention/Uwacyererewe arafungwa  Student who is late helps with cleaning, other tasks/Uwakererewe akora isuku nindi mirimo  Other forms of disciplining/Ibindi 9.7 Are there discipline measures for children who come miss school?/ Hari ibihano bihabwa abana basiba ishuli?  Yes /Yego  No/Oya  Depends on a teacher, it’s up to them/Biva k’umwarimu 10 Challenges: To what extent are the following inhibit teaching and learning in your school)? /Ni kuruhe rugero ibi bikurikira bibangamire imyigishirije n’imyijyire kuri iri shuli? Challenge/Imbogamizi 1=Not a problem at all/Si ikibazo namba 2=Hardly a problem/Ni ikibazo gito 3=A moderate problem/Ni ikibazo kiringaniye 4=A severe problem/Ni ikibazo gikomeye 10.1 Too many students in a class/Abanyeshuri benshi mu ishuli rimwe 10.2 Students are over age/under age for a particular class/Abanyeshuri barengeje imyaka/abatagejeje imyaka mu mashuli amwe 10.3 Students are hungry/Abanyesuli bashonje 10.4 Students are tired/Abanyeshuri bananiwe 10.5 Students are sick/Abanyeshuri barwaye 10.6 Students misbehave/Abanyeshuri bitwara nabi 10.7 Students do not attend class consistently or arrive late/Abanyeshuri School Form Page A-136 Challenge/Imbogamizi 1=Not a problem at all/Si ikibazo namba 2=Hardly a problem/Ni ikibazo gito 3=A moderate problem/Ni ikibazo kiringaniye 4=A severe problem/Ni ikibazo gikomeye basiba cyane cyangwa bacyererwa kuza kwiga 10.8 Students receive little help with school work at home/do not complete their homework/Abanyeshuri babona ubufasha bucye mu gukora imikoro/ntibarangize imikoro 10.9 Parents/Caregivers are not literate/Ababyeyi batazi gusoma no kwandika 10.10 Parents/Caregivers do not support their children’s education/Ababyeyi batagira uruhari muburezi bw’abana babo 10.11 Students have to walk far to school/Abanyeshuli bakora urugendo rurerure ngo bagere ku ishuli 10.12 Large number of students drop out/Umubare munini w’abanyeshuri bata ishuli 10.13 Other/Ikindi_______________________ 11 Learning environment 11.1 In P1, how many students share one desk ?/Mu mwaka wa mbere, itebe yicaraho abanyeshuri bangahe? ______ students/abanyeshuri  No desk/Nta ntebe zihari 11.2 In P2, how many students share one desk ?/ Mu mwaka wa kabiri, itebe yicaraho abanyeshuri bangahe? ______ students/abanyeshuri  No desk/Nta ntebe zihari 11.3 In P3, how many students share one desk ?/ Mu mwaka wa gatatu, itebe yicaraho abanyeshuri bangahe? ______ students/abanyeshuri  No desk/Nta ntebe zihari 11.4 In P4, how many students share one desk ?/ Mu mwaka wa kane, itebe yicaraho abanyeshuri bangahe? ______ students/abanyeshuri  No desk/Nta ntebe zihari School Form Page A-137 11.5 Which items must parents purchase for a student to attend school?/Ni ibihe bikoresho ababyeyi basabwa kugurira abanyeshuri?  Books/Ibitabo  Pens/pencils/Amakaramu  Notebooks/Amakayi  Uniform/Imyambaro y’ishuli  School Fees/Kwishura asabwa n’ishuli  Tuition/Amafaranga y’ishuli  School bag/Udukapu  Food to eat at school/Ibiryo barira ku ishuli  Other/Ikindi : _______  None of the above/Ntanakimwe mubiri hejuru 11.6 How easily can MOST of the families whose children attend this school pay for these items?/Ababyeyi boroherewe gute mu kugura ibi bikoresho?  Very easily/Biraborohera cyane  Somewhat easily/Biraborohera  With some difficulty/Bahura n’imbogamizi  With extreme difficulty/Bahura n’imbogamizi nyinshi 11.7 Most of the students who attend this school come from families that are/Abanyeshuri biga kuri iki kigo bava mu miryango :  Very poor/Ikennye cyane  Somewhat poor/Ikennye  Of moderate means/ifite ubushobozi bugereranyije  Somewhat wealthy/Ikize  Very wealthy/Ikize cyane 12. Rank the following items as to their importance in your decision to hold a student back (where 4=most important and 1=least important)./Erekana ikigero uha impamvu zikurikira mu gusibiza umunyeshuri? (4= ngombwa cyane naho 1= si ngombwa nabusa) Reason/Impamvu 1=Not important/Si ngombwa 2=Hardly important/Ni ngombwa gacye cyane 3=Somewhat important/Ni ngombwa 4=Very important/Ni ngombwa cyane 12.1 Poor attendance/Gusiba ishuli cyane 12.2 Low grades/Amanota macye 12.3 Behavioral problems/Ibibazo by’imyitwarire 12.4 Student is an inappropriate age for the grade/Umunyeshuri ufite imyaka irenze ishuli yigamo 12.5 Parent requests that the student be held back/Bisabwe n’ababyeyi b’umunyeshuli School Form Page A-138 12 Are there discipline measures for children who come late to school?/Hari ibihano bihabwa bana baza bakererewe?  Yes /Yego  No/Oya  Depends on a teacher, it’s up to them/Biva k’umwarimu 13 If yes, what are they?/Niba Bihari, bivuge  Corporal punishment/Ibihanobibabaza (gukubita, gupfukama, etc)  Student who is late is not admitted to class/Uwacyererewe ntiyemererwa kwinjira mu ishuli  Student who is late is sent for detention/Uwacyererewe arafungwa  Student who is late helps with cleaning, other tasks/Uwakererewe akora isuku nindi mirimo  Other forms of disciplining/Ibindi 14 Are there discipline measures for children who come miss school?/ Hari ibihano bihabwa abana basiba ishuli?  Yes /Yego  No/Oya  Depends on a teacher, it’s up to them/Biva k’umwarimu School Observation Form Page A139 GRADE MONITORING FORM A. Assessor Name/Izina ry’umukozi wa L3 B. Date of the visit /Itariki C. Province/Intara: D. School District/Akarere: E. School Name/Izina ry’ishuli: 1. What is your name? Amazina yawe ni ayahe? Family name/Izina ry’umuryango_________ Other names/ Andi mazina ____________ 2. Sex of teacher/Igitsina Male/Gabo Female/Gore 3. Professional preparation in teaching/Wize ubwarimu? From TTC Genera Secondary education Through Distance learning with CoE ( KIE) None 4. Teaching experience/Uburamb e kukazi? ___________years/imyaka 5. Have you ever been trained by NGO on literacy and numeracy?/Waba warigeze uhugurirwa kwigisha gusoma no kubara na ONG? Yes/yego No/Oya 6. Have you been teaching these students since they entered this grade?/Ni wowe wigishije aba Yes No School Observation Form Page A-140 banyeshuru kuva umwaka watangira? 7. What documents do you use when preparing your lessons plans?/Ukoresha izihe mfashanyigisho iyo utegura amasomo? Curriculum documents Schemes of work provided by REB L3 teachers guides None 8. What grade are you teaching?/Ni uwuhe mwaka wigishamo? P1 P2 P3 P4 9. What subject are you teaching this grade? Ni irihe somo wigisha muri uwo mwaka? Kinyarwanda Math English 10. Do you use L3 TLMs while teaching this subject with this grade?/ Waba ujya wifashisha imfashanyigisho za L3 mu kwigisha iri somo? Yes/Yego No/Oya (skip to Q13) (komeza ku kibazo cya 13 niba ukoresha izindi mfashanyigisho zitari iza L3) Q-s for L3 TLMs only 11. Which term? (Ni ikihe gihembwe ugezeho wigisha?) Term 1 Term 2 Term 3 12. Which week?(Ni icyumweru cyakangahe ugezeho wigisha) _________(week #)(icyumweru cya) 13. Which lesson?(Ni isomo rya kangahe ugezeho wigisha?) _________(lesson #)(isomo rya) 1. How many children are enrolled in your class? Ni abanyeshuli Number____________ School Observation Form Page A-141 bangahe biga muri iri shuli/iki gice? 14. How many children are absent today? Muri aba banyeshuri, ni bangahe baje uyu munsi? Number____________ 15. How many children in your class are with learning barriers? Ni abanyeshuri bangahe muri iri shuli bafite inzitizi mu myigire? 16. What is the age range of the pupils in your class?/Tubwire ikigero cy’imyaka y’abana wigisha (umwana muto n’umukuru mu ishuli) Between/Hagati ya ________ and/na___________ 17. Do you take attendance every day? /Urahamagara se buri munsi? Yes No 18. Do you have a list of attendance of the pupils you teach? (If yes, ask teacher to show it to you)/Waba ufite ikaye uhamagariramo abanyeshuli? Register not available to be examined /Ntayo Register available to be examined /Irahari 19. How many children in this class are repeaters? Ni bangahe basibiye muri iri shuli? Number/Umubare _______ 20. How many of the pupils in your class arrive to school (Select the teacher’s response for each category (early, at school starting time and late)/Hitamo ukurikije ikigero (Abaza kare, abaza ku gihe n’abakererwa) School Observation Form Page A-142 …/Tanga ikigero abana wigisha bazira ku ishuli a) early/kare b) at school startingtime/Ku gihe c) Late/Bakererewe a) Early/Kar e A few/ba ke Some/B aringani ye Many/B enshi Most of all/Hafi yabose b) At school starting time/Ku gihe A few/Ba ke Some//B aringani ye Many/B enshi Most of all/ Hafi yabose c) Late/Bake rerewe A few/Ba ke Some//B aringani ye Many/B enshi Most of all/ Hafi yabose 21. Are there discipline measures for children who come late to schools?/Hari ibihano bihabwa abana baza ku ishuli bakererewe? Yes/Yego No/Oya 22. Have you attended L3 training?(Wigeze ujya mu mahugurwa ya L3) Yes/Yego No/Oya 23. Have you attended a training by your school-based mentor on L3 materials? (wigeze uhugurwa na school based mentor w’ikigo cy’amashuri cyawe?) Yes/Yego No/Oya Our school does not have mentor(ikigo cy’amashuri cyacu nta school based mentor kigira) 24. Did you receive technology from L3? /(Ni ibihe ibikoresha by’ikoranabuhanga bya L3 mwakiriye) Yes, cell phone/Yego twakiriye telephone Yes, speakers/Yego, indangururamajwi Yes, SD card/Yego twakiriye memory card No 25. If you said yes in the previous question, how often do you use this technology in teaching pupils this subject (s) in your classroom?(Niba warakiriye ibikoresho Every day (buri munsi) 2-4 times a week(kabiri-kane mu cyumweru) Once a week(rimwe mu cyumweru) More rarely than once a week(Gacye munsi ya rimwe mu cyumweru) Never(Nta na rimwe) If never, why? Niba nta narimwe ni kubera iki? School Observation Form Page A-143 by’ikoranabuhanga, ni inshuro zingahe ujya ubikoresha iyo wigisha abanyeshuri iri somo?) 26. Do you have any comments about L3 materials, training, or school-based mentors? Haba hari igitekerezo cyangwa icyivuzo watanga ku bikoresho bya L3, ku mahugurwa cyangwa kuri gahunda y’aba school based mentor? 27. Do you find it easy to teach reading? / Ubona byoroshye kwigisha gusoma?  Not easy at all/ Ntabwo byoroshye na gato  Sometimes not easy/ Rimwe na rimwe ntabwo biba byoroshye  Mostly easy/ Akenshi biba byoroshye  Very easy/ Biroroshye cyane 28. Why?/Kubera iki? 29. Do you find it easier to teach boys or girls how to read? / Ubona byoroshye kwigisha gusoma abahungu cyangwa abakobwa?  Boys/ Abahungu  Girls/ Abakobwa  There is no difference/ Nta tandukaniro ririmo 30. Why?/Kubera iki? 31. Last week, how many days were you absent?/Icyumweru gishize, wasibye iminsi ingahe?  None/Nta numwe  1  2  3  4  All/Yose 32. How often does the school administration observe you teaching?/Ni kangahe abayobozi b’ikigo bakugenzura wigisha?  At least once a week/Nibura rimwe mu cyumweru School Observation Form Page A-144  At least once a month/Nibura rimwe mu kwezi  At least once a semester/Nibura rimwe mu gihembwe  At least once a year/Nibura rimwe mu kwezi  Not at all/Nta narimwe 33. Does the school administration provide you with following materials? (tick all that apply)/Ubuyobozi bw’ikigo bwaba bubaha ibikoresho bikurikira? (hitamo muri ibi bikurikira)  Paper for students/Impapuro z’abanyeshuri  Chalk/Ingwa  Posters for use in classroom/Impapuro nini zo mu ishuli  Books for students/Ibitabo by’abanyeshuri  Instructional technology (e.g., cell phones with speakers)/Ibikoresho by’ikoranabuhanga (terefoni n’indangururamajwi)  Laptops for students/Mudasobwa z’abanyeshuri  None of the above/Nta nakimwe muri ibi  Other/ikindi: _______ 34. How often do parents/caregivers of your students come to school to talk with teachers?/Ni inshuro zingahe ababyeyi b’abana wigisha baza ku ishuli kukureba?  At least once a week/Rimwe mu cyumweru  At least once a month/Rimwe mu kwezi  At least once a semester/Rimwe mu gihembwe  At least once a year/Rimwe mu mwaka  Never/Nta narimwe 35. How do you decide when to hold a student back a year?/Ugendera kuki mu gusibiza umunyeshuri? Reason/Impamvu YES/yego NO/Oya 35.1 Poor attendance//Gusiba ishuli cyane 35.2 Low grades/amanota macye 35.3 Behavioral problems/Ibibazo by’imyitwarire 35.4 Student is an inappropriate age for the grade/ Umunyeshuri ufite imyaka irenze ishuli yigamo 35.5 Parent requests that the student be held back/ Bisabwe n’ababyeyi b’umunyeshuli 36. Are there any comments you would like to add?/ Haba hari ikindi gitekerezo wifuza kongeraho? School Observation Form Page A-145 SCHOOL OBSERVATION FORM Infrastructure/Imyubakire Please note availability and condition of the following/Garagaza niba ibi bikurikira bihari ndetse nuko bimeze None/ Ntibihari Poor condition/ availability/Bimez e nabi cyane/Nibicye cyane Adequate condition/ availability/ Bimeze neza mu rugero/Birahari mu rugero Good condition/ availability/ Bimeze neza cyane/Birahari bihagije 1. School building/ Amazu 2. Roof/Ibisenge 3. Electricity/solar panels/Umuriro 4. Drinking water/Amazi meza 5. Separate latrines for boys and girls/Ubwiherero bw’amahungu n’abakobwa 6. Blackboards in classrooms/Ibibaho byo mumashuli 7. Clean, ventilated classroom space/Amashuli asukuye, yisanzuye 8. Good lighting in classrooms/urumuri ruhagije mu mashuli 9. Desks for students/Intebe z’abanyeshuri 10. Reading materials for students/Ibikoresho byo gusoma by’abanyeshuri 11. Writing materials for students (e.g, paper, slate boards)/Ibikoresho byo kwandikaho by‘abanyeshuri (impapuro, nibindi 12. Library/resource center/Isomero School Observation Form Page A-146 13. Safe space for students to run and play outside/Imbuga yo hanze itekanye yo gukiniramo 14. Are there print materials (posters, signs, etc) on school or classroom walls?/Haba hari imfashanyigisho (posters, signs, nibindi) bimanitse ku bikuta mu mashuli?  Yes, print materials in classrooms/Yego, mu mashuli  Yes, print materials in hallways/Yego, hanze mu kigo  No print materials displayed/Ntabigaragara 15. Where are L3-provided printed teacher guides observed?/Ni hehe wabonye ibitabo by’abarimu bya L3?  In teachers’ hands/Nabibonanye abarimu  In the library/Nabibonye mu isomero  In the headmaster office/Nabibonye mu biro by’umuyobozi w’ikigo  In boxes in which they were delivered/Mu makarito byatangiwem 16. Where are L3-provided student books observed?/Ni hehe wabonye ibitabo by’abanyeshuri bya L3?  In student hands/Nabibonanye abanyeshuri  In classrooms on shelves/Mutubati mu mashuri  In teachers’ hands/Nabibonanye abarimu  In the library/Mu Isomero  In the headmaster office/Mu biro by’umuyobozi w’ikigo  In boxes in which they were delivered/Mu makarito byatangiwemo  None observed/Ntabyo nabonye 17. Do student books look used?/Ibitabo by’abanyeshuri bya L3 byaba bisa nibikoreshwa?  Yes, all look used/Yego, byose bisa nibikoreshwa  Yes, some look used/Yego, bimwe nibyo bisa nibikoreshwa  No/Oya 18. Comments/Andika ibindi waba wabonye:___________________ Teacher Questionnaire about Repeaters Page A147 TEACHER QUESTIONNAIRE ABOUT REPEATERS IN THE STUDY Province/Intara School District/ Akarera School name/Izina ry’ishuli _________ Student Name/Izina ry’umunyeshuri First Name/Izina ry’idini___________ Family Name/Izina ry’umuryango______________ Grade/Umwaka  P1  P2  P3 1. Do you know why this student was retained to repeat this grade?/Ni iyihe mpamvu yatumwe uyu mwana asibira?  Poor attendance/Gusiba ishuli cyane  Low grades/Amanota macye  Problems with behavior/Ibibazo by’imyitwarire  Problems with health/Ibibazo by’uburwayi/ubuzima  Student was too young/Uyu mwana yari mutoya cyane  Parent requested student repeat the grade/Byasabwe n’ababyeyi be  Other____ 2. Did the student attend school regularly this year?/Uyu mwana  Yes/Yego  No/Oya  Don’t know 3. Does this student have learning barriers? (e.g., poor vision or hearing, disability, chronic diseases)/Uyu mwana afite inzitize mu myigire ye (e.g. Kutareba neza cg kutumva, ubumuga, indwara idakira)?  yes, a lot/Yego, cyane  yes, some/Yego, gacye  no/Oya  Don’t know Teacher Questionnaire about Repeaters Page A-148 4. Is this child an orphan?/Uyu mwana ni imfubyi?  Yes/Yego  No/Oya  Don’t know 5. Did the student improve this year sufficiently to be promoted to the next grade next year?/Ukurikije imyigire ye uyu mwaka, urabona azimuka noneho?  Yes/Yego  No/Oya  Don’t know 6. Do you think this student might be at risk of dropping out?/Waba utekereza ko uyu mwana ashobora guta ishuli?  Yes/Yego  No/Oya  Don’t know 7. Please share what you know about this student and his/her family that might impact student’s attendance and performance at school?/Mwatubwira ibyo muzi kuri uyu mwana n’umuryango bigira ingaruka ku myigire ye? Thank you/Murakoze Detailed Results of Statistical Analysis of FARS and MARS Page A149 APPENDIX D. DETAILED RESULTS OF STATISTICAL ANALYSES OF FARS AND MARS MIDLINE RESULTS DETAILED RESULTS FOR FARS SUBTESTS Descriptive statistics for P1 FARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Oral Passage Reading (wcpm) 4.8 60.3 7.5 50.4 2.7 (± 1.1) Oral Passage Reading (pct) 17.2 60.3 25.9 50.4 8.6% (± 3.5%) Comprehension Questions (pct) 13.6 68.6 20.4 63.7 6.8% (± 3.3%) Descriptive statistics for P1 FARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Oral Passage Reading (wcpm) 5.1 60.0 8.4 45.2 3.3 (± 1.6) Oral Passage Reading (pct) 18.1 60.0 29.5 45.2 11.4% (± 5.2%) Comprehension Questions (pct) 14.0 69.1 23.0 59.8 8.9% (± 4.9%) Boys Oral Passage Reading (wcpm) 4.5 60.6 6.6 55.3 2.1 (± 1.5) Oral Passage Reading (pct) 16.4 60.6 22.4 55.3 5.9% (± 4.6%) Comprehension Questions (pct) 13.2 68.5 17.9 67.4 4.7% (± 4.4%) Descriptive statistics for P2 FARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Oral Passage Reading (wcpm) 19.2 32.7 21.5 25.5 2.3 (± 2.0) Oral Passage Reading (pct) 43.2 32.7 50.5 25.5 7.4% (± 4.4%) Comprehension Questions (pct) 44.5 37.7 29.7 40.6 -14.8% (± 4%) Descriptive statistics for P2 FARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Oral Passage Reading (wcpm) 22.2 26.9 24.8 22.3 2.6 (± 2.9) Oral Passage Reading (pct) 49.5 26.9 58.5 22.3 9.0% (± 6.4%) Comprehension Questions (pct) 48.8 33.1 32.5 34.4 -16.3% (± 5.7%) Detailed Results of Statistical Analysis of FARS and MARS Page A-150 Boys Oral Passage Reading (wcpm) 16.1 38.7 18.0 28.7 2.0 (± 2.7) Oral Passage Reading (pct) 36.5 38.7 42.5 28.7 5.9% (± 5.9%) Comprehension Questions (pct) 40.0 42.4 26.7 46.8 -13.3% (± 5.5%) Descriptive statistics for P3 FARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Oral Passage Reading (wcpm) 22.1 21.3 25.1 18.6 3.0 (± 1.8) Oral Passage Reading (pct) 37.5 21.3 44.7 18.6 7.2% (± 3.1%) Comprehension Questions (pct) 33.9 26.2 36.8 21.4 2.9% (± 2.9%) Descriptive statistics for P3 FARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Oral Passage Reading (wcpm) 23.5 19.6 28.4 14.7 4.9 (± 2.5) Oral Passage Reading (pct) 40.1 19.6 50.6 14.7 10.4% (± 4.3%) Comprehension Questions (pct) 34.9 24.8 39.6 16.7 4.7% (± 4.1%) Boys Oral Passage Reading (wcpm) 20.7 23.0 21.8 22.5 1.1 (± 2.5) Oral Passage Reading (pct) 34.9 23.0 38.9 22.5 3.9% (± 4.2%) Comprehension Questions (pct) 32.9 27.5 34.0 26.2 1.2% (± 4.1%) Descriptive statistics for P4 FARS subtests Baseline (SY 2014)* Midline (SY 2015) Subtest Mean % of students with zero scores Mean % of students with zero scores Kinyarwanda Oral Passage Reading (wcpm) -- -- 40.6 8.2 Kinyarwanda Oral Passage Reading (pct) -- -- 60.5 8.2 Kinyarwanda Comprehension Questions (pct) -- -- 56.5 11.3 English Oral Passage Reading (wcpm) -- -- 26.3 21.8 English Oral Passage Reading (pct) -- -- 41.9 22.5 English Comprehension Questions (pct) -- -- 19.6 58.3 *Data was not collected in SY 2014 for P4 given that L3 will not roll-out the intervention until SY 2016. Descriptive statistics for P4 FARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Gir lsKinyarwanda Oral Passage Reading (wcpm) -- -- 43.0 7.0 Kinyarwanda Oral Passage Reading (pct) -- -- 63.9 7.0 Detailed Results of Statistical Analysis of FARS and MARS Page A-151 Kinyarwanda Comprehension Questions (pct) -- -- 59.7 10.9 English Oral Passage Reading (wcpm) -- -- 27.4 20.3 English Oral Passage Reading (pct) -- -- 43.4 20.7 English Comprehension Questions (pct) -- -- 21.2 55.1 Boys Kinyarwanda Oral Passage Reading (wcpm) -- -- 38.2 9.3 Kinyarwanda Oral Passage Reading (pct) -- -- 57.0 9.3 Kinyarwanda Comprehension Questions (pct) -- -- 53.3 11.7 English Oral Passage Reading (wcpm) -- -- 25.1 23.3 English Oral Passage Reading (pct) -- -- 40.3 24.3 English Comprehension Questions (pct) -- -- 17.9 61.5 DETAILED RESULTS FOR MARS SUBTESTS Descriptive statistics for P1 MARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Task 1: Adding Numbers (pct) 22.4 41.0 43.6 23.1 21.2% (± 3.5%) Task 1: Adding Numbers (cpm) 2.3 41.0 4.5 23.1 2.2 (± 0.4) Task 2: Subtracting Numbers (pct) 15.1 58.5 39.2 32.7 24.1% (± 3.4%) Task 2: Subtracting numbers (cpm) 1.6 58.5 4.0 32.7 2.4 (± 0.4) Task 3. Comparing Numbers (pct) 39.6 19.2 60.8 9.9 21.2% (± 3.6%) Task 3. Comparing Numbers (cpm) 4.7 19.2 7.0 9.9 2.3 (± 0.8) Descriptive statistics for P1 MARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Task 1: Adding Numbers (pct) 17.7 45.0 45.7 18.9 28.0% (± 4.7%) Task 1: Adding Numbers (cpm) 1.8 45.0 4.6 18.9 2.8 (± 0.5) Task 2: Subtracting Numbers (pct) 12.5 62.4 40.2 30.5 27.7% (± 4.7%) Task 2: Subtracting numbers (cpm) 1.3 62.4 4.1 30.5 2.8 (± 0.5) Task 3. Comparing Numbers (pct) 38.3 18.6 62.1 7.8 23.8% (± 5.0%) Task 3. Comparing Numbers (cpm) 4.8 18.6 7.2 7.5 2.4 (± 1.4) Boys Task 1: Adding Numbers (pct) 26.9 37 41.6 27.0 14.7% (± 5.1%) Task 1: Adding Numbers (cpm) 2.8 37 4.3 26.8 1.5 (± 0.6) Task 2: Subtracting Numbers (pct) 17.6 54.7 38.3 34.7 20.8% (± 4.8%) Task 2: Subtracting numbers (cpm) 1.8 54.7 3.9 34.7 2.1 (± 0.5) Task 3. Comparing Numbers (pct) 40.9 19.8 59.7 12.0 18.8% (± 5.3%) Task 3. Comparing Numbers (cpm) 4.7 19.8 6.8 12.0 2.1 (± 0.7) Detailed Results of Statistical Analysis of FARS and MARS Page A-152 Descriptive statistics for P2 MARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Task 1: Adding Numbers (pct) 31.5 28.4 43.7 21.2 12.2% (± 3.7%) Task 1: Adding Numbers (cpm) 4.0 28.4 4.5 21.2 0.5 (± 0.6) Task 2: Subtracting Numbers (pct) 45.3 21.9 50.9 16.6 5.5% (± 3.9%) Task 2: Subtracting numbers (cpm) 5.7 21.4 5.4 16.6 -0.2 (± 0.6) Task 3. Multiplying Numbers (pct) 24.8 15.7 32.1 11.0 7.3% (± 2.5%) Task 3. Multiplying Numbers (cpm) 4.2 15.7 3.4 11.0 -0.8 (± 0.7) Descriptive statistics for P2 MARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Task 1: Adding Numbers (pct) 30.0 29.5 41.7 21.3 11.8% (± 5.1%) Task 1: Adding Numbers (cpm) 3.8 29.5 4.3 21.3 0.5 (± 0.9) Task 2: Subtracting Numbers (pct) 42.8 23.3 48.7 16.0 5.9% (± 5.3%) Task 2: Subtracting numbers (cpm) 5.3 22.4 5.2 16.0 -0.1 (± 0.9) Task 3. Multiplying Numbers (pct) 27.4 12.4 34.5 9.3 7.1% (± 3.5%) Task 3. Multiplying Numbers (cpm) 4.4 12.4 3.7 9.3 -0.8 (± 0.9) Boys Task 1: Adding Numbers (pct) 33.1 27.2 45.7 21.1 12.6% (± 5.4%) Task 1: Adding Numbers (cpm) 4.2 27.2 4.7 21.1 0.5 (± 0.8) Task 2: Subtracting Numbers (pct) 48.0 20.4 53.1 17.3 5.1% (± 5.6%) Task 2: Subtracting numbers (cpm) 6.0 20.4 5.7 17.3 -0.3 (± 0.9) Task 3. Multiplying Numbers (pct) 22.1 19.2 29.7 12.6 7.6% (± 3.5%) Task 3. Multiplying Numbers (cpm) 4.0 19.2 3.0 12.6 -0.9 (± 1.1) Descriptive statistics for P3 MARS subtests Baseline (SY 2014) Midline (SY 2015) Gain/Loss Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Task 1: Multiplying Numbers (pct) 45.5 10.1 52.2 9.9 6.7% (± 3.4%) Task 1: Multiplying Numbers (cpm) 5.5 10.1 3.6 9.9 -1.9 (± 0.5) Task 2: Dividing Numbers (pct) 26.8 26.1 32.2 27.8 5.3% (± 3.3%) Task 2: Dividing numbers (cpm) 3.5 26.0 2.2 27.8 -1.2 (± 0.4) Task 3: Adding Numbers (pct) 45.5 9.5 55.5 5.4 10.0% (± 3.1%) Task 3: Adding Numbers (cpm) 5.6 9.5 3.3 5.4 0.2 (± 0.5) Task 4: Subtracting Numbers (pct) 38.3 17.6 45.9 12.0 1.6% (± 3.1%) Task 4: Subtracting Numbers (cpm) 6.1 17.5 2.7 12.0 0.6 (± 1.2) Detailed Results of Statistical Analysis of FARS and MARS Page A-153 Descriptive statistics for P3 MARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Gain/Loss Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Mean (95% Confidence Interval) Girls Task 1: Multiplying Numbers (pct) 46.7 8.7 54.2 9.6 7.5% (± 4.9%) Task 1: Multiplying Numbers (cpm) 5.3 8.7 3.8 9.6 -1.6 (± 0.6) Task 2: Dividing Numbers (pct) 26.0 28.3 31.0 30.6 5.0% (± 4.7%) Task 2: Dividing numbers (cpm) 3.3 28.3 2.2 30.6 -1.1 (± 0.60 Task 3: Adding Numbers (pct) 42.3 9.7 53.6 5.6 11.3% (± 4.1%) Task 3: Adding Numbers (cpm) 4.8 9.7 3.1 5.6 -1.7 (± 0.5) Task 4: Subtracting Numbers (pct) 34.1 19.7 42.0 12.4 7.9% (± 4.2%) Task 4: Subtracting Numbers (cpm) 4.5 19.7 2.4 12.4 -2.1 (± 0.9) Boys Task 1: Multiplying Numbers (pct) 44.3 11.5 50.3 10.1 6.0% (± 4.8%) Task 1: Multiplying Numbers (cpm) 5.7 11.5 3.3 10.1 -2.3 (± 0.8) Task 2: Dividing Numbers (pct) 27.7 24.0 33.3 24.9 5.6% (± 4.8%) Task 2: Dividing numbers (cpm) 3.7 23.7 2.3 24.9 -1.4 (± 0.6) Task 3: Adding Numbers (pct) 48.7 9.4 57.4 5.2 8.7% (± 4.5%) Task 3: Adding Numbers (cpm) 6.4 9.4 3.4 5.2 -2.9 (± 0.8) Task 4: Subtracting Numbers (pct) 42.6 15.5 49.9 11.7 7.3% (± 4.6%) Task 4: Subtracting Numbers (cpm) 7.8 15.3 2.9 11.7 -4.9 (± 2.2) Descriptive statistics for P4 MARS subtests Baseline (SY 2014)* Midline (SY 2015) Subtest Mean % of students with zero scores Mean % of students with zero scores Task 1: Adding Numbers (pct) -- -- 77.0 2.6% Task 1: Adding Numbers (cpm) -- -- 10.0 2.6 Task 2: Subtracting Numbers (pct) -- -- 62.9 7.8% Task 2: Subtracting Numbers (cpm) -- -- 8.2 7.8 Task 3: Multiplying Numbers (pct) -- -- 51.8 5.6% Task 3: Multiplying Numbers (cpm) -- -- 7.1 5.6 Task 4: Dividing Numbers (pct) -- -- 33.1 19.7% Task 4: Dividing numbers (cpm) -- -- 4.9 19.7 Task 5: Comparing Numbers (pct) -- -- 34.6 27.6% Task 5: Comparing Numbers (cpm) -- -- 6.3 27.6 *Data was not collected in SY 2014 for P4 given that L3 will not roll-out the intervention until SY 2016. Detailed Results of Statistical Analysis of FARS and MARS Page A-154 Descriptive statistics for P4 MARS subtests, by sex Baseline (SY 2014) Midline (SY 2015) Sex Subtest Mean % of students with zero scores Mean % of students with zero scores Girls Task 1: Adding Numbers (pct) -- -- 74.0 2.6% Task 1: Adding Numbers (cpm) -- -- 9.4 2.6 Task 2: Subtracting Numbers (pct) -- -- 58.7 8.4% Task 2: Subtracting Numbers (cpm) -- -- 7.5 8.4 Task 3: Multiplying Numbers (pct) -- -- 52.0 4.1% Task 3: Multiplying Numbers (cpm) -- -- 7.0 4.1 Task 4: Dividing Numbers (pct) -- -- 32.0 20.4% Task 4: Dividing numbers (cpm) -- -- 4.5 20.4 Task 5: Comparing Numbers (pct) -- -- 35.1 27.8% Task 5: Comparing Numbers (cpm) -- -- 6.3 27.8 Boys Task 1: Adding Numbers (pct) -- -- 80.0 2.7% Task 1: Adding Numbers (cpm) -- -- 10.6 7.3 Task 2: Subtracting Numbers (pct) -- -- 67.0 7.3% Task 2: Subtracting Numbers (cpm) -- -- 8.8 7.3 Task 3: Multiplying Numbers (pct) -- -- 51.6 7.0% Task 3: Multiplying Numbers (cpm) -- -- 7.2 7.0 Task 4: Dividing Numbers (pct) -- -- 34.3 19.0% Task 4: Dividing numbers (cpm) -- -- 5.3 19.0 Task 5: Comparing Numbers (pct) -- -- 34.2 27.4% Task 5: Comparing Numbers (cpm) -- -- 6.3 27.4