April 24, 2020 This publication was prepared for review by the United States Agency for International Development. It was prepared by Simon King and Jennifer Pressley of RTI International. All Children Reading – Asia (ACR-Asia) All Children Reading–Cambodia Student Performance in Early Literacy: Midterm Impact Report All Children Reading – Asia (ACR-Asia) All Children Reading – Cambodia Student Performance in Early Literacy: Midterm Report Activity Start Date and End Date: September 30, 2016, to September 29, 2021 Contract No. AID-OAA-TO-16-00017 Number: REQ-ASIA-16-00017 Prepared for Mitch Kirby USAID/Asia Bureau, Senior Education Advisor 1300 Pennsylvania Ave, NW Washington, DC 20523 And USAID / Cambodia Mission, Education Team Prepared by Simon King and Jennifer Pressley RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709-0155 Tel: (919) 541-6000 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. All Children Reading–Cambodia, Midterm Impact Report iii Acknowledgements The authors of this report, on behalf of the All Children Reading–Cambodia activity, wish to acknowledge the contribution of all the collaborators who made it possible to complete this midterm survey. First and foremost, it is important to appreciate the close collaboration of the Ministry of Education, Youth and Sport. Several offices in Phnom Penh, including the Primary Education Department, the Education Quality Assurance Department, the Department of Curriculum Development, and the Teacher Training Department, were instrumental in supporting the finalization of the instruments and helping organize the survey. The Education Quality Assurance Department also provided significant inputs and support during data collection, which was much appreciated. We wish also to acknowledge the school directors, teachers, and students of the schools who gave generously of their time to enable us to collect the necessary data successfully. Organizations that partnered with All Children Reading–Cambodia and made important contributions to the preparation and implementation of this midterm survey include Real-Time Evaluations, Kampuchean Action for Primary Education, World Education International, Save the Children, and the Open Institute. All Children Reading–Cambodia, Midterm Impact Report iv Table of Contents Page Acknowledgements......................................................................................................iii List of Exhibits.............................................................................................................. v Acronyms....................................................................................................................vii Executive Summary ..................................................................................................... 1 1. Background ...................................................................................................... 4 2. Design of the Study .......................................................................................... 6 2.1 Research Questions ............................................................................. 6 2.2 Measuring Impact ................................................................................. 6 2.3 Sampling............................................................................................... 7 2.4 Instruments........................................................................................... 8 2.4.1 Survey Instruments ................................................................... 8 2.4.2 Pre-literacy Assessment for Upper Pre-school Students ........... 8 2.4.3 Early Grade Reading Assessment for Grade 1 and 2 Students . 9 2.4.4 Early Grade Mathematics Assessment for Grade 1 Students .. 10 2.4.5 School Survey ......................................................................... 10 2.4.6 Instrument Reliability and Validity............................................ 11 3. Main Results...................................................................................................11 3.1 Impact Evaluation ...............................................................................11 3.1.1 Pre-school Intervention A in Kampong Thom .......................... 12 3.1.2 Grade 1, Interventions A, B, and C.......................................... 12 3.1.3 Findings by Gender................................................................. 17 3.1.4 School-Level Findings ............................................................. 19 3.2 Baseline Findings for Grade 2 and Kampot, Grade 1 .......................... 20 3.2.1 Upper Pre-school Khmer—Kampot Baseline........................... 20 3.2.2 Grade 1 Khmer—Kampot Baseline.......................................... 21 3.2.3 Grade 2 Khmer—Kampong Thom and Siem Reap Baseline ... 22 3.2.4 Baseline Balance—Kampot..................................................... 26 3.2.5 Grade 1 Mathematics—Siem Reap ......................................... 26 3.3 Student Age and School Characteristics ............................................. 30 4. Conclusions....................................................................................................33 Annex 1: Instruments Used in the Assessments ........................................................ 35 Annex 2: Reliability and Validity .................................................................................53 All Children Reading–Cambodia, Midterm Impact Report v List of Exhibits Exhibit ES-1: Pre-school Impact............................................................................................ 1 Exhibit ES-2: Grade 1 Impact (selected subtasks) ................................................................ 2 Exhibit 1: Pilot Intervention Rollout Plan................................................................................ 4 Exhibit 2: Comparison Groups Included in the Evaluation Design for the Pilot Implementation of the National Early Grade Learning Program............................................... 5 Exhibit 3: Impact Calculation................................................................................................. 7 Exhibit 4: Number of Schools Sampled—End of School Year 2018–2019............................. 8 Exhibit 5: Program Impact on Learning Outcomes, Pre-school ........................................... 12 Exhibit 9: Program Impact on Letter Knowledge, Grade 1................................................... 13 Exhibit 10: Distribution of Vowel Scores (out of 23)—Three Interventions Combined, Baseline and Midterm Grade 1..................................................................... 14 Exhibit 11: Percent Correct of Vowel Items—Three Interventions Combined, Baseline and Midterm Grade 1.......................................................................................... 14 Exhibit 12: Percent Correct of Consonant Items—Three Interventions Combined, Baseline and Midterm Grade 1 ................................................................................... 15 Exhibit 13: Program Impact on Recognizing Words and Reading and Comprehending a Text Passage, Grade 1........................................................................................ 16 Exhibit 14: Distribution of Passage Reading Fluency—Kampong Thom Intervention A, Baseline and Midterm Grade 1..................................................................... 17 Exhibit 15: Learning Gains by Gender for Vocabulary Score Percent Between Baseline and Midterm Grade 1, Kampong Thom—Intervention A...................................... 18 Exhibit 16: Change in Zero Scores by Gender for Oral Reading Fluency Between Baseline and Midterm Grade 1, Kampong Thom Intervention A ................................. 18 Exhibit 17: School-Level Average Letters per Minute Gains, Kampong Thom Intervention A Grade 1........................................................................................................ 19 Exhibit 18: Mean Scores and Percent of Upper Pre-school Students Scoring Zero, Kampot and Battambang Provinces .......................................................................... 20 Exhibit 19: Mean Scores and Percent of Grade 1 Students Scoring Zero for Letter Subtasks, Kampot And Battambang Provinces............................................................. 21 Exhibit 20: Mean Scores and Percent of Grade 1 Students Scoring Zero for Word Reading and Comprehension Subtasks, Kampot and Battambang Provinces............ 22 Exhibit 21: Mean Scores and Percent of Grade 2 Students Scoring Zero for Letter Subtasks, Kampong Thom and Siem Reap Interventions and Control Group............... 22 Exhibit 22: Distribution of Vowel and Consonant Subtasks—All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2....................................... 23 Exhibit 23: Mean Scores and Percent of Grade 2 Students Scoring Zero for Word Reading and Comprehension Subtasks, All Treatment Groups Combined (Kampong Thom A & B and Siem Reap)....................................................................... 24 Exhibit 24: Distribution of Letter and Word Knowledge and Passage Reading Fluency Subtasks— All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2 ................................................................................... 25 All Children Reading–Cambodia, Midterm Impact Report vi Exhibit 25: Item Scores for Dictation Subtask — All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2 ........................................................ 26 Exhibit 26: Zero Scores and Averages of Grade 1 Students for EGMA Subtasks, Siem Reap and Battambang Provinces .......................................................................... 27 Exhibit 27: Item Scores for Number Identification Subtask—Siem Reap and Battambang Combined, Grade 1...................................................................................... 27 Exhibit 28: Item Scores for Number Discrimination Subtask—Siem Reap and Battambang Combined, Grade 1...................................................................................... 28 Exhibit 29: Item Scores for Addition and Subtraction Subtasks—Siem Reap and Battambang Combined, Grade 1...................................................................................... 29 Exhibit 30: Average Math Subtask Score by Gender........................................................... 30 Exhibit 31: Student Age Category (Self-Reported), All Grades and Provinces .................... 31 Exhibit 32: Average Vowels Score Percent for Grade 1, Kampong Thom and Siem Reap for Age Classifications Between Baseline and Midterm..................................... 31 Exhibit 33: Means for School Variables by Comparison Group, 2019.................................. 32 Exhibit 34: Percent Correct of Vowel Items—Kampong Thom and Siem Reap Combined, Baseline, and Midterm Grade 1 and Baseline Grade 2 ................................ 34 Exhibit A2-1: Grade 1 Early Grade Reading Assessment.................................................... 53 Exhibit A2-2: Grade 1 Early Grade Mathematics Assessment............................................. 53 All Children Reading–Cambodia, Midterm Impact Report vii Acronyms EGMA Early Grade Mathematics Assessment EGR early grade reading EGRA Early Grade Reading Assessment GPE-3 Global Partnership for Education—third grant MoEYS Ministry of Education, Youth and Sport NGO non-governmental organization USAID United States Agency for International Development All Children Reading–Cambodia, Midterm Impact Report 1 Executive Summary This report is a first midterm evaluation of the United States Agency for International Development (USAID) All Children Reading–Cambodia program, which supports the Ministry of Education, Youth and Sport (MoEYS) in the development and piloting of a national early grade reading program. This report presents the evaluation of two USAID-funded intervention groups in Kampong Thom and the Global Partnership for Education—third grant- (GPE-3-) funded intervention group in Siem Reap for pre￾school and grade 1 after one year of implementation. The program pilots were implemented using a phased approach, beginning with the literacy pilot rolled out for grade 1 in 2018–2019 for all intervention groups, with the addition of pre-school in one of the Kampong Thom treatment groups. Grade 1 math and grade 2 literacy will be rolled out in 2019–2020. In addition, the program will also be used in Kampot with USAID funding. Consequently, this report also details baseline outcomes for grade 1 mathematics, grade 2 literacy, and grade 1 literacy in Kampot. The report addresses the following research questions: 1. What are the impacts of the pilot programs in upper pre-school and grade 1 on basic pre-literacy and early literacy skills? 2. In what ways does this impact differ for boys and girls? 3. Are there differences in the impacts on student learning outcomes in each of the four comparison groups? 4. What are some defining characteristics of the schools included in the study in Kampot? 5. Are there differences in those characteristics compared with the other groups? 6. How do students perform in grade 1 on basic mathematics skills? 7. In what ways do those levels of performance differ for boys and girls? The baseline assessment was conducted in June 2018 with the follow-up midterm in June 2019. The exhibits in this executive summary respond to research questions 1 and 3, assessing program impact after one year. The program impact for pre-school grades is shown in Exhibit ES-1 which compares the pre-school intervention in Kampong Thom with the Battambang control group. Exhibit ES-1: Pre-school Impact Subtask Treatment Mean Score Impact Baseline 2018 Midterm 2019 Expressive Vocabulary (mean correct words out of 40) Kampong Thom Intervention A 9.3 10.4 -0.3 Battambang Control 10.8 12.2 Identifying Letters (mean correct letters out of 20) Kampong Thom Intervention A 5.8 5.2 -0.1 Battambang Control 5.6 5.1 Listening Comprehension (mean correct questions out of 5) Kampong Thom Intervention A 3.0 2.5 -0.8** Battambang Control 2.8 3.1 ** p<0.01 All Children Reading–Cambodia, Midterm Impact Report 2 The Impact column displays the intervention gain between baseline and midterm averages minus the corresponding average gain from the control group. The exhibit shows that there was no intervention impact for expressive vocabulary and identifying letters subtasks. The control group significantly gained over the intervention group for listening comprehension with a small net gain of an additional 0.8 correct questions out of five. The program impact for grade 1 is shown in Exhibit ES-2. This table compares each of the three grade 1 interventions with the Battambang control group. The program impact is again displayed in the final column. Reading instruction for grade 1 in Cambodia has a strong focus on learning consonants and vowels, which is where some of the biggest reading skills gains were demonstrated at midterm. Exhibit ES-2: Grade 1 Impact (selected subtasks) Subtask Intervention Mean Scores Impact 2018 Baseline 2019 Midterm Mean Consonant Identification (mean percent correct out of 33) Kampong Thom Intervention A 41.8 57.3 12.0*** Kampong Thom Intervention B 47.7 55.6 4.5 Siem Reap Intervention C 46.6 50.8 0.8 Battambang Control 47.0 50.4 3 Treatment Groups Combined 46.4 53.1 3.2 Vowel Identification (mean percent correct out of 23) Kampong Thom Intervention A 35.1 56.3 18.4*** Kampong Thom Intervention B 36.2 53.6 14.5*** Siem Reap Intervention C 42.9 57.5 11.8*** Battambang Control 42.0 44.9 3 Treatment Groups Combined 39.8 56.1 13.4*** Familiar Word Reading Fluency (number correct per minute) Kampong Thom Intervention A 1.6 3.3 1.7*** Kampong Thom Intervention B 1.5 2.4 0.9* Siem Reap Intervention C 2.4 2.5 0.1 Battambang Control 2.3 2.3 3 Treatment Groups Combined 2.0 2.5 0.5 Text Reading—Oral Reading Fluency (words correct per minute) Kampong Thom Intervention A 0.8 2.3 1.3** Kampong Thom Intervention B 0.9 1.6 0.6 Siem Reap Intervention C 1.1 1.3 0.1 Battambang Control 1.2 1.3 3 Treatment Groups Combined 1.0 1.5 0.4 * p<0.05, ** p<0.01, *** p<0.001 For each subtask, Kampong Thom Intervention A demonstrated the strongest and most statistically significant gains compared to the Battambang control group. This is particularly evident with vowel identification, where students correctly identified vowels at a rate 18.4 percentage points higher. It is important to note that the interventions were different in design, as will be detailed later in this report. The gains for text reading fluency were modest; for example, reading fluency had a small impact of an additional 1.3 correct words per minute for Kampong Thom Intervention A. This gain should not be surprising in grade 1; we would expect the programs to have a bigger impact on word knowledge and text reading in grade 2, next year. This impact is likely to be amplified by the fact that students were tested more than one month before the end of the school year (this was necessary during the baseline due to the national election in July 2018). All Children Reading–Cambodia, Midterm Impact Report 3 It may also be helpful to the MoEYS to consider the overall scores when results are combined across treatment groups. The combined pilot program impact on vowel recognition for grade 1 students must be highlighted. At the 2018 grade 1 baseline, 50% or more of the students combined across the pilot treatment groups could correctly identify only 4 out of the 23 items. At midterm the number of items that at least 50% of the students could recognize increased to 18 out of 23. Further, grade 1 students participating in the pilot program outperformed the grade 2 students (who had not yet received the intervention)in 15 out of the 23 vowel items. This indicates that the pilot programs are well on their way to giving grade 1 students the letter knowledge foundation needed to build their skills in grade 2 in word knowledge, passage reading, and comprehension. Girls participating in the pilot program demonstrated stronger reading gains across all reading subtasks, including a more rapid reduction in zero scores. At the school-level of analysis, many schools were demonstrating reasonable improvements in learning outcomes. However, there were still many struggling to have an impact. It would be good to further investigate the main challenges individual schools that are struggling may face to further improve learning gains. All Children Reading–Cambodia, Midterm Impact Report 4 1. Background USAID’s All Children Reading–Cambodia is designed to support the Cambodian MoEYS in developing and piloting a national early grade reading program. The project has helped the MoEYS update the curriculum and materials for grade 1 instruction in Khmer literacy and helped develop supplemental materials for use in upper pre-school. USAID supports piloting of this work in Kampong Thom and Kampot Provinces. Concurrently, GPE-3 funding is supporting implementation in Siem Reap Province. The rollout of the interventions evaluated in this report is shown in Exhibit 1, below. Exhibit 1: Pilot Intervention Rollout Plan Location School Year School Year School Year 2018–2019 2019–2020 2020–2021 Kampong Thom Province (Interventions A & B) Khmer Pre-school (Intervention A) Khmer Grade 2 Math Grade 1 Khmer Grade 1 Siem Reap Province Intervention C Khmer Grade 1 Khmer Grade 2 Math Grade 1 Math Grade 2 Kampot Province Intervention D Khmer Pre-school Khmer Grade 2 Khmer Grade 1 Math Grade 1 This phased rollout approach, starting with program implementation in just grade 1 and pre-school in 2018–2019, facilitates more focused program design and implementation. Consequently, this midterm report, using data collected at the end of the 2017–2018 and 2018–2019 school years, focuses on the impact of the reading program pilots after one year of implementation for pre-school in Kampong Thom (Intervention A), and grade 1 in Kampong Thom (Interventions A and B) and Siem Reap. The objective of this impact evaluation is to make an early determination of how the interventions are contributing to improved learning outcomes in early grade reading in Cambodia. This early insight into program impact will inform both improvements to implementation and provide evidence for planning the scale-up to additional provinces. The grade 2 intervention for reading will be rolled out in the 2019–2020 school year as part of the Kampong Thom and Siem Reap pilots, so this report will also present the baseline for this grade. MoEYS is also piloting a mathematics program in Siem Reap starting with grade 1 in 2019–2020. This report will provide a baseline for this math pilot, as well. Finally, the USAID All Children Reading– Cambodia project will support introduction of the national early grade reading program in Kampot starting in the 2019–2020 school year. Therefore, this report also provides baseline data for the pre-school and grade 1 Khmer implementation in Kampot. MoEYS, donor partners, and other stakeholders agreed to pilot different intervention approaches, facilitating a comparison of different program supports and thus generating a better understanding of how to operationalize and optimize support for early grade reading and mathematics programs at a national level. The summary details for the pilot interventions are shown in Exhibit 2, below. All Children Reading–Cambodia, Midterm Impact Report 5 Exhibit 2: Comparison Groups Included in the Evaluation Design for the Pilot Implementation of the National Early Grade Learning Program Control USAID-Funded GPE-3 Funded Intervention A Intervention B Intervention D Intervention C Location Battambang Kampong Thom Kampong Thom Kampot Siem Reap (6 districts) (2 districts) (6 districts) (all districts) Upper No Yes No Yes No Pre-school Textbooks Existing books Existing books Existing books Existing books Existing books Teacher Activity Guide No Provided for reading and math Provided for reading and math Provided for reading and math Provided for reading and math Supplementary Materials No Provided Provided Provided Provided Training No Staggered, Khmer starting in year 1 and math in year 3 Staggered, Khmer starting in year 1 and math in year 3 Staggered, Khmer in year 1 and math in year 2 Staggered, Khmer in year 1 and math in year 2 Coaching/ No Twice per month Six visits per year from senior mentor Twice per month Six visits per year from senior Mentoring mentor from NGO coaches from NGO coaches In Siem Reap, the approach is to support teachers with school-based senior mentors who will interact with teachers roughly six times during the school year. In Kampong Thom, All Children Reading–Cambodia and MoEYS implement that same model of support in two districts (Intervention B) and a model of more intensive teacher support and coaching in the other six districts (Intervention A). The USAID-supported Kampong Thom Intervention A also includes support to upper pre-school. This includes supplementary materials and training for pre-school teachers to support improvements in pre-literacy skill development. To capture the effect of this effort, this midterm report also looks at performance of upper pre-school students in the six districts included in Kampong Thom Intervention A to then compare their performance to that of upper pre-school students in the control province of Battambang. The Kampot Province pilot, which will be implemented a year later, mirrors the program design of Kampong Thom Intervention A and will be informed by the results of this evaluation. The midterm survey of student performance was designed to enable comparison across the groups shown in Exhibit 2 after one year of implementation of the national early learning program and then again at future points—as explained in this section. Following discussion of the survey design and methodology, Section 2 presents the study design and instruments that were used to collect information on student performance and on each school’s context, and Section 3 presents the study findings. Section 4 presents the authors’ conclusions. The baseline evaluation is available on the EGR Barometer. All Children Reading–Cambodia, Midterm Impact Report 6 This report is the companion document to the baseline study which can be downloaded from the Early Grade Reading (EGR) Barometer, under “About the Data.”1 2. Design of the Study 2.1 Research Questions All Children Reading – Cambodia and MoEYS designed this study to answer a simple set of questions. 1. What are the impacts of the pilot programs in upper pre-school and grade 1 on basic pre-literacy and early literacy skills? 2. In what ways does this impact differ for boys and girls? 3. Are there differences in the impacts on student learning outcomes in each of the four comparison groups? 4. What are some defining characteristics of the schools included in the study in Kampot? 5. Are there differences in those characteristics compared with the other groups? 6. How do students perform in grade 1 on basic mathematics skills? 7. In what ways do those levels of performance differ for boys and girls? 2.2 Measuring Impact The impact evaluation was designed to answer the above research questions for the pilot study. This design included a comparison group based in Battambang. The comparison group provides a comparison for “business as usual”—how the reading and mathematics learning outcomes in the early grades are progressing without a pilot intervention. By accounting for this comparison group, this report details the added value of the pilot interventions above typical early grade reading and mathematics instruction in Cambodia’s public schools. The control area was carefully selected to be comparable to the areas of intervention. Similarity was further ensured by selecting a control group with similar socio-economic and urban/rural makeup as the intervention area and by balancing learning outcomes post-baseline. The impact evaluation measures all intervention groups at two (or more) timepoints, returning to the same schools and grades and sampling new students each time. This cross-sectional design measures the impact of improved teaching on student learning outcomes while maintaining the comparability between intervention areas. Sampling weights were applied to all analysis; this combined with the proportional sampling approach ensured that estimates of learning outcomes were representative of the intended areas of intervention. 1 The EGR Barometer is available at https://dev.earlygradereadingbarometer.org/cambodia/countries/technical_information . All Children Reading–Cambodia, Midterm Impact Report 7 A typical difference-in-differences analysis was applied to measure impact. This is a calculation of the difference between midterm average and baseline average for the control and pilot treatment groups, as shown in Exhibit 3, below. Exhibit 3: Impact Calculation If control and treatment averages were not balanced, a control model difference-in￾differences would be calculated; however, that was not necessary for the impact evaluation presented in this report. Impact was assessed in two formats, first, as a difference-in-differences in the unit of assessment, such as percent correct responses or correct words per minute. Second, the impact will be shown as a standardized average, specifically a Cohen’s d effect size. Cohen’s d is calculated by dividing the difference-in-differences by the pooled standard deviation. An effect size of 0.2 is considered a small effect, 0.5 a medium effect, and 0.8 a large effect.2 This effect size ranking creates a useful comparable measure by standardizing the unit of impact, so with careful consideration of context, impacts can be compared across program pilots and other studies. 2.3 Sampling To answer the research questions, All Children Reading–Cambodia collected data from schools in each of the four comparison groups. The school sample was drawn to detect an effect size of 0.2 standard deviation difference between performance in the intervention groups and performance in the control group. When sampling school clusters, variability across and within schools is an important consideration of estimating an optimal and efficient sample size. It was found that at baseline, the intra-cluster correlation (which compared the variance within and between school clusters) to be 0.07, lower than the 0.2 used to estimate sample sizes. This means that for this impact evaluation, the sample size was sufficient. The sampling was conducted using probability proportional to school size, resulting in a sample with demographic characteristics very similar to the population it was intended to represent. The number of schools and students sampled at midterm is shown in Exhibit 4, below. The same schools were also sampled at baseline with the exception of Kampot, for which these data are an initial baseline. 2 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, N.J.: L. Erlbaum Associates. All Children Reading–Cambodia, Midterm Impact Report 8 Exhibit 4: Number of Schools Sampled—End of School Year 2018–2019 Treatment Sampled Number of Schools Sampled Number of Students with Pre￾school without Pre￾school Total Pre-school Grade 1 Grade 2 Boys Girls Boys Girls Boys Girls Total Kampong Thom Intervention A 21 43 64 82 87 255 256 246 259 1185 Kampong Thom Intervention B 80 80 317 310 321 303 1251 Siem Reap Intervention C 58 58 231 234 227 233 925 Kampot Intervention D 28 42 70 111 112 278 280 781 Battambang Control 27 40 67 105 106 266 265 262 263 1267 Total 76 263 339 298 305 1347 1345 1056 1058 5409 A total of 339 schools were sampled. Of these schools, 76 had pre-schools attached. Schools with and without pre-schools were sampled proportionately to the number of schools in the population area. Changes were made to the sampled school lists if a selected school was unavailable for data collection. To maintain sample size, a matched replacement school was selected; in other words, a school nearby the unavailable school with similar characteristics such as school size was used to replace the unavailable school. This approach maintained the representativeness of the sample. When field data collection teams visited schools, attention was paid to make sure roughly the same number of students were sampled by gender to enable a large enough sample size to detect differences in boys’ and girls’ performance. Approximately 8 students were sampled per grade, per school. The total number of students sampled at midterm was 603 in pre-school, 2,692 in grade 1, and 2,114 in grade 2, for a total of 5,409 students. 2.4 Instruments 2.4.1 Survey Instruments The midterm study employed the same instruments used in the baseline for pre￾school and grade 1 students. These instruments were: • Pre-literacy skills assessment for pre-school students • Early Grade Reading Assessment (EGRA) for grade 1 and 2 students • School resources survey For grade 2, a dictation writing subtask was added to the EGRA. All Children Reading-Cambodia also developed and piloted an Early Grade Mathematics Assessment (EGMA) for grade 1. These instruments are discussed below and are included in Annex 1 of this report. 2.4.2 Pre-literacy Assessment for Upper Pre-school Students The project developed the pre-literacy instrument to assess learning outcomes for upper pre-school students. Prior to the baseline assessment, three tasks were taken from an assessment already adapted for Cambodia and for the upper pre-school age All Children Reading–Cambodia, Midterm Impact Report 9 group and translated into Khmer (the IDELA3 instrument), then piloted. The three subtasks in the instrument assessed the pre-literacy skills below. Expressive vocabulary—students were asked to name food items and animals (up to 20 of each). Letter name identification—students were asked to name 20 Khmer letters, consonants only. Listening comprehension—students listened to a short, simple story and were verbally asked to respond to five comprehension questions about the story. The questions required both explicit and inferential comprehension. 2.4.3 Early Grade Reading Assessment for Grade 1 and 2 Students EGRA is an oral assessment of pre-reading and reading skills that serve as a foundation for student academic success. The Khmer EGRA instrument used for this assessment was adapted and piloted in collaboration with MoEYS and several non￾governmental organizations (NGOs). There is a fundamental difference between the Khmer EGRA and EGRAs in other languages, namely that the students are assessed on letters in additional separate vowel and consonant subtasks. These subtasks were added at the request of MoEYS as these literacy skills are an essential component of literacy teaching and learning in grade 1, and students are expected to have mastered these letter components by the end of the grade. In the Khmer EGRA, students were given a stimulus sheet that had the letters for each individual letters subtask arranged randomly in a grid, along with a grid of familiar words and a text passage for oral reading. The following EGRA subtasks were used as a diagnostic of student pre-literacy and literacy skills. Consonant letter identification—students were asked to correctly identify 33 consonant letters. If students did not know the answer, the enumerator moved to the next letter, but students would have the chance to read all letters. Vowel letter identification—students were asked to correctly identify 23 vowel letters. Again, students had the opportunity to read all vowels. Letter identification (advanced)—students were given a 10 by 10 grid of letters and asked to identify as many letters as possible in 60 seconds. Their score was then converted into a correct letters per minute rate. Familiar words—students were given a 5 by 10 grid of words and asked to read as many as they could in 60 seconds. Their score was then converted into a correct words per minute rate. The words chosen for this subtask were frequently used in the grade 1–3 textbooks. Oral reading fluency—students were given a simple reading passage (61 words) and given up to three minutes to read. Their score was converted to a correct words per minute rate. 3 Pisani, L., Borisova, I., & Dowd, A.J. (2017). International development and early learning assessment: Technical working paper. Washington DC: Save the Children. Available at https://resourcecentre.savethechildren.net/library/international-development-and-early-learning￾assessment-technical-paper Vowel Identification Stimulus Consonant Identification Stimulus All Children Reading–Cambodia, Midterm Impact Report 10 Reading comprehension—students were given up to three minutes to completely read a simple reading passage and then asked five reading comprehension questions to assess their ability to understand the meaning of the text they had just read. Dictation—students were read two short sentences and asked to write them on paper. This writing subtask was used for grade 2 students only. Partial credit (half a point) scoring was also possible for each item. For letter identification, familiar words, and oral reading fluency, if a child could not identify or read the first few designated items, the subtask was discontinued. 2.4.4 Early Grade Mathematics Assessment for Grade 1 Students EGMA was designed to provide information about basic mathematics competencies that students should be mastering in the early grades. The subtasks used in an EGMA cover early numeracy skills students need in order to progress academically. The numeracy skills and abilities demonstrated form part of the foundation for students to be able to solve more advanced problems and facilitate the acquisition of more advanced mathematics skills. The criteria of the subtask selection for the EGMA include, but are not limited to • skills that are predictive of future academic success and proficiency in mathematics, • skills that can be improved through classroom instruction, and • skills that meet international standards of numeracy competency in the early grades. EGMA is administered orally with individual students so that each child’s ability to complete a subtask is not dependent on his or her ability to read. In this way mathematics skills are assessed independent of reading ability. The subtasks for EGMA are described below. The instrument is displayed in Annex 1. Number identification—students identify up to 20 one- and two-digit numbers in one minute. The score is converted to a correct problems per minute rate. Number discrimination—students identify the larger of 10 pairs of one- or two-digit numbers. The score is presented as percent correct. Addition level 1—students solve 20 items of addition of one- and two-digit numbers. The score is converted to a correct items per minute rate. Subtraction level 1—students solve 20 items of subtraction of one- and two-digit numbers. The score is converted to a correct items per minute rate. These subtasks were developed in collaboration with MoEYS and other stakeholders and were judged appropriate assessments for grade 1. All Children Reading– Cambodia has made recommendations on which subtasks should be included when the MoEYS conducts the grade 2 baseline. 2.4.5 School Survey The final instrument developed for the evaluation was a school-level survey. To differentiate between the conditions in schools, the survey gathered objective information on factors such as school size, presence of a library, basic infrastructure Oral Reading Passage Subtask All Children Reading–Cambodia, Midterm Impact Report 11 (e.g., toilets, electricity, water source), etc. This approach is similar to that used in large cross-national assessments such as the Trends in International Mathematics and Science Study, which differentiate student “advantages” at the school level of analysis. This approach reduced the assessment time for the children. Additionally, these data collected were used to assess and balance the treatment groups with the control group at baseline and differentiate learning gains by school demographic groupings. This survey is included in Annex 1. 2.4.6 Instrument Reliability and Validity Internal consistency is an appropriate and standard classical evaluation approach for cross-sectional data. Cronbach’s alpha was 0.84 for the reading subtasks and 0.89 for the mathematics subtasks. Cronbach’s alpha should be at least 0.70 for adequacy, and coefficients closer to 1 indicate a good assessment.4 Technical details about instrument reliability and validity testing are presented in Annex 2 of this report. 3. Main Results The results presented in this section vary according to the intervention rollout plans detailed in Exhibit 1. Consequently, the focus of this section of the report is to • evaluate the impact of the Khmer reading pilot for Kampong Thom Intervention A after one year of the pre-school program; • evaluate the impact of the Khmer reading pilots in Kampong Thom Interventions A and B and Siem Reap Intervention C after one year of the grade 1 program; and • present baseline findings for − Khmer Reading pre-school for Kampot Intervention D, − Khmer Reading grade 1 for Kampot Intervention D, − Khmer Reading grade 2 for Kampong Thom Interventions A and B and Siem Reap Intervention C, and − Mathematics grade 1 for Siem Reap Intervention C. The next USAID All Children Reading–Cambodia evaluation report in 2020 will report impact for all the baseline findings presented in this report alongside a second year impact evaluation of the programs being evaluated after one year in this report. This report will present findings in the order presented in the bullets above. 3.1 Impact Evaluation Presented in this sub-section are the impact evaluation findings after one year of implementation of reading programs in Kampong Thom Interventions A and B in pre￾school and grade 1, and in Siem Reap Intervention C in grade 1. As mentioned earlier in this report, it is important that we contextualize the outcomes in terms of support provided to the different interventions. 4 Aron, A., Coups, E.J., Aron, E.N. (2010). Statistics for the Behavioral and Social Sciences: A Brief Course, 5 edition. ed. Boston: Pearson. All Children Reading–Cambodia, Midterm Impact Report 12 3.1.1 Pre-school Intervention A in Kampong Thom Students on average were not showing any significant gains after one year of program support for pre-school. The impact of one year of pre-school intervention for Kampong Thom Intervention A is shown in Exhibit 5, below. Exhibit 5: Program Impact on Learning Outcomes, Pre-school Subtask Treatment Mean Scores Program Impact Baseline 2018 Midterm 2019 Average Effect Size Expressive Vocabulary Kampong Thom Intervention A 9.3 10.4 -0.3 0.05 (mean correct out of 40) Battambang Control 10.8 12.2 Identifying Letters Kampong Thom Intervention A 5.8 5.2 -0.1 0.03 (mean correct out of out of 20) Battambang Control 5.6 5.1 Listening Comprehension Kampong Thom Intervention A 3.0 2.5 -0.8** 0.59 (mean correct out of 5) Battambang Control 2.8 3.1 ** p<0.01 The data in Exhibit 5 show that there was no intervention impact for the expressive vocabulary and identifying letters subtasks, with statistically insignificant differences between the mean scores for both. Since letter knowledge was a focus of the intervention, this result deserves further investigation by the project team to understand how teachers have been teaching letters. Furthermore, average listening comprehension in the treatment schools declined slightly, while it increased a minor amount in control schools. This is also a finding that bears further exploration, The project staff will need to work with the MoEYS to reflect on and consider how the preschool intervention can be strengthened. 3.1.2 Grade 1, Interventions A, B, and C The EGRA used a total of six reading skills subtasks to assess learning impact for grade 1. This report separates them, as shown in Exhibits 9 and 10, using assessment of letters or word reading as the identifying feature. Letter Knowledge The learning impact on letter knowledge for the grade 1 pilot programs is shown in Exhibit 9, below. Three subtasks are featured; the first two assess vowels and consonants separately, the third combines vowels and consonants and is a timed assessment. All Children Reading–Cambodia, Midterm Impact Report 13 Exhibit 9: Program Impact on Letter Knowledge, Grade 1 Subtask Treatment Percent Scoring Zero Mean Score Baseline 2018 Midterm 2019 Impact Versus Control Baseline 2018 Midterm 2019 Impact Versus Control Effect Size Versus Control Vowels (percent correct) Kampong Thom Intervention A 18.2% 8.0% -12.3%*** 35.1% 56.3% 18.4%*** 0.56 Kampong Thom Intervention B 14.9% 8.3% -8.7%** 36.2% 53.6% 14.5%*** 0.45 Siem Reap Intervention C 13.7% 8.6% -7.1%* 42.9% 57.5% 11.8%*** 0.35 Battambang Control 10.5% 12.6% 42.0% 44.9% Consonants (percent correct) Kampong Thom Intervention A 9.5% 3.8% -3.4% 41.8% 57.3% 12.0%*** 0.37 Kampong Thom Intervention B 7.8% 4.8% -0.8% 47.7% 55.6% 4.5% 0.14 Siem Reap Intervention C 7.7% 6.0% 0.6% 46.6% 50.8% 0.8% 0.02 Battambang Control 7.6% 5.3% 47.0% 50.4% Correct Letters per Minute Kampong Thom Intervention A 15.1% 10.2% -2.3% 8.6 12.6 3.2** 0.33 Kampong Thom Intervention B 17.3% 12.5% -2.2% 9.1 11.2 1.3 0.14 Siem Reap Intervention C 19.4% 11.2% -5.6% 9.5 10.6 0.4 0.04 Battambang Control 12.6% 10.0% 9.8 10.6 * p<0.05, ** p<0.01, *** p<0.001 The pilot program impact on students’ letter knowledge was particularly significant on vowel knowledge. Kampong Thom Intervention A improved the average percent of vowels correctly identified by over 18 percentage points in a single year. This corresponds with a medium effect size of 0.56. Impact on consonant knowledge was also good; only the impact of Kampong Thom A was statistically significant (p<0.001). The correct letters per minute had more moderate impact, with a statistically significant increase of 3.2 additional correct letters per minute only in Kampong Thom Intervention A. The difference between this subtask and the vowels and consonants subtasks is accounted for by some combination of increased difficultly and that 60 seconds was given students in the third subtask to recognize as many letters as possible, compared with untimed vowels and consonants subtasks. The distribution of scores for vowels, combining the three intervention groups, is shown in Exhibit 10, below. All Children Reading–Cambodia, Midterm Impact Report 14 Exhibit 10: Distribution of Vowel Scores (out of 23)—Three Interventions Combined, Baseline and Midterm Grade 1 As shown in Exhibit 10, 47% of the students were able to identify at least 16 out of 23 vowels, compared with 24% at baseline. It is encouraging that more students were able to demonstrate competency with letter recognition at midterm, as this literacy skill provides a good foundation on which to build more advanced reading skills. At baseline, there were only four vowels that more than 50% of students could identify. The progress against this benchmark is shown in Exhibit 11; at least 50% of the students could recognize 18 of the 23 vowels at midterm. Exhibit 11: Percent Correct of Vowel Items—Three Interventions Combined, Baseline and Midterm Grade 1 The improvement in vowel recognition was quite uniform; the overall improvement in the percent correct score for vowels is explained by an overall improvement in all the items, rather than just a few items showing the biggest gains. 16% 29% 16% 15% 16% 8% 8% 17% 13% 15% 26% 21% 0% 10% 20% 30% 40% 50% 0 1-4 5-10 11-15 16-20 21-23 Percentage of Students Vowels Score out of 23 Baseline 2018 Midterm 2019 0 10 20 30 40 50 60 70 80 90 100 ◌ា ◌ួ ◌ិ ◌ី េ◌ៅ ែ◌ ◌ុ េ◌ៀ ៃ◌ ◌ា◌ំ ◌ូ ◌ំ េ◌ឿ េ◌ោ◌ះ េ◌ោ េ◌ េ◌ើ េ◌◌ះ ◌ុ◌ះ ◌ុ◌ំ ◌ះ ◌ឺ ◌ឹ Percentage Correct Percentage Correct - Vowels Item Level 2018 % Correct 2019 Change Vowels that 50% or fewer of students could identify. All Children Reading–Cambodia, Midterm Impact Report 15 At baseline, 12 of the 33 consonants were correctly identified by at least 50% of grade 1 students. Exhibit 12 demonstrates the student progress on this literacy skill after one year of the literacy programs. Fifty percent or more of students could recognize at least 18 items. As it is with vowel identification, this improvement is helping to lay the groundwork for students to be able to move on to higher level literacy skills. Exhibit 12: Percent Correct of Consonant Items—Three Interventions Combined, Baseline and Midterm Grade 1 The black bars in Exhibit 12 indicate that at midterm fewer students were able to identity two uncommon consonants than during the baseline testing, but these differences are small, not statistically significant, and do not likely warrant cause for concern. Recognizing Words and Reading Comprehension Recognition of familiar words and the ability to read and comprehend a passage of text are more advanced reading skills. The impact of the interventions on these skills among the grade 1 students are shown in Exhibit 13. 0 10 20 30 40 50 60 70 80 90 100 ក គ ខ ច ស ត ទ ដ រ ង ប ឆ ល ជ ព ញ អ ណ យ ម វ ឡ ហ ធ ឈ ន ភ ឃ ថ ឋ ឌ ឍ ផ Percentage Correct Consonants Item Level 2018 2019 Gain Consonants that 50% or fewer of students could identify. Decreases for 2 letters below shown in black All Children Reading–Cambodia, Midterm Impact Report 16 Exhibit 13: Program Impact on Recognizing Words and Reading and Comprehending a Text Passage, Grade 1 Subtask Treatment Percent Scoring Zero Mean Score Baseline 2018 Midter m 2019 Impact Versus Control Baseli ne 2018 Midterm 2019 Impact Versus Control Effect Size Versus Control Correct Familiar Words per Minute Kampong Thom A 75.1% 63.4% -11.4%** 1.6 3.3 1.7*** 0.32 Kampong Thom B 77.5% 72.2% -5.0% 1.5 2.4 0.9* 0.18 Siem Reap C 68.1% 62.4% -5.4% 2.4 2.5 0.1 0.01 Battambang Control 67.2% 66.9% 2.3 2.3 Oral Reading Fluency (correct words per minute) Kampong Thom A 87.7% 77.2% -11.7%*** 0.8 2.3 1.3** 0.29 Kampong Thom B 87.3% 83.9% -4.6% 0.9 1.6 0.6 0.12 Siem Reap C 86.4% 81.3% -6.3% 1.1 1.3 0.1 0.02 Battambang Control 83.2% 84.4% 1.2 1.3 Reading Comprehen￾sion (score out of 5) Kampong Thom A 91.7% 80.7% -9.2%*** 0.1 0.4 0.2*** 0.35 Kampong Thom B 91.7% 86.4% -3.6% 0.1 0.3 0.1* 0.17 Siem Reap C 91.4% 86.0% -3.6% 0.1 0.3 0.1 0.12 Battambang Control 88.4% 86.7% 0.2 0.2 * p<0.05, ** p<0.01, *** p<0.001 The program impact on familiar words and passage reading and comprehension is far more modest than that seen for letter knowledge. The biggest impact is Kampong Thom Intervention A, which saw an increase from 1.7 to 3.3 correct words per minute, with an effect size of 0.32. Kampong Thom Intervention A had the most statically significant impacts, with reductions in zero scores for all three subtasks highly significant (p<0.0001). That the impacts on average word and text reading are modest is no real surprise considering the focus in grade 1 teaching of literacy in Cambodia was on letter knowledge. The reduction in zero scores does indicate that the intervention is beginning to help students who would otherwise be at the bottom of the distribution. In addition, considering the high level of baseline zero scores, there is no reason to expect high percentages of students in grade 1 to be reading text with comprehension. Next year as impact of grade 2 is reported, one would hope for a different story as grade 2 builds on the pilot progress demonstrated in grade 1. The change in distribution between baseline and midterm for oral passage reading is shown in Exhibit 14, below. All Children Reading–Cambodia, Midterm Impact Report 17 Exhibit 14: Distribution of Passage Reading Fluency—Kampong Thom Intervention A, Baseline and Midterm Grade 1 Exhibit 14 confirms the 11% reduction in zero scores, and most of these students were performing at the 1–20 correct words per minute rate at midterm, meaning they had progressed to sounding out and recognizing words, but had not yet progressed to reading a passage fluently with accuracy. 3.1.3 Findings by Gender An important research question was to determine if there was a performance difference by gender. The baseline report determined that girls were outperforming boys. Following on from that, this evaluation determines whether there was a difference in learning gains by gender at midterm. It is important to note that the sample size calculations for this study were focused on overall impact aggregated at the pilot program level; gender comparisons would mostly be underpowered. This proved to be the case, and while girls’ learning gains were stronger than boys’ in every subtask without exception, hardly any of these differences proved to be statistically significant. Two of these findings are shown in the next two exhibits. To determine if a pilot program was more beneficial to a single gender, the analysis looked at the learning gains by gender within the programs, irrespective of the control group’s performance. Exhibit 15 shows the learning gains in the vowel knowledge score percent between baseline and midterm for Kampong Thom Intervention A by gender. 88% 10% 2% 0% 0% 77% 15% 5% 2% 1% 0% 25% 50% 75% 100% 0 1-10 11-20 21-30 31-40 Percentage of students Oral Reading Fluency, Correct Words per Minute Baseline 2018 Midterm 2019 All Children Reading–Cambodia, Midterm Impact Report 18 Exhibit 15: Learning Gains by Gender for Vocabulary Score Percent Between Baseline and Midterm Grade 1, Kampong Thom—Intervention A The exhibit shows the difference in performance by gender is getting slightly more pronounced as time progresses. In just one year, the girls improved their average score percent by 4.6 percentage points over the boys. A similar pattern was seen across all subtasks and intervention groups. While no gender showed huge gains in the word reading subtasks, the decrease in the percentage of girls scoring zero was greater than for the boys. Exhibit 16, below shows the change in zero scores in oral reading fluency for Kampong Thom Intervention A. Exhibit 16: Change in Zero Scores by Gender for Oral Reading Fluency Between Baseline and Midterm Grade 1, Kampong Thom Intervention A The overall decrease in zero scores for this population was about 11%. However, the girls’ decrease was a full 6.1 percentage points more than the boys (13.5% versus 7.4%). gain +18.9 gain +23.5 0 20 40 60 80 Baseline 2018 Midterm 2019 Vowels Score Percent Male Female decrease 7.4 decrease 13.5 0 20 40 60 80 100 Baseline 2018 Midterm 2019 Percenrage of Students Scoring Zero, Oral Reading Fluency Male Female All Children Reading–Cambodia, Midterm Impact Report 19 3.1.4 School-Level Findings The cross-sectional design used for this evaluation collects data from the same sampled schools at multiple points in time but resamples students at those schools each year. Therefore, while student data do not accommodate any longitudinal analysis, the data can be reshaped to facilitate a school-level analysis. This is a useful exercise as it breaks down the school cluster impact on the overall pilot program gain; for example, how many schools are showing learning outcome improvement and how many are having challenges in implementation? This analysis also helps programs focus on the learning gains as opposed to midterm performance; for example, if a school with a low baseline student achievement average score is showing strong learning gains, but is still behind schools at midterm because it is “catching-up,” we should most likely consider this to be a positive result. The hope would be that over the life of the pilot program, these schools with low baseline learning outcomes can narrow the gap with schools that had much higher baseline scores. Exhibit 17 shows the aggregated school-level learning gains for correct letters per minute, for Kampong Thom Intervention A. Each arrow is a school and the ends of the arrow are the baseline and midterm average student performance. Exhibit 17: School-Level Average Letters per Minute Gains, Kampong Thom Intervention A Grade 1 0 10 20 30 40 50 Letter Identification, Average Correct Letters per Minute Kampong Thom Intervention A Schools Average gain - 3.2 clpm Schools with above average gain Schools with below average gain All Children Reading–Cambodia, Midterm Impact Report 20 The limitation of this analysis does need to be considered: due to the small sample size per school (an average of eight students sampled), some of the school-level arrows suffer from natural variability or “noise.” This is most likely evident with the few schools at the bottom of the exhibit that show the average score decreasing. It is possible that the students in this school were not demonstrating lower performance, rather the small sample creates this variability. However, until these schools are visited, it would be difficult to ascertain if learning outcomes are truly decreasing. Depending on how one classifies a “performing school,” the percentage of schools considered to be performing will vary according to opinion. However, what the exhibit clearly shows is that the school-level gain is independent of the baseline average. In other words, no matter the baseline average performance of the school and the demographic challenges faced by the school and community, once enrolled in the pilot program, each school has the same opportunity to support students’ learning. 3.2 Baseline Findings for Grade 2 and Kampot, Grade 1 This section presents the findings for program components that will be implemented in the school year 2019–2020. The full rollout plan is described earlier in this report in Exhibit 1. Specifically, this section presents baseline findings for the following: • Khmer pre-literacy for upper pre-school for Kampot Intervention D • Khmer reading grade 1 for Kampot Intervention D • Khmer reading grade 2 for Kampong Thom Interventions A and B and Siem Reap Intervention C • Mathematics grade 1 for Siem Reap Intervention C The baseline design for all the above interventions included a comparison group in Battambang Province. Impact for all these interventions will be shown in the second midterm report in 12 months. 3.2.1 Upper Pre-school Khmer—Kampot Baseline Implementation of the Khmer reading program in Kampot Province will commence in school year 2019–2020. The program, with support from USAID, will be similar to the intervention implemented in Kampong Thom Intervention A as shown in Exhibit 1. Once teachers have been initially trained, they will receive coaching visits twice per month from NGOs supporting the program. Exhibit 18 below shows the average student score and percentage of students scoring zero for the three pre-school subtasks. Battambang will again be used as a counterfactual to control for “business as usual” in schools. Exhibit 18: Mean Scores and Percent of Upper Pre-school Students Scoring Zero, Kampot and Battambang Provinces Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Expressive Vocabulary (mean correct out of 40) Kampot Intervention D 2.2% 10.2 [±1.5] Battambang Control 0.5% 12.2 [±1.0] Identifying Letters (mean correct out of out of 20) Kampot Intervention D 30.5% 4.5 [±1.1] Battambang Control 26.8% 5.1 [±1.1] Listening Comprehension (mean correct out of 5) Kampot Intervention D 5.8% 2.7 [±0.2] Battambang Control 2.4% 3.1 [±0.2] All Children Reading–Cambodia, Midterm Impact Report 21 The scores are very similar to the baseline pre-school scores for Kampong Thom and Battambang last year, where the baseline reported average scores of 10.3, 28.3%, and 2.9 for vocabulary, letter knowledge, and listening comprehension, respectively. Taking the confidence intervals into account, there was no significant difference between the baseline pre-school scores for Kampot and those for Kampong Thom. The only significant difference was the percentage of students scoring zero on letter knowledge. The baseline in Kampong Thom Intervention A and Battambang combined the percent of zero scores was 20.7%, and this baseline for Kampot is 30.5%. The Kampot pilot was implemented a year later, so true comparability to the previous year’s baseline should be done with caution. 3.2.2 Grade 1 Khmer—Kampot Baseline The grade 1 intervention will be rolled out alongside upper pre-school for the Kampot pilot in 2019–2020. This section of the report presents the baseline outcomes for the Kampot pilot. As was the case for pre-school, Battambang Province was used as the control group. The outcomes paint a similar picture as the prior baseline for Kampong Thom and Siem Reap for letter subtasks, as shown in Exhibit 19. Exhibit 19: Mean Scores and Percent of Grade 1 Students Scoring Zero for Letter Subtasks, Kampot And Battambang Provinces Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Vowels (score percent) Kampot Intervention D 15.8% 34.4% [±3.6%] Battambang Control 12.6% 44.9% [±4.1%] Consonants (score percent) Kampot Intervention D 6.5% 38.8% [±3.7%] Battambang Control 5.3% 50.4% [±4.0%] Correct Letters per Minute Kampot Intervention D 15.6% 7.5 [±1.0%] Battambang Control 10.0% 10.6 [±1.1%] The learning outcomes demonstrate relatively few students scoring zero; the highest rate of zero scores was 15.6% of the students in Kampot unable to identify a letter on the vowel and consonant combined letter fluency subtasks. The overall mean scores on the combined baseline from 2018 were 34.1%, 46.5%, and 9.3 correct letters per minute for the vowels, consonants, and combined letters subtasks. The Battambang averages for grade 1 in 2019 were consistently on the high end of these estimates, which raises a concern regarding baseline balance. This will be discussed in the grade 2 baseline sections of this report. As with the 2018 baseline for grade 1, the baseline 2019 EGRA also assessed the literacy skills of familiar word recognition, passage reading, and comprehension. These results are presented in Exhibit 20, below. All Children Reading–Cambodia, Midterm Impact Report 22 Exhibit 20: Mean Scores and Percent of Grade 1 Students Scoring Zero for Word Reading and Comprehension Subtasks, Kampot and Battambang Provinces Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Correct Familiar Words per Minute Kampot Intervention D 78.6% 1.2 [±1.0] Battambang Control 66.9% 2.3 [±1.1] Oral Reading Fluency (correct words per minute) Kampot Intervention D 94.5% 0.5 [±0.3] Battambang Control 84.4% 1.3 [±0.4] Reading Comprehension (score out of 5, extended time) Kampot Intervention D 96.4% 0.1 [±0.0] Battambang Control 86.7% 0.2 [±0.1] As with the other subtasks, the outcomes reflect the previous baseline in 2018. The high zero scores on oral reading fluency and reading comprehension indicate students’ skills in these areas are very underdeveloped. As with the midterm for Kampong Thom and Siem Reap, it is most likely that if there is impact with the Kampot pilot in grade 1, it will be seen mostly in learning outcome improvement for the letter knowledge subtasks, rather than for word recognition and reading and comprehension. 3.2.3 Grade 2 Khmer—Kampong Thom and Siem Reap Baseline This section of the report offers the first look at findings for grade 2 learning outcomes. The pilot programs in Kampong Thom and Siem Reap will include grade 2 in the 2019–2020 school year, so the midterm assessment included a baseline for grade 2 in these provinces. As before, Battambang served as a control group. Exhibit 21 shows the grade 2 student learning outcomes for the letter subtasks. The mean scores present a reasonable level of letter knowledge at the end of grade 2. Exhibit 21: Mean Scores and Percent of Grade 2 Students Scoring Zero for Letter Subtasks, Kampong Thom and Siem Reap Interventions and Control Group Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Vowels (score percent) Kampong Thom Int. A 4.9% 54.0% [±4.1%] Kampong Thom Int. B 3.6% 51.7% [±3.3%] Siem Reap Int. C 5.4% 59.0% [±4.3%] Battambang Control 3.0% 61.5% [±4.0%] 3 Treatment Groups Combined 4.8% 56.1% [±2.7%] Consonants (score percent) Kampong Thom Int. A 1.5% 66.9% [±4.0%] Kampong Thom Int. B 2.0% 68.4% [±3.4%] Siem Reap Int. C 1.1% 68.3% [±3.9%] Battambang Control 1.7% 73.3% [±3.5%] 3 Treatment Groups Combined 1.5% 68.2% [±2.5%] Correct Letters per Minute Kampong Thom Int. A 5.9% 19.0 [±2.0] Kampong Thom Int. B 4.3% 18.6 [±1.6] Siem Reap Int. C 8.0% 20.6 [±1.8] Battambang Control 5.8% 20.5 [±1.7] 3 Treatment Groups Combined 6.6% 19.8 [±1.2] All Children Reading–Cambodia, Midterm Impact Report 23 The percentage of grade 2 students unable to recognize a single letter was low across all subtasks and treatment groups, with the highest being 8% of students in Siem Reap unable to recognize a letter in the fluency subtask. The mean scores were also good, but this should be the case at the end of grade 2, which should have a greater focus on word reading and comprehension. Letter recognition should be mastered at the end of grade 1. The distribution of the percent correct scores shown in Exhibit 22 confirms the general mastery of these skills at the end of grade 2; the left skewed distributions indicate that many students on the right of the distributions were scoring sufficiently well. Exhibit 22: Distribution of Vowel and Consonant Subtasks—All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2 As discussed above, students entering grade 2 should have mastered letter knowledge in grade 1 and move on to learning familiar words and text passage reading and comprehension. Next year, the initial cohort of students who received the pilot intervention in Kampong Thom or Siem Reap in grade 1 will also receive the pilot intervention in grade 2, allowing for evaluation of learning gains for students after two years of intervention. The results presented for grade 2 in this report are of students who did not receive any intervention in grade 1. Consequently, they did not benefit from improved letter knowledge learning instruction in grade 1. Therefore, while the outcomes in grade 2 presented in Exhibit 23 show students who are progressing in their reading fluency, those students are generally not yet at a level of mastery. 1% 4% 5% 5% 6% 6% 5% 9% 9% 14% 36% 4% 7% 8% 6% 9% 7% 6% 10% 10% 13% 19% 0% 20% 40% 60% 0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 Percentage of Students Score Percent Consonants Score Percent Vowels Score Percent All Children Reading–Cambodia, Midterm Impact Report 24 Exhibit 23: Mean Scores and Percent of Grade 2 Students Scoring Zero for Word Reading and Comprehension Subtasks, All Treatment Groups Combined (Kampong Thom A & B and Siem Reap) Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Correct Familiar Words per Minute Kampong Thom Int. A 37.1% 11.5 [±2.1] Kampong Thom Int. B 39.9% 11.2 [±1.7] Siem Reap Int. C 33.1% 13.4 [±2.2] Battambang Control 30.6% 13.5 [±1.7] 3 Treatment Groups Combined 35.8% 12.5 [±1.3] Oral Reading Fluency (correct words per minute) Kampong Thom Int. A 43.0% 11.9 [±2.6] Kampong Thom Int. B 50.3% 12.1 [±2.1] Siem Reap Int. C 46.3% 14.5 [±2.6] Battambang Control 41.7% 14.8 [±2.2] 3 Treatment Groups Combined 47.2% 13.4 [±1.6] Reading Comprehension (percent score out of 5, extended time) Kampong Thom Int. A 49.7% 1.2 [±0.2] Kampong Thom Int. B 54.5% 1.2 [±0.2] Siem Reap Int. C 46.7% 1.4 [±0.2] Battambang Control 43.0% 1.5 [±0.2] 3 Treatment Groups Combined 49.6% 1.3 [±0.1] Dictation (score out of 10) Kampong Thom Int. A 39.2% 2.3 [±0.4] Kampong Thom Int. B 40.6% 2.3 [±0.4] Siem Reap Int. C 38.6% 2.4 [±0.4] Battambang Control 38.8% 2.3 [±0.3] 3 Treatment Groups Combined 39.3% 2.4 [±0.3] The mean scores for reading fluency are in the 12–15 correct words per minute range. This means that on average the students were reading slowly and methodically, probably taking their time to sound out words and recognize familiar ones. Consequently, they were struggling to understand the meaning of the text. This interpretation is reinforced by three points, that for the three treatment groups combined: • 72% of the grade 2 students were reading at, or below, 20 correct words per minute, as shown in Exhibit 24, and • Average reading comprehension scores (students had 180 seconds to read a short passage) were no higher than 1.5 correct responses on average out of 5 questions. • Only 14.2% of students achieved a score of over 4 out of 5 correct answers about the passage. All Children Reading–Cambodia, Midterm Impact Report 25 Exhibit 24: Distribution of Letter and Word Knowledge and Passage Reading Fluency Subtasks— All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2 The grade 2 EGRA added a dictation subtask. Students were read two short sentences and asked to write them down. The sentences roughly translated as: “The teacher / is good at / explaining / the lesson” and “The farmer / transplants / rice / in / rainy / season.” The first sentence was marked out of four and the second out of six. Students were read both sentences and were scored as “correct,” “partially correct,” or “incorrect” by the assessor. Partial credit solutions were awarded a half mark. Exhibit 23 shows that on average students scored about 2.4. The item distribution in Exhibit 25 below shows that no more than 38% of the students were able to correctly write a word item. 46% 15% 11% 10% 7% 4% 3% 2% 1% 1% 0% 10% 20% 30% 40% 50% 60% 0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 Percentage of Students Fluency Rate Correct letters per minute Correct Familiar words per minute Oral Reading Fluency All Children Reading–Cambodia, Midterm Impact Report 26 Exhibit 25: Item Scores for Dictation Subtask — All Treatment Groups Combined (Kampong Thom A & B and Siem Reap), Grade 2 For the second sentence, which was more challenging than the first, at most 24% of the students were able to correctly write one of the words. 3.2.4 Baseline Balance—Kampot As explained in Section 2, part of the objective of a baseline study is to assess, create and maintain balance between treatment and control groups to ensure an “apples to apples” comparison. The study assessed balance by looking at the effect size difference between the treatment and control baseline grade 2 estimates for reading fluency. It was found that this difference was 0.24 standard deviations. This value is within acceptable range, but What Works Clearinghouse Standards 4.0 recommends running the difference-in-differences impact analysis using a regression control model.5 This is something that will be considered for the follow-up midterm report. 3.2.5 Grade 1 Mathematics—Siem Reap The Siem Reap pilot also included a mathematics teaching and learning component. As shown in Exhibit 1, this will roll out for grade 1 in the 2019–2020 school year. Therefore, this report will present the baseline learning outcomes. For this assessment, an EGMA tool was used with four subtasks described earlier in this report. The zero scores and averages for the four subtasks in Exhibit 26 show that the students at the end of grade 1 were demonstrating just a rudimentary understanding of numeracy. 5 The What Works Clearinghouse Standards Handbook, version 4.0, is available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf 49 57 76 62 69 74 79 74 85 79 38 30 7 29 8 13 12 24 10 10 14 13 17 9 24 13 9 2 5 12 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ្រគ ពូែក ពន្យល់ េមេរៀន កសិករ ស ្រ វ េនៅ រដូវ វស្សោ Student Response Percentage Dictation - Item Response Incorrect Correct Partially Correct All Children Reading–Cambodia, Midterm Impact Report 27 Exhibit 26: Zero Scores and Averages of Grade 1 Students for EGMA Subtasks, Siem Reap and Battambang Provinces Subtask Treatment Percent Scoring Zero Mean Scores Confidence Interval Number Identification (correct items per minute) Siem Reap Intervention C 3% 9.6 [±1.1] Battambang Control 4% 10.2 [±1.0] Number Discrimination (mean correct out of out of 10) Siem Reap Intervention C 24% 3.4 [±0.4] Battambang Control 23% 3.6 [±0.4] Addition Level 1 (correct items per minute) Siem Reap Intervention C 45% 3.2 [±0.5] Battambang Control 40% 3.7 [±0.5] Subtraction Level 1 (correct items per minute) Siem Reap Intervention C 60% 1.9 [±0.4] Battambang Control 60% 2.3 [±0.4] Number identification is the simplest of mathematics subtasks, students are given one minute to correctly identify 20 single- and two-digit numbers. On average, students were able to correctly identify 10 numbers, which is about 1 number correctly identified every six seconds, which is quite slow for a skill that should be automatic at the end of grade 1. In Exhibit 27, the item scores for number identification show that students started well in correctly identifying single-digit numbers (the first three items), then struggled on two-digit numbers. For the number discrimination subtask, for each item students had to say which of a pair of numbers was larger. The average number of problems correct out of 10 was about 3.5, indicating students were limited in the ability to compare the quantities represented by different numbers. Exhibit 27: Item Scores for Number Identification Subtask—Siem Reap and Battambang Combined, Grade 1 For addition and subtraction subtasks, students were given one minute (for each subtask) to correctly solve up to 20 items. Exhibit 26 shows the percentage of students with zero scores to be a concern: 40% or more were unable to solve a single addition problem, and 60% could not do subtraction. 94 70 92 69 58 47 51 28 38 48 26 22 25 18 20 16 26 25 18 23 6 30 8 31 42 53 50 72 62 52 74 78 75 82 80 84 74 75 82 77 0 10 20 30 40 50 60 70 80 90 100 2 9 0 12 18 30 22 45 39 23 48 66 57 74 91 83 52 65 87 79 Percent of Students Attempting Item Correct Number Identification - Items %Correct %Incorrect All Children Reading–Cambodia, Midterm Impact Report 28 Exhibit 28 shows a pattern similar to that of the previous subtask—students struggling with understanding quantity. While around 60% of students could discriminate between single-digit numbers, they struggled to even attempt discrimination of two-digit numbers. For example, around 76% of students attempted the fourth item (39 or 23) and of them, only 43% correctly identified 39 as a larger number. Exhibit 28: Item Scores for Number Discrimination Subtask—Siem Reap and Battambang Combined, Grade 1 Grade 1 students attempting the addition and subtraction subtasks were given one minute to correctly answer as many items as possible. If they had already learned mental techniques of addition and had some of the simple addition and subtraction memorized through practice, students would be able to move through the simple addition tasks quickly. However, this was evidently not frequently the case as almost half the students did not get beyond the fifth item on both subtasks as shown in Exhibit 29, below. This indicates that most students were struggling, most likely because their teachers are failing to fully develop their techniques for doing addition and subtraction. 66 62 58 40 43 24 32 33 41 63 34 38 42 60 57 76 68 67 59 37 0 10 20 30 40 50 60 70 80 90 100 7 or 5 2 or 9 16 or 6 11 or 24 39 or 23 49 or 58 65 or 67 74 or 47 66 or 86 94 or 78 Percentage of Students Attempting Item Number Discrimination - Items %Correct %Incorrect All Children Reading–Cambodia, Midterm Impact Report 29 Exhibit 29: Item Scores for Addition and Subtraction Subtasks—Siem Reap and Battambang Combined, Grade 1 Differences by gender for the grade 1 mathematics achievement at baseline are explored here. The findings present a contrast to the literacy skill achievement. In Exhibit 30, the average subtask score is shown for grade 1 by gender. 51 43 27 30 43 62 37 29 38 31 8 17 26 22 24 23 33 28 26 30 49 57 74 70 57 38 63 71 63 69 92 83 74 78 76 77 67 72 74 70 0 10 20 30 40 50 60 70 80 90 100 1+3 2+3 6+2 4+5 3+3 8+1 7+3 3+9 2+8 9+3 7+8 4+7 7+5 8+6 9+8 6+7 8+8 8+5 10+2 8+10 Percentage of Students Attempting Item Addition Level 1 - Items %Correct %Incorrect 32 31 19 21 23 62 40 11 30 19 7 10 15 13 17 13 13 18 16 7 68 69 81 79 77 38 60 89 70 81 93 90 85 87 84 88 87 82 84 93 0 10 20 30 40 50 60 70 80 90 100 4-3 5-3 8-2 9-5 6-3 9-1 10-3 12-9 10-8 12-3 15-8 11-7 12-5 14-6 17-8 13-7 16-8 13-5 12-2 18-0 Percentage of Students Attempting Item Subtraction Level 1 - Items %Correct %Incorrect All Children Reading–Cambodia, Midterm Impact Report 30 Exhibit 30: Average Math Subtask Score by Gender *p<0.01 Opposite the gender differences in pre-literacy and literacy skills, gender differences in mathematics show the boys, on average, outperforming the girls in mathematics, significantly so for demonstrating addition and subtraction skills. In literacy, Exhibits 23 and 24 show how the gender gap increased after one year, with girls improving their literacy skills at a greater rate than boys. It will be important to see how this gap between genders changes for mathematics after one year. 3.3 Student Age and School Characteristics The last set of findings from the midterm survey concerns the student ages and characteristics of the schools in the different comparison groups: Kampong Thom A, Kampong Thom B, Siem Reap, Kampot, and Battambang. Students were asked their birthdate during the survey. Approximately 79% of students in grades 1 and 2 were able to provide a date. Using this information, it was determined from their response if they were at-age, under-age, or over-age for their grade. This information, presented in Exhibit 31, shows an increasing number of students were classified as over-age as they progressed through the grades, starting with 27% in pre-school, rising to 53% by the time they reached grade 2. 9.4 10.1 3.0 3.7* 1.8 2.3* 3.3 3.6 0 2 4 6 8 10 12 Female Male Female Male Female Male Female Male Number Idenitification (correct problems per minute) Additiona Level 1 (correct problems per minute) Subtraction Level 1 (correct problems per minute) Number Discrimination (out of 10) Correct problems per minute / score Mathematics Subtask All Children Reading–Cambodia, Midterm Impact Report 31 Exhibit 31: Student Age Category (Self-Reported), All Grades and Provinces We did not determine why more and more students were over-age. It would be interesting to find out how many of these students repeated a grade due to absences or poor scores on teacher assessments. The pilot programs were also struggling to help these over-age students keep pace. Exhibit 32 shows the average vowels score percent for grade 1 students by age classification between baseline and midterm. Note that the under-age students represented a small sample size, not large enough to present reliable estimates. Exhibit 32: Average Vowels Score Percent for Grade 1, Kampong Thom and Siem Reap for Age Classifications Between Baseline and Midterm Students who were over-age for their grade, for whatever reason, were struggling to keep up with their at-age peers year on year; Exhibit 32 shows that the at-age students gained 4 percentage points over their over-age peers in just one year. It is possible that the factors that cause students to be over-age may be the same ones that cause them to fall behind in performance. < 5 years, 13% <6 years, 8% < 7 years, 5% 5 to 6 years, 60% 6 to 7 years, 49% 7 to 8 years, 42% > 6 years, 27% > 7 years, 44% > 8 years, 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Preschool Grade 1 Grade 2 Age Category by Grade 2019 (All Provinces) Under Age Of Age Over Age gain +17.9 gain +13.9 0 10 20 30 40 50 60 2018 baseline 2019 midterm Vowels Score Percent Of Age for Grade Over Age for Grade All Children Reading–Cambodia, Midterm Impact Report 32 School Characteristics This evaluation included a survey of school characteristics to understand the context of schools in each of the treatment groups (Exhibit 33). These data are useful in attempting to explain differences that may occur among or between treatment groups, if there are marked differences. Some characteristics that stand out from this analysis include the following: • Most schools were of moderate size, having an average enrollment in grades 1–3 of 231 among all treatment groups and a range of 132 to 336. • Over 40% of schools had attached pre-schools in most treatment groups, except in Siem Reap, where 93% of schools had attached pre-schools. • NGO support for early grade reading was high among all intervention groups at midterm, with a lower percentage of schools receiving support in Battambang, the control province. Interestingly, Kampot at this baseline had very little NGO support, with less than 9% of schools reporting receiving NGO support for early grade reading. • Most schools had enough Khmer textbooks for grade 1, but the percent of schools with storybooks in the grade 1 classroom varied across treatment groups. • Most schools had electricity, functioning toilets, and a safe water source, but very few schools across treatment groups had piped water as their water source. Exhibit 33: Means for School Variables by Comparison Group, 2019 Mean Values Kampong Thom A Kampong Thom B Siem Reap Kampot Battam￾bang Average percent schools that are complete primary (1–6) 87.50% 88.75% 100.00% 97.14% 82.09% Average total enrollment for grades 1–3 (combined) 221.2 132.4 336 232.4 268.1 Average total number of teachers grades 1–3 (combined) 3.7 3.9 3 4.4 4.9 Average pupil-teacher ratio grades 1–3 55.8 40.6 83.1 55 53.9 Average age of grade 1 students (years) 7.1 7.1 7.1 6.9 7.2 Average percent of grade 1–3 teachers with training In Khmer literacy 60.5% 76.1% 64.7% 18.7% 40.3% Average percent with pre-school 43.8% 46.3% 93.1% 42.86% 47.8% If yes, average upper pre-school enrollment 37.1 25.1 94.7 21.8 41.9 Average percent of schools receiving NGO support for early grade reading 75.0% 80.0% 77.6% 8.6% 29.9% Average percent with enough Khmer textbooks for grade 1 75.0% 70.0% 84.5% 57.1% 64.2% Average percent of schools with storybooks in grade 1 classroom 40.6% 61.3% 43.1% 11.4% 10.5% Average percent of schools with a library 57.8% 32.5% 86.2% 40.0% 50.8% If yes, average years the library has been at school: More than 7 years 67.7% 80.0% 57.1% 71.4% 66.7% 5 to 7 years 17.7% 16.0% 18.4% 10.7% 9.1% 3 to 5 years 14.7% 4.0% 12.2% 3.6% 9.1% Less than 3 years 0.0% 0.0% 12.2% 14.3% 15.2% if yes, average percent receiving support for their library 54.7% 32.5% 79.3% 34.3% 40.3% if yes, average percent with kids inside the library on day of visit 59.5% 65.4% 88.0% 17.9% 67.7% All Children Reading–Cambodia, Midterm Impact Report 33 Mean Values Kampong Thom A Kampong Thom B Siem Reap Kampot Battam￾bang Average percent of schools with electricity 78.1% 77.5% 86.2% 85.7% 91.0% Average percent of schools with any water source 96.9% 90.0% 94.8% 91.4% 91.0% Average percent of schools with piped water 12.9% 0.0% 12.7% 12.5% 21.3% Average percent of schools w/functioning toilets 93.8% 96.3% 98.3% 95.7% 97.0% Average percent of schools with two shifts 59.4% 27.5% 41.4% 57.1% 80.6% Average percent of schools with morning session only 21.9% 68.8% 27.6% 37.1% 9.0% Average percent of schools with full-day session 18.8% 2.5% 31.0% 5.7% 10.5% Average percent of schools with student-feeding program 37.5% 66.3% 65.5% 0.0% 1.5% Percent of schools where most children come to school: on foot 64.1% 76.3% 51.7% 30.0% 35.8% by bicycle 20.3% 16.3% 32.8% 34.3% 23.9% by motorized transport 15.6% 7.5% 15.5% 35.7% 40.3% 4. Conclusions The main purpose of this study was to report the impact of the pilot programs on students’ pre-literacy and early literacy performance in upper pre-school and grade 1, respectively. Analysis of the results of the survey across the comparison groups allowed us to isolate “business as usual” and confidently attribute the impact seen to the teaching and learning enhancements implemented by the pilots. The study found that the pre-school is not yet demonstrating a positive impact on students’ literacy skills. The project should explore ways in which the pre-school intervention can be strengthened, and what lessons can be learned from the grade 1 program design and implementation, which has demonstrated a significant positive impact on students’ basic literacy skills. Impact for grade 1 in Kampong Thom Interventions A and B and Siem Reap was strongest for letter knowledge. As the focus of teaching and learning in grade 1 is on letter knowledge, this was expected. At baseline, only 5 vowel items were correctly identified by at least 50% of students; at midterm this had increased to 18 correctly identified by at least 50% of the students. This helps give students the foundational letter knowledge they need to build on the more advanced word reading skills in grade 2. This survey also presents the proposition that grade 1 students in the reading pilots caught up with their grade 2 peers. This is shown in Exhibit 34, which shows the grade 1 baseline and midterm gain alongside the grade 2 baseline percent scores for the vowel items. All Children Reading–Cambodia, Midterm Impact Report 34 Exhibit 34: Percent Correct of Vowel Items—Kampong Thom and Siem Reap Combined, Baseline, and Midterm Grade 1 and Baseline Grade 2 Encouragingly, grade 1 students in pilot schools could on average correctly identify 15 out of the 23 vowel items better than their grade 2 counterparts. For next year’s evaluation, as the cohorts move through the pilot programs, it will be interesting to see how students who have received more than one year of the intervention perform compared with this year. Future evaluations should assess the impact on performance among students who have received both pre-school and grade 1 instruction and students who have received grade 1 and 2 instruction from the pilot programs. Analysis of individual school performance as shown in section 3.1.4 and Exhibit 17 indicates that more schools in the treatment group Kampong Thom A are experiencing above average gains on letter recognition, but that some schools experienced lower gains, if not minimal ones. It will be of particular interest to follow up this statistical analysis of program impacts by a more qualitative appreciation of what schools that are experiencing large gains are doing compared to those that did not. 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 ◌ា ◌ួ ◌ិ ◌ី េ◌ៅ ែ◌ ◌ុ េ◌ៀ ៃ◌ ◌ា◌ំ ◌ូ ◌ំ េ◌ឿ េ◌ោ◌ះ េ◌ោ េ◌ េ◌ើ េ◌◌ះ ◌ុ◌ះ ◌ុ◌ំ ◌ះ ◌ឺ ◌ឹ Percentage Correct Vowels Item Level 2018 Grade 1 2019 Grade 1 Gain 2019 Grade 2 All Children Reading–Cambodia, Midterm Impact Report 35 Annex 1: Instruments Used in the Assessments All Children Reading–Cambodia, Midterm Impact Report 36 Pre-literacy Assessment All Children Reading–Cambodia, Midterm Impact Report 37 All Children Reading–Cambodia, Midterm Impact Report 38 All Children Reading–Cambodia, Midterm Impact Report 39 All Children Reading–Cambodia, Midterm Impact Report 40 Adapted Early Grade Reading Assessment All Children Reading–Cambodia, Midterm Impact Report 41 All Children Reading–Cambodia, Midterm Impact Report 42 All Children Reading–Cambodia, Midterm Impact Report 43 All Children Reading–Cambodia, Midterm Impact Report 44 All Children Reading–Cambodia, Midterm Impact Report 45 All Children Reading–Cambodia, Midterm Impact Report 46 All Children Reading–Cambodia, Midterm Impact Report 47 All Children Reading–Cambodia, Midterm Impact Report 48 All Children Reading–Cambodia, Midterm Impact Report 49 All Children Reading–Cambodia, Midterm Impact Report 50 Adapted Early Grade Mathematics Assessment All Children Reading–Cambodia, Midterm Impact Report 51 All Children Reading–Cambodia, Midterm Impact Report 52 All Children Reading–Cambodia, Midterm Impact Report 53 Annex 2: Reliability and Validity The research team conducted an assessment of internal consistency to evaluate reliability. Internal consistency is an appropriate and standard classical evaluation approach for cross-sectional data, and when combined with item-level evaluative psychometric methods, provides insight on item and/or subtask functioning. Internal consistency is the average correlation of all possible half-scale divisions and is frequently provided in published assessment psychometrics.6 The range of the internal consistency statistic is from zero to one, where higher values are desired and a value of zero indicates inconsistency of measurement. As a general guideline, Cronbach’s alpha should be at least 0.70 for adequacy and coefficients closer to one indicate a good assessment. 7 The Cronbach’s alpha coefficient was computed using the STATA analytic software, which produced Exhibits A2-1 and A2-2, below, showing results of the reading and mathematics subtasks for grade 1. The first three columns of these tables provide general subtask information including the subtask name and the number of students accounted for within the subtask. The next three columns provide interrelationship information, including item-test correlations (the correlation between a subtask and the entire scale), item-rest correlations (the correlation between a subtask and the scale that is formed by all other subtasks), and the Cronbach’s alpha (discussed above). Overall, the subtask scores show good reliability statistics (Cronbach’s alpha of at least 0.80) with alpha scores of 0.84 and 0.89 for the grade 1 Early Grade Reading Assessment and Early Grade Mathematics Assessment, respectively. Exhibit A2-1: Grade 1 Early Grade Reading Assessment Grade 1 Reading Subtask Number of students Item-Test Correlation Item-Rest Correlation Alpha Letter Identification Score Percent 2692 0.90 0.87 0.82 Familiar Words Score Percent 2692 0.78 0.73 0.82 Extended Time Oral Reading Score Percent 2692 0.74 0.65 0.82 Reading Comprehension Score Percent 2692 0.70 0.61 0.83 Vowels Identification Score Percent 2692 0.89 0.76 0.81 Consonants Identification Score Percent 2692 0.90 0.80 0.79 Overall Test Cronbach’s Alpha 0.84 Exhibit A2-2: Grade 1 Early Grade Mathematics Assessment Grade 1 Mathematics Subtask Number of students Item-Test Correlation Item-Rest Correlation Alpha Number Identification Score Percent 995 0.94 0.87 0.82 Quantitative Comparison Score Percent 995 0.94 0.87 0.82 Addition Score Percent 995 0.85 0.76 0.87 Subtraction Score Percent 995 0.78 0.69 0.90 Overall Test Cronbach’s Alpha 0.89 6 Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 15(3), 297–334. 7 Aron, A., & Aron, E. (1999). Statistics for psychology (2nd ed.). Saddle River, NJ: Prentice Hall.