Latin American and the Caribbean (LAC) Reading Evaluation Contract Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Guatemala: Midline Report December 2016 This publication was produced for review by the United States Agency for International Development under task order AID-OAA-M-12-00020. It was prepared by Julieta Lugo-Gil, Nancy Murray, Camila Fernandez, Steven Glazerman, and Larissa Campuzano. Latin American and the Caribbean (LAC) Reading Evaluation Contract. Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Guatemala: Midline Report Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH i ABSTRACT This report presents the results from the midline data collection of the Leer Juntos, Aprender Juntos (Read Together, Learn Together) evaluation in Guatemala. In this evaluation, we randomly assigned schools to three groups: Group A schools implemented the in-school and community action program components; Group B schools implemented only the in-school component; and Group C schools did not implement any of the components of the program. The findings from the midline data collection confirm that the program is being implemented according to plans. Teachers in the treatment groups (Groups A and B) received the teacher training and coaching for which they were targeted. Also, children and families living in the communities in Group A participated in the community-based reading activities for which they were targeted. Moreover, children and families in Groups B and C (which were not assigned to receive the community action component) experienced low levels of exposure to services similar to the community-based reading activities offered to Leer Juntos, Aprender Juntos program participants in Group A. The findings from the midline data collection also provide evidence of program impacts on intermediate outcomes, such as teaching practices in the classroom and the literacy environment at home. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH iii ACKNOWLEDGMENTS The authors gratefully acknowledge numerous people and organizations that contributed to the implementation of all evaluation activities in Guatemala and to the production of this report. Luisa Fernanda Müller Duran and Maria José del Valle from the Evaluation Division of the Guatemala Ministry of Education, Juárez and Associates’ Regina da Costa, Justo Magzul, and Fernando E. Rubio, and Cristina Perdomo from the U.S. Agency for International Development (USAID)/Reforma Educativa en el Aula, all shared invaluable insights and measurement tools for assessing emerging literacy and early-grade reading in Spanish and K’iche’ in Guatemala. The State Department of Education in El Quiché (DIDEDUC El Quiché) provided a warm welcome and essential departmental support for the evaluation. Hember Herrera, departmental director of education, and Marcelino Cabrera, assistant director of intercultural bilingual education, offered continuous assistance during our data collection efforts in schools and communities in the Department of El Quiché. The staff from Save the Children provided us with key information on program implementation and valuable support in coordinating data collection activities. They include Katy Anis, chief of party for Leer Juntos, Aprender Juntos, and from Save the Children/Guatemala, Lilia Cifuentes, education project manager, and Ana Quixtán, monitoring and evaluation coordinator. Save the Children’s technical and administrative project team in El Quiché also provided valuable information and assistance in the coordination of the evaluation activities. USAID/Washington and USAID/Guatemala staff guided and supported the evaluation design and its implementation. They include, from USAID/Washington, Barbara Knox-Seith, senior education advisor, and Michael Lisman, education advisor, Bureau for Latin America and the Caribbean; from USAID/Guatemala, Laura Villegas Lalinde, project management specialist for education, Health and Education Office; and while at USAID/Guatemala, Juan Luis Córdova, education officer, Health and Education Office. We thank our local data collection partner, DMC Consultores, and its staff for high quality data collection in the evaluation schools and communities. Marco Oliva provided overall project management support in Guatemala, oversaw data entry, and ensured consistency of data programming. Otto Rego contributed thoughtful technical support, skilled training for our field staff, and generous collaboration in building a local team in the Department of El Quiché. Guillermo Duarte ensured careful planning of evaluation activities in Guatemala, provided close oversight of all data collection activities, and developed a collaborative relationship with staff at the State Department of Education in the Department of El Quiché. We are also indebted to Sylvia Linan-Thompson for her thorough and incisive input at different stages of the evaluation. Her expertise in models of reading interventions in bilingual classrooms and ample experience with measurement of early-grade reading skills were essential in designing the evaluation and interpreting baseline findings. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH iv At Mathematica Policy Research, Galina Lapadatova, Seth Morgan, Ivonne Padilla, Sarah Chan, Maura Butler, and Colleen Fitts provided critical support in data cleaning, programming, editing, and presentation. We are enormously appreciative of all the school directors and teachers who opened their classrooms and allowed us to observe the valuable work they do with students. Last, but not least, we thank the students and parents who provided us with the important information we are able to share in this report. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH v CONTENTS ABSTRACT .................................................................................................................................................... I ACKNOWLEDGMENTS............................................................................................................................... III EXECUTIVE SUMMARY ......................................................................................................................... ES-I I. INTRODUCTION .................................................................................................................................... 1 II. THE INTERVENTION, PREVAILING PRACTICE IN GUATEMALA, AND EARLY IMPLEMENTATION OVERVIEW ........................................................................................................... 3 A. Three approaches to improving early grade reading ................................................................ 3 1. Leer Juntos, Aprender Juntos............................................................................................. 3 2. Leer Juntos-school only ...................................................................................................... 7 3. Prevailing practice............................................................................................................... 7 B. Timeline of implementation of Leer Juntos, Aprender Juntos in Guatemala ............................ 8 III. EVALUATION QUESTIONS AND ANALYTICAL APPROACH ........................................................... 11 A. Research questions and data.................................................................................................. 11 B. Analytical approach ................................................................................................................. 12 IV. FINDINGS............................................................................................................................................. 15 A. Implementation milestones, challenges, and differences between groups............................. 15 1. Implementation milestones and challenges...................................................................... 15 2. Teacher-reported participation in professional development............................................ 16 3. Reading activities in the community.................................................................................. 20 B. Early impacts ........................................................................................................................... 23 1. Literacy materials in the classroom................................................................................... 23 2. Teachers’ approach to teaching reading .......................................................................... 24 3. Teacher-reported use of time............................................................................................ 26 4. Observed practice............................................................................................................. 27 5. The literacy environment and reading activities in the home............................................ 31 V. SUMMARY OF FINDINGS AND NEXT STEPS................................................................................... 39 REFERENCES............................................................................................................................................ 41 APPENDIX: BACKGROUND CHARACTERISTICS OF EVALUATION SCHOOLS, TEACHERS, STUDENTS, AND HOUSEHOLDS IN GUATEMALA.........................................................A.1 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH vi TABLES IV.1. Teacher-reported participation in selected reading instruction professional development topics .............................................................................................................................................. 19 IV.2. Parents’ reports on children’s exposure to and participation in reading activities in the community...................................................................................................................................... 21 IV.3. Observed teachers’ use of literacy instructional practices in the classroom ................................. 30 IV.4. Children’s school attendance, as reported in the household survey ............................................. 34 IV.5. Availability of books for children and scheduled time for reading at home.................................... 35 IV.6. Children’s activities at home that are not related to reading.......................................................... 38 A.1. Teacher background characteristics at midline ............................................................................ A.2 A.2. Respondents to the household survey..........................................................................................A.3 A.3. Households’ socioeconomic status...............................................................................................A.5 A.4. Children’s school attendance history, as reported in the household survey................................. A.7 FIGURES II.1. Timeline of Leer Juntos, Aprender Juntos activities 2012–2015 ..................................................... 9 III.1. Leer Juntos, Aprender Juntos evaluation design........................................................................... 11 IV.1. Teacher’ participation in reading instruction professional development activities ......................... 17 IV.2. Percentage of households reporting that reading activities or games for children are offered in their community.............................................................................................................. 20 IV.3. Participation in reading workshops and volunteering activities (percentage of households) ........ 22 IV.4. Available materials with text in the classroom (percentage of classrooms) .................................. 23 IV.5. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) ........................................................................................................... 24 IV.6. Teacher-reported approach to teaching reading (percentage of teachers)................................... 26 IV.7. Teacher-reported number of hours in most recent work week spent in activities related to their job........................................................................................................................................... 27 IV.8. Observed teachers’ time using literacy instruction activities (average number of minutes) .......... 28 IV.9. Reading with the evaluation children (percentage of children)...................................................... 31 IV.10. Other family/household members who read with the child (percentage of children)..................... 32 IV.11. Reading-related activities at home (percentage of children) ......................................................... 36 IV.12. Number of days in previous week in which child did reading-related activities at home (average number of days).............................................................................................................. 37 A.1. Household assets, utilities, and commodities (percentage of households)..................................A.6 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH ES-i EXECUTIVE SUMMARY A. Introduction This report presents the results from the midline data collection of the Leer Juntos, Aprender Juntos (Read Together, Learn Together) evaluation in Guatemala. The findings confirm that the program is being implemented according to plans and provide evidence of impacts on intermediate outcomes, such as the characteristics of the reading environments and activities outside of school hours. The Leer Juntos, Aprender Juntos evaluation includes 295 schools in two countries, Guatemala and Peru. The sample sizes were nearly evenly divided between the two countries, with Guatemala accounting for 51 percent of the schools and 58 percent of the students. The research team randomly assigned the schools to three groups—which we call Groups A, B, and C—each taking a different approach to literacy instruction. Schools in Group A receive an in￾school intervention that includes teacher training and coaching, and a community-based intervention that relies on community volunteers to engage with young students and promote a culture of reading outside of regular instructional periods at school. The schools in Group B receive only the in-school teacher training and coaching component. Schools in Group C do not receive either of the in-school or community-based intervention components; instead, they receive the prevailing practice approaches available to all schools in each country. For both Guatemala and Peru, we collected data in the first year (baseline) and at the end of the second year after baseline (midline); we will collect data again at the end of the third year (endline). Our analysis of the data collected in the second year of the Leer Juntos, Aprender Juntos program implementation (midline) yielded two main findings: 1. The program is being implemented according to its original vision. 2. There is evidence of early impacts of the Leer Juntos, Aprender Juntos intervention on reading environments and activities outside of school hours. B. The implementation is following its original vision Our findings confirm that teachers in the treatment groups (A and B) received the teacher training and coaching for which they were targeted, whereas the children and their families living in the communities in Group A are participating in community-based reading activities. For instance, most of the teachers in the two treatment groups (96 percent in Group A and 94 percent in Group B) reported participating in reading instruction professional activities during the second year of program implementation, whereas only 22 percent of teachers in Group C did. In addition, most teachers in the two treatment groups (at least 86 percent of teachers in Groups A and B) reported receiving training focused on teaching letter knowledge, fluency, reading comprehension, vocabulary, and phonological awareness, which are the main focus of the teacher training component of the Leer Juntos, Aprender Juntos intervention. Children and families living in the communities in Group A are participating in the community-based activities of the intervention, and we found no evidence that children and families in Groups B and C could be participating in similar community-based activities. Of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH ES-ii households in Group A, 80 percent reported that reading activities or games for children are available in their community, and only 3 and 1 percent of households in Groups B and C, respectively, reported that those activities are available in their communities. Children in Group A have actually participated in the reading activities offered in the community as part of the Leer Juntos, Aprender Juntos program, such as borrowing books from book banks, reading camps, story time activities, and tutoring and reading games with older children (reading buddies). The parents of the children in Group A also participated in the community reading activities offered by the intervention: 34 percent of the parents in this group reported participating in reading workshops focused on helping parents support their children in learning to read, and in 21 percent of the households in Group A at least one adult reported serving as a volunteer in leading the reading activities offered in the community as part of the intervention. C. The intervention is having impacts on intermediate outcomes The literacy environment in the classrooms of students in the two treatment groups (Groups A and B) is improving. For instance, in the second year of program implementation, more classrooms in the treatment groups than in the prevailing practice group (Group C) have materials to facilitate early reading instruction, such as printed materials (posters, banners, and so on), an alphabet, and familiar words in Spanish and K’iche’ that are displayed prominently in the classroom. We found a few differences between the groups in terms of teacher-reported practices. For example, more teachers in Groups A and B (76 and 82 percent, respectively) than in Group C (46 percent) reported using the syllabic method. This finding means that teachers in the treatment groups, compared with their counterparts in the prevailing practice group, might be incorporating more instruction on letter knowledge and phonological awareness—which are the focus of the syllabic method—into their reading teaching practice. We found no statistically significant differences between groups in terms of the teaching practices observed during reading instruction periods in the 2nd-grade classrooms, such as the language used to provide instruction, time spent in specific literacy instruction activities, and use of literacy instructional practices. In addition, our findings suggest that the community action component of the intervention is starting to have an effect in the reading activities children conduct outside of school hours. More children in Group A than in B read with siblings at home, and more parents in Group A than in Groups B and C spent time looking at books with their children and reported that children’s books are available in the household (however, the difference between Groups A and B was not statistically significant). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 1 I. INTRODUCTION In the past 15 years, countries in Latin America and the Caribbean (LAC) have made considerable progress on increasing primary school enrollment and completion among the very poor (United Nations Educational, Scientific and Cultural Organization [UNESCO] 2015). However, challenges persist in improving literacy, particularly for groups with significant language diversity (Benavides et al. 2010). For example, in Guatemala, as in other LAC countries with large, linguistically diverse populations, children enter school with a wide range of ability to understand and speak the country’s official language (Spanish); some children have no knowledge of the official language, whereas others speak and understand it fluently. Intercultural bilingual education is increasingly used to help poor, marginalized children whose mother tongue differs from their country’s dominant language (Trapnell 2003; U.S. Agency for International Development [USAID] 2011). Intercultural bilingual approaches to education include formal school-based programs with instruction in the mother tongue as well as informal community-wide interventions (King and Arnal 2016). Previous research has shown that children should be taught in their mother tongue (UNESCO 2016), and models aimed at increasing instruction time in the classroom and reading practice time at home have shown promise (Banerjee et al. 2010, 2007; Bruns and Luque 2014). These models include community engagement, training volunteers to teach low-performing students, and providing remediation by locally trained teachers. However, comprehensive reading programs that incorporate both language transition instructional approaches in the classroom and mother tongue supports for building reading abilities outside the classroom have not been rigorously evaluated in the LAC region, and their cost-effectiveness is unknown. To address this need for rigorous evidence, USAID contracted with Mathematica Policy Research to design and conduct an evaluation of an approach to early grade reading instruction in LAC communities with linguistically diverse populations: the Leer Juntos, Aprender Juntos program, an adaptation of Literacy Boost model of Save the Children, which includes a teacher training component and a community involvement component. The Leer Juntos, Aprender Juntos evaluation includes 295 schools in two countries, Guatemala and Peru. The research team randomly assigned the schools to three groups, which we call Groups A, B, and C. Schools in Group A received two intervention components: (1) teacher training and coaching and (2) community-based activities aimed to promote reading. The schools in Group B received only the teacher training and coaching intervention component, and the schools in Group C did not receive either of the two intervention components. The sample sizes were nearly evenly divided between the two countries, with Guatemala accounting for 51 percent of the schools and 58 percent of the students. Because Save the Children planned to roll out the program in a phased manner, we also carried out the evaluation in phases, with Phase I in each country starting in 2013 and Phase II starting in 2014. The rollout in the second phase lagged the first phase by one year, but unless stated otherwise, we report results of the samples from both phases combined, recording key study milestones in terms of years since baseline. For both phases, we collected data in the first year (baseline, in 2013 for Phase I and 2014 for Phase II), at the end of the second year after baseline (midline, in 2014 for Phase I and 2015 for Phase II), and at the end of the third year (endline, in 2015 for Phase I and 2016 for Phase II). The samples were spread out approximately evenly between the two phases. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 2 This report presents findings from the midline data collection in Guatemala, which included classroom observations and teacher and household surveys. Our findings reveal two major outcomes. First, we found that the two components of the Leer Juntos, Aprender Juntos intervention were implemented as intended. Specifically, we found that in the second year of the evaluation, at least 92 percent of the teachers in the treatment groups (A and B) reported participating in reading instruction professional activities through the Leer Juntos, Aprender Juntos intervention, and 100 percent of the teachers in the two treatment groups received at least one coaching visit in their classrooms every three months. In addition, reading activities were conducted in the communities in Group A, and the children and their families in this group did participate in these community activities. Second, we found evidence of early impacts of the Leer Juntos, Aprender Juntos intervention in Guatemala: the teaching practices and the literacy environment in the school and at home in the two treatment groups (Groups A and B) improved by the second year of the evaluation. The final (endline) evaluation report will include findings on the impacts of Leer Juntos, Aprender Juntos on the reading comprehension outcomes of students. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 3 II. THE INTERVENTION, PREVAILING PRACTICE IN GUATEMALA, AND EARLY IMPLEMENTATION OVERVIEW This chapter describes the key elements of the Leer Juntos, Aprender Juntos program, presents a summary of the prevailing early-grade reading instruction approach in Guatemala, and provides an overview of the first two years of implementation of the Leer Juntos, Aprender Juntos program in Guatemala (2013-2015). A. Three approaches to improving early grade reading In the evaluation of the Leer Juntos, Aprender Juntos program, we examined three distinct approaches to improving early-grade reading in linguistically diverse communities: (1) an approach combining teacher training and coaching with a community component aimed at creating a culture of reading both in school and outside of school, (2) an approach that includes only teacher training and coaching, and (3) an approach that does not include either the teacher or community components included in the first two approaches, but rather represents the prevailing early-grade reading instruction approach in Guatemala. We provide more details about these approaches below. 1. Leer Juntos, Aprender Juntos Leer Juntos, Aprender Juntos (Group A, or Leer Juntos for ease of presentation) means “Read Together, Learn Together.” Save the Children, an international non-governmental organization focused on promoting children's rights and providing relief and support to children in developing countries, developed this program based on its experiences with its Literacy Boost program in El Salvador, Haiti, Ethiopia, Pakistan, and more recently, Rwanda (Friedlander and Goldenberg 2016), among other developing countries. Like Literacy Boost, Leer Juntos, Aprender Juntos combines an in-school component that includes teacher training and coaching with a community component that relies on community volunteers to engage with young students and promote a culture of reading outside of regular instructional periods at school. The in-school and community components of the Leer Juntos, Aprender Juntos program align with the elements of reading instruction that existing research (Comings 2012) has identified as effective for reading instruction when implemented simultaneously: (1) text and materials, (2) teacher training and support, (3) community and parental support, and (4) assessment and tracking. Therefore, in addition to targeting important changes in the classroom, the Leer Juntos, Aprender Juntos program engages community members and parents in the learning process. The community outreach component is a potentially low-cost, effective way to increase children’s time on task. The Leer Juntos, Aprender Juntos program adapted the teacher training and community action components of Literacy Boost to K’iche’- and Quechua-speaking populations in Guatemala and Peru, respectively. The program’s primary implementers—Save the Children in Guatemala and Kallpa, its local partner in Peru—trained and empowered teachers to deliberately teach and monitor students’ mastery of core reading skills. Save the Children and Kallpa have also worked to strengthen parent and community involvement in building children’s reading abilities in their mother tongue and opportunities to practice, including creating and providing EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 4 context-relevant reading materials in their mother tongue and community activities that promote reading engagement and a culture of literacy. In-school component: teacher training and coaching. The in-school component of Leer Juntos, Aprender Juntos, which is based on the Literacy Boost model, follows the recommendations of the National Reading Panel for best approaches in reading instruction (National Institute of Child Health and Human Development 2000): this component aims to train and coach teachers so they are better equipped to provide mother tongue and/or Spanish reading instruction focusing on five foundational reading skills (alphabet knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension), to monitor students’ mastery of these skills, and to increase students’ access to and use of mother tongue reading materials. The program incorporates instruction in the mother tongue but adapts to the linguistic background of teachers and students. The school-based component entails six main activities, intended to increase allocation of class time on reading instruction and improve the quality of reading instruction: 1. Training of trainers in reading instruction techniques 2. Training of teachers in the five core skills of reading 3. Creation of materials for enhanced, print-rich classrooms 4. Mentoring and coaching of teachers in classrooms 5. Ongoing formative assessments to track children’s progress in reading 6. Guidance in the adaptation of existing curricula to incorporate five core skills of reading and related reading instruction techniques into daily school activities The teacher training activity (for more details, see the teacher training toolkit [Save the Children 2012a]) consists of nine modules focused on the following topics: 1. Introduction to reading development and instruction for young children 2. Formative assessment 3. Addressing language issues in the literacy classroom 4. Letter knowledge/alphabetic principle, 5. Phonological awareness 6. Reading fluency 7. Vocabulary 8. Reading comprehension 9. Reflecting on and applying best practices for teaching reading comprehension and formative assessment EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 5 Teachers are presented with most of the training content at the beginning of the school year,1 and they complete the rest of their training throughout the school year. For example, in the first year of implementation of Leer Juntos, Aprender Juntos in Guatemala, teachers were trained on the first five modules (introduction to reading development, formative assessment, language issues, letter knowledge, and phonological awareness) over two days (four to six hours each day) in June 2013. The rest of the training was delivered in one-day sessions held in July (training on fluency and vocabulary modules) and August (training on reading comprehension and reflecting and practicing modules). In the second year of implementation of the program, teachers were trained on the first five modules during two days in January 2014, with the rest of the training provided in one-day sessions held in March (covering vocabulary and fluency modules), May (covering reading comprehension module), and September (covering the last module on reflecting and practicing). Following the first training session, teachers received a visit in the classroom from the program’s technical staff (coaches) at least once every three months, and in most cases two times in those three months (Save the Children and Kallpa 2013, 2014, 2015). The purpose of the visits was to directly support teachers by observing their work, interacting in the classrooms and/or demonstrating teaching techniques, and making recommendations and suggestions to teachers for improving the application of the teaching strategies taught in the program’s training. In addition, coaches supported open discussion sessions (called Teacher Circles), which are held during recess or in the afternoons or evenings. These sessions focused on (1) strengthening and practicing the topics learned in the training, (2) conducting activities to promote the development of the five core reading skills, (3) assisting teachers with designing lesson plans that incorporate the five core reading skills, and (4) identifying language problems that arise in different contexts and proposing strategies to promote the development of literacy skills in the mother tongue. Community action component. The community action component of the program, also based on the Literacy Boost model, aims to strengthen parental and community involvement in building children’s reading abilities and increasing their opportunities to practice reading outside of school. For example, the component provides guidelines on how to adapt and create mother tongue reading materials and use them to support literacy and language activities in children’s communities and homes. (For more details, see the Literacy Boost community action toolkit [Save the Children 2012b].) This component is delivered through playful group activities, such as reading camps, reading buddy sessions (peer assistance), reading festivals, and reading contests led by community volunteers. The following main activities comprise the community action component (Nieto 2015; Save the Children and Kallpa 2013, 2014, 2015): • Creating printed materials for book banks—collections of books and reading materials—in the children’s mother tongue and/or in Spanish to build portable libraries. In Guatemala, the program’s team also selected and purchased books from book fairs and other appropriate sources to contribute to the community book banks. 1 The school year in Guatemala starts in January and ends in October. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 6 • Promoting the use of book banks among community members. Each community has at least one book bank. • Supporting the implementation of reading activities in the community. These activities were implemented during the school year (January through October) so that invitations for children and their families to participate in the activities could be distributed in the classrooms and community volunteers could have time off during the school vacation period. The intervention’s community activities are as follows: - Story hour. This activity is usually conducted in the afternoons and in parallel with the distribution of books through the community book banks. Story hour activities are conducted at least once every three months in each community. In Guatemala, the story hour has been a popular activity, sometimes drawing more than 100 children. During story hour, community volunteers tell stories about the community or about how corn was first planted and other traditional stories. - Reading camps. This activity is similar to the story hour activity but is implemented in a more structured way. First, the program team develops the plans for the reading camps and shares them with the volunteers so they can implement them in their communities. The purpose of the reading camps is to stimulate the development of a specific reading skill, in the mother tongue or in Spanish, and incorporate storytelling and reading games to engage children. In Guatemala, reading camps are implemented once a week in each community in Group A. - Reading buddies (peer assistance). In this activity, older children (in grades 4 to 6) read together with the younger children (grades 1 to 3) to help them improve their reading skills. This activity has been implemented one or two times per week in each Group A community in Guatemala, mostly within the schools to maximize the number of children participating in the activity. - Reading festivals or fairs. Children participate in games and activities in which they can practice, develop, and strengthen their reading skills. In Guatemala, volunteers have organized reading fairs in coordination with community leaders and some school principals and teachers, and implemented the fairs in each community at least once during 2013 and 2014. - School–community accountability meetings. Local educations leaders and councils meet with community authorities (such as members of the school board and the community development councils) to discuss the funding sources of reading activities, the progress achieved with the reading activities implemented in schools and the communities, the resources used and their purpose, and the future sustainability of the activities. In Guatemala, school–community accountability meetings have been held at least once in each community in the first two years of the project. • Training volunteers to lead reading activities in the community. The program offers at least one community volunteer training session every three months in each community. The purpose of this training is to enable volunteers to implement the community action component activities (such as promoting the use of book banks, reading camps, reading buddies, and story hour) with children in their communities. In the training, volunteers EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 7 experience each of the community reading activities and develop a weekly schedule of activities for their community to guide the rollout of implementation. • Conducting reading awareness workshops with parents and community members. The purpose of these workshops is to promote awareness and understanding among parents and community members about how the community reading activities help children learn to read. The workshops also aim to raise awareness about how important it is for children to start learning to read in their mother tongue. In the second year of implementation in Guatemala, reading awareness workshops with parents were held about once every three months in each community. 2. Leer Juntos-school only The second approach considered in the evaluation is a variant of Leer Juntos, Aprender Juntos that includes the in-school component but not the community component. We refer to this as Leer Juntos-school only (Group B). Including this approach in the evaluation enables us to isolate the effects of the in-school component from the effects of the community action component. 3. Prevailing practice The third approach is the prevailing early-grade reading approach currently being implemented by the Ministry of Education in the regions where the evaluation is taking place (Group C). Here, we discuss the strategies currently being pursued in Guatemala. Information on the strategies being pursued in Peru are available in a separate midline report. Since the 1980s, the government of Guatemala, through the Ministry of Education’s DIGEBI, has designed and implemented initiatives that aim to deliver bilingual education services in the mother tongue and in Spanish. The DIGEBI prescribes that children whose mother tongue is not Spanish enter bilingual education programs as early as possible, preferably in preschool or kindergarten (if preschool and/or kindergarten are available) or in 1st grade (Rubio 2004). According to DIGEBI, children should first acquire literacy skills in their mother tongue and then gradually transition to learning those skills in Spanish. Therefore, in Guatemala there is not a specific grade level at which instruction should transition fully from the mother tongue to Spanish; rather, the transition should be gradual, and at each grade level the amount of instruction in Spanish should be increased. One of the most recent and largest bilingual education initiatives in Guatemala is the Leamos Juntos (We Read Together) program, which was established in 2012 and is implemented by the Guatemalan Ministry of Education. The primary objective of the program is to promote reading comprehension and values among bilingual and monolingual students at all grade levels. To meet this objective, the Ministry of Education is training teachers, developing and distributing reading materials in Spanish and in all the languages of the numerous ethnic groups in the country, establishing community alliances, and encouraging parents to visit communal libraries with their children and practice reading at home. In addition to national-level programs to promote literacy among Mayan language speakers, schools in the control group could also be subject to local decisions about education priorities and practices. Both national and local efforts can likely influence the curriculum in use in a EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 8 particular school. Therefore, the control group in this evaluation must be understood to include exposure to practices and programs aimed specifically at improving early grade literacy. It is the effects of the Leer Juntos, Aprender Juntos program above and beyond these coexisting efforts that we aim to discover in the present evaluation. B. Timeline of implementation of Leer Juntos, Aprender Juntos in Guatemala In Guatemala (as in Peru), the schools and communities in Group A (Leer Juntos group) received the in-school (teacher training and coaching) and community-based activities components of the intervention. Schools in Group B (Leer Juntos-school only group) received the teacher training and coaching component, but the community action component of the intervention was not implemented in this group. Neither component of the Leer Juntos, Aprender Juntos intervention was implemented in schools or communities in the prevailing practice group (Group C). Planning for the start of implementation of the Leer Juntos, Aprender Juntos program activities in Groups A and B in Guatemala started in fall 2012 with the development of a work plan and project strategy, hiring of program implementation staff, a training workshop for key implementation staff from Save the Children, and meetings of key stakeholders, for example, USAID, local education authorities, and relevant local government entities (Save the Children and Kallpa 2012, 2013). Preparations for the program rollout in Guatemala continued in winter and early spring of 2013; with activities that included recruiting and orienting technical and project administration staff, developing an implementation monitoring and evaluation system, mapping elements of the teacher training with the national curriculum, translating project materials to K’iche’, producing reading materials for local book banks, and recruiting schools to participate in the evaluation. Figure II.1 presents the timeline of Leer Juntos, Aprender Juntos activities from 2013 to 2016 in Guatemala. Random assignment for the Phase I cohort was conducted in May 2013. The Leer Juntos, Aprender Juntos intervention was rolled out in Guatemala in June 2013 with the first teacher training sessions for Phase I teachers in Groups A and B. The teacher training continued in 2013 with sessions conducted in July, August, and October. Data collection for the baseline of the Phase I cohort occurred in August 2013, approximately two months after teacher training began.2 Recruitment and training of the community volunteers started in August 2013 in Group A communities, as did the community-based intervention activities such as reading banks, story hour, reading buddies, and reading camps (Figure II.1). 2 While it would have been ideal to have the baseline data collected prior to the rollout of teacher training in Phase I schools, it was not possible to procure a local data collection firm in the time between reaching an agreement with stakeholders on the evaluation design (May 2013) and the roll out of the teacher training (June 2013). We submitted and agreed upon the evaluation design with stakeholders in May 2013, and only after that we were able to share the evaluation design in the terms of reference in the request for proposals from prospective data collectors. We also had to allow time for preparation and review of proposals and for the newly-procured data collection partner, DMC Consultores, to recruit and train interviewers. For Phase II schools we were able to accomplish baseline data collection closer to the time of teacher training roll out. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 9 Figure II.1. Timeline of Leer Juntos, Aprender Juntos activities 2012–2015 Note: The school year in Guatemala starts in January and ends in October. The months of school recess are indicated in grey. RA = random assignment of schools; BL = baseline data collection; TT = rollout of teacher training and support component; CA = rollout of community action component; ML = midline data collection; EA = end of intervention implementation activities; EL = Endline data collection. For the Phase II cohort, random assignment ended in December 2013. The first teacher training sessions for Phase II teachers in the treatment groups occurred in January 2014, and teacher training continued with sessions held in March, May, July, September, and November. Data collection for the baseline of the Phase II cohort was conducted from March through May 2014.The first training of community volunteers for Phase II took place in May 2014, and in July the community action components began for Phase II communities in Group A (Figure II.1). Data collection for the midline of the Phase I cohort was conducted in July and September through November 2014. The midline data collection for the Phase II cohort was conducted in June and July 2015 (Figure II.1). The teacher training and support (coaching) activities, as well as training of teacher leaders and principals, continued throughout 2014 and 2015 for the Phase I and II cohorts in the treatment groups (A and B). Recruitment and training of community volunteers and community reading activities also continued in 2014 and 2015 for the Phase I and II communities in Group A. The implementation of the teacher training and support and community components of the intervention (Groups A and B) ended in December 2015 for the Phase I and II cohorts (Figure II.1). The final data collection (endline) took place in August and September 2015 for the Phase I cohort and in August and September 2016 for the Phase II cohort. The final report will include the impact findings based on the endline data collection for the two cohorts. M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S Phase I RA TT BL CA ML ML ML ML EL EL EA Phase II RA TT BL BL BL CA ML ML EA EL EL 2013 2014 2015 2016 This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 11 III. EVALUATION QUESTIONS AND ANALYTICAL APPROACH The purpose of the Leer Juntos, Aprender Juntos evaluation is to determine the relative impact and cost-effectiveness of the three approaches: Leer Juntos (A), Leer Juntos-school only (B), and prevailing practice (C). The primary outcomes of interest for the final impact analysis will be early grade reading skills, such as fluency and comprehension, but we will also examine other outcomes, such as improved classroom practices and culture of reading at home and in schools. To provide rigorous evidence on the relative impacts of the three alternative approaches, we randomly assigned schools to the three intervention groups (Figure III.1). The research team selected and assigned 75 schools to intervention groups in 2013 (Phase I) and another 75 in 2014 (Phase II). 3 Figure III.1. Leer Juntos, Aprender Juntos evaluation design Source: Leer Juntos, Aprender Juntos evaluation plan (Glazerman et al. 2013). A. Research questions and data The evaluation aims to answer two primary research questions: 1. What is the impact of the community action component of Leer Juntos, Aprender Juntos on early grade reading and other outcomes relative to an intervention that does not have the community action component? To answer this question, we will compare outcomes for schools in Group A with those in Group B. 3 A similar number of schools were assigned in the same manner in Peru: 74 schools in Phase I and 71 schools in Phase II. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 12 2. What is the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early grade reading and other outcomes relative to prevailing practice? To answer this question, we will compare outcomes for schools in Group B with those in Group C. After randomly assigning the 150 schools to the three groups, the evaluation team randomly selected one 1st grade classroom, one teacher, and roughly 10 1st grade students from each school to include in the analysis sample. We used this approach to minimize data collection cost and burden while still obtaining data that were representative of the schools in the evaluation. The Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016) presents the findings from the examination of the baseline data, collected from a sample of students in 1st grade at baseline. This midline report focuses on the data from the second year of program implementation for both Phase I and II children, when most of the children in the evaluation have progressed to 2nd grade. Therefore, it describes the evaluation sample after about one year of potential exposure to the intervention. We conducted random assignment in May 2013 for the Phase I cohort and in December 2013 for the Phase II cohort. Thus, the midline data collection took place about 12 to 18 months after randomization of schools. For the midline data collection, we observed schools and classrooms, interviewed teachers, and interviewed parents or main caregivers of the children in the evaluation. The school observation used a checklist that could be completed based on an environmental scan of the school premises. During a one-hour class period, the evaluation team observed and recorded information on teachers’ instructional practices, teachers’ and students’ language use in the classroom and time on task, and other school and classroom characteristics. The evaluation team also administered an in-person survey of teachers about their instructional practices; education and experience; participation in professional development activities; occupational needs; career expectations; and background characteristics, such as K’iche’ proficiency, ethnicity, and income. Finally, the evaluation team administered an in-person survey of the main caregivers (usually the mother or father) of the children in the evaluation to learn about household composition, family socioeconomic status, household assets, children’s schooling background and routines at home, and the participation of children and families in reading activities offered in their communities. B. Analytical approach We examined differences between groups in measures at the school level (characteristics of schools, teachers, and classrooms) and at the student level (characteristics of students’ households). To assess the differences between groups in those measures, we conducted regression analyses that controlled for the stratification of the random assignment of schools4 and for the data cohort (Phases I and II). From those regressions, we computed regression-adjusted means of the relevant measures for each group (A, B, and C) and tested whether the differences in the adjusted means between Groups A and B, and between Groups B and C, were statistically different from zero. We used 4 The stratification was a procedure for randomization wherein we first grouped schools with similar characteristics (within phase) into strata and randomly assigned to the three groups within these strata. This increases the likelihood that certain observable characteristics are evenly balanced across the three groups. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 13 the p-values from the tests of differences in group means to assess statistical significance of the differences in means. We defined the difference in means between groups as statistically significant whenever the p-value of the test was lower than 0.05. In determining the statistical significance of mean differences between groups in measures at the student level (characteristics of the household), we took into account the clustering of students within schools, school size, and student-level nonresponse. For more details on how we conducted the stratified randomization of schools; the sample selection; and the methods we used to estimate the differences between groups in measurable characteristics of schools, teachers, classrooms, and students, refer to the Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016). This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 15 IV. FINDINGS This chapter presents our findings on the differences in implementation of approaches to improve early grade reading among the groups, and the early impacts of Leer Juntos, Aprender Juntos. These results are based on data collected in the second year of program implementation for both Phase I and II samples (midline) through classroom observations and teacher and household surveys. The midline data also allowed us to confirm the baseline finding that the experimental groups were well balanced, by including household information on the children in our sample. We examined differences between groups in the background characteristics of schools, teachers, and children in the second year of program implementation (at midline), and present those findings in Appendix A. We examined background characteristics that are constant over time (or very unlikely to be affected by treatment) because some of those characteristics could affect the impacts of the intervention and were not measured at baseline. 5 As with our examination of the groups at baseline, we found no noteworthy differences between groups in terms of school infrastructure and background characteristics of teachers at midline (Appendix A, Table A.1), nor did we find differences between groups in terms of children’s participation in preschool or households’ socioeconomic characteristics (Appendix A, Figure A.1 and Tables A.2 through A.4). Our endline analysis will control for child-level and school level variables that might influence student outcomes. A. Implementation milestones, challenges, and differences between groups In Chapter II, we describe how the Leer Juntos, Aprender Juntos program was planned. In this chapter, we present our examination of the program implementation milestones and challenges in Guatemala and the differences in available teacher training and community-based activities between the treatment groups and the nonintervention group in the second year of program implementation. The main goals of this analysis are to (1) assess whether the Leer Juntos, Aprender Juntos program was implemented according to plan and (2) understand what is happening in Group C (the counterfactual), including ways in which the prevailing practice may have changed since baseline. 1. Implementation milestones and challenges The Leer Juntos, Aprender Juntos team achieved several milestones in the first two years of program implementation in Guatemala with the Phase I and II cohorts (2013–2015). These implementation milestones include the following (Save the Children and Kallpa 2013, 2014, 2015): • Completion of curriculum mapping, linking the five core reading skills with the National Reading Curriculum (Curriculum Nacional Base) 5 We were only able to do one household survey and for logistical reasons chose to conduct the household survey at midline so as to assess early effects of the community intervention as well as household socioeconomic and demographic characteristics that we may want to control for in the final impact analysis. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 16 • Delivery of all training modules to Phase I and II teachers in 2013 and 2014, respectively, with an average attendance rate of more than 90 percent across training sessions; by the end of 2015, training had been provided to 400 teachers, teacher leaders, and school principals • At least one technical support visit every three months completed with each teacher from the Phase I and II cohorts; this means that more than 3,000 technical support visits to teachers and schools were completed from 2013 to 2015 • Establishment of one book bank in each of the 50 communities in Group A, with about 150 different books and printed materials available to children in each community • Delivery of training to 220 community volunteers in 2013 and 2014 • Implementation of all community action component activities in each of the 50 communities in Group A, with reading camps and reading buddies activities implemented once or twice a week, story hour activities implemented once every three months, and with one or two reading fairs per community The Leer Juntos, Aprender Juntos team also experienced challenges while implementing the intervention. During the early stages (Phase I, first year), the main challenges were: • The need to provide comprehensive teacher training and coaching for the in-school component in a compressed period (June to October); • Reluctance from teachers to use K’iche’ as the language of literacy instruction; and • Difficulties in identifying, engaging, and retaining volunteers to lead the community-based reading activities. As implementation of the intervention progressed, the evaluation team encountered further challenges (Save the Children and Kallpa 2013, 2014, 2015): • Engaging teachers and ensuring that they completed all the modules of the intervention training before the school year ended; this challenge was particularly evident in the first year of implementation of the program (2013) with Phase I teachers, as all the training materials had to be covered in about five months • Working with teachers to help them incorporate the Literacy Boost methodology (on teaching the five core literacy skills) into their lesson planning and teaching strategies • Disruption of community-based activities during national and school holidays • High turnover among the volunteers leading the community-based reading activities • Parents’ absenteeism and/or lack of time to attend reading awareness workshops 2. Teacher-reported participation in professional development Results from the teacher survey suggest that teachers in the treatment groups (A and B) are receiving the training for which they were targeted by Leer Juntos, Aprender Juntos. Because teachers in Groups A and B received the teacher training and coaching component of the intervention, we do not expect to see statistically significant differences between those groups in their participation in reading instruction professional development (PD) activities. However, EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 17 because the teachers in the prevailing practice group (Group C) did not receive the intervention’s training and coaching, we expect to see statistically significant differences between Groups B and C in their participation in reading PD. Figure IV.1 shows teachers’ reports on their participation on reading instruction professional activities at midline. Figure IV.1 is the first of a series of bar charts/figures presented in this chapter. The colored bars in each figure represent groups (Leer Juntos [Group A] is dark blue, Leer Juntos-school only [Group B] is medium blue, and prevailing practice [Group C] is light blue). The group average or mean is placed within each bar. When the difference between groups is statistically significant, the magnitude of the difference is presented alongside the Group B bar. Statistically significant differences between Groups A and B are on the top left side, and statistically significant differences between Groups B and C are on the top right. A large percentage of the teachers in the two treatment groups (96 percent in Group A and 92 percent in Group B) reported participating in reading instruction professional activities during the second year of program implementation (2013 for teachers in Phase I schools and 2014 for teachers in Phase II school), as illustrated in Figure IV.1. This is in contrast to the teachers in 22 percent of schools implementing prevailing practices (Group C). Figure IV.1. Teacher’ participation in reading instruction professional development activities Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. As expected, most Group A and Group B teachers reported getting their training through Save the Children, whereas most Group C teachers received their training from the Ministry of Education. There is minimal evidence of crossover. Four percent of the teachers in the prevailing practice group reported they participated in reading instruction professional activities provided by Save the Children through Leer Juntos, Aprender Juntos. 96 10 94 92 6 92 22 19 4 0 10 20 30 40 50 60 70 80 90 100 Any reading instruction professional development activities Activities provided by the Ministry of Education or other institution Activities provided by Save the Children through Leer Juntos, Aprender Juntos Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 70* 88* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 18 Most teachers in the two Leer Juntos, Aprender Juntos treatment groups also reported participating in professional development activities focused on teaching specific literacy skills. As shown in Table IV.1, more than 90 percent of the teachers in Groups A and B reported receiving training on teaching letter knowledge, reading fluency, reading comprehension, and vocabulary. Also, at least 86 percent of the teachers in the treatment groups reported participating in professional development activities focused on teaching phonological awareness. In contrast, less than 20 percent of the teachers in the prevailing practice group reported participating in professional development activities focused on teaching specific literacy skills. Teachers reported participating in other reading instruction professional development activities: at least 82 percent of teachers in each of the treatment groups (A and B) reported participating in professional development activities focused on teaching reading in K’iche’ or teaching bilingual students, making/adapting materials to teach reading, and creating a suitable classroom environment for learning reading. Around 70 percent of teachers in the intervention groups reported receiving training focused on teaching instructions for reading books. Teachers in the treatment groups also participated in training focused on providing support to students and their parents for conducting literacy activities at home and the community (Table IV.1). Finally, teachers in the treatment groups reported that they use what they learned from the reading instruction professional development activities frequently. At least 50 percent of the teachers in each of the treatment groups (Groups A and B) reported using what they learned from the professional development activities every day, and about 80 percent of teachers in these two groups reported using what they learned at least twice a week (Table IV.1). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 19 Table IV.1. Teacher-reported participation in selected reading instruction professional development topics Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of teachers who reported participating in professional development (PD) activities focused on teaching Letter knowledge 96.3 92.3 18.2 4.0 (0.506) 74.1* (0.000) Reading fluency 96.3 92.3 16.1 4.0 (0.497) 76.1* (0.000) Reading comprehension 94.6 92.6 17.5 2.0 (0.732) 75.1* (0.000) Vocabulary 94.2 90.2 16.2 4.0 (0.506) 74.0* (0.000) Phonological awareness 86.2 92.2 18.4 −6.0 (0.347) 73.8* (0.000) Percentage of teachers who reported participating in other PD focused on: Teaching reading in K’iche’ or teaching bilingual students 86.1 84.1 8.1 2.0 (0.766) 76.1* (0.000) Making/adapting materials to teach reading 90.5 90.5 15.5 0.0 (1.000) 75.0* (0.000) Creating a suitable classroom environment for learning reading 88.1 82.1 14.3 6.0 (0.392) 67.8* (0.000) Teaching instructions for reading books 70.6 66.6 17.5 4.0 (0.627) 49.0* (0.000) Teaching classroom management 58.1 50.1 16.6 8.0 (0.354) 33.5* (0.000) Use of students’ evaluations 70.3 80.3 13.9 −10.0 (0.242) 76.1* (0.000) Other curricular areas 52.0 52.0 14.5 0.0 (1.000) 37.5* (0.000) Management of multiple-grade classrooms 45.6 37.6 13.4 8.0 (0.364) 24.1* (0.008) Providing support for conducting literacy activities at home 69.9 57.9 10.7 12.0 (0.189) 47.1* (0.000) Conducting reading activities with parents and community members 44.1 20.1 4.0 24.0* (0.003) 16.1* (0.049) Percentage of teachers who reported: Using what they learned from the reading PD daily 58.0 48.0 6.2 10.0 (0.247) 41.9* (0.000) Using what they learned from the reading PD at least twice a week 84.1 78.1 20.7 6.0 (0.424) 57.4* (0.000) Number of teachersa 50 50 48 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a These are the teachers who completed a survey. * Difference in group means is statistically significant at the .05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 20 3. Reading activities in the community Through the household survey, we asked parents/caregivers of the children in the evaluation about the reading-related activities available to their children and families in the community and their participation in those activities. The three main goals for collecting these data were to (1) learn whether the families in Group A knew about the services they were supposed to receive— the services from the community action component of the intervention, (2) learn whether families in Group A had used these services, and (3) identify reading-related activities available to children and their families in the communities in Groups B and C to check that there is a strong enough contrast between the groups. If no other programs other than Leer Juntos, Aprender Juntos are offering activities to promote reading in the group communities, or if there is no substantial crossover (where members of one experimental group receive services intended for another group), then we would expect to see a strong contrast between groups (in particular between Groups A and B, which is the comparison we are interested in making to assess the impacts of the community action component of the intervention). Children in Group A participate in the reading activities offered in the community. As illustrated in Figure IV.2, 80 percent of the households in Group A reported that reading activities or games for children are offered in their community. In contrast, in Groups B and C only 3 and 1 percent of the households, respectively, reported that reading activities or games for children are offered in their community. This result suggests that a majority of the households in Group A are aware that reading activities for children are offered in their communities. In addition, in Groups B and C, it seems that reading activities for children are rarely offered in the community. Figure IV.2. Percentage of households reporting that reading activities or games for children are offered in their community Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. 80 3 1 0 10 20 30 40 50 60 70 80 90 100 Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 77* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 21 Looking at the specific reading activities (shown in Table IV.2), the household survey data suggest that children in Group A were invited to participate in reading activities offered in the community, whereas a small percentage of children in Groups B and C were offered reading activities in their community outside of school: the parents/caregivers of 52 to 57 percent of the children in Group A reported their children were invited to participate in activities like reading camps, story time, and reading buddies, and the parents of at most 1 percent of the children in Groups B and C reported their children were invited to participate in those activities. Table IV.2. Parents’ reports on children’s exposure to and participation in reading activities in the community Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children invited to participate in: Reading camps 57.4 1.4 0.1 56.0* (0.000) 1.3 (0.526) Story time with community volunteers 53.2 1.2 0.0 52.0* (0.000) 1.2 (0.381) Tutoring and reading games with older children (reading buddies) 51.5 0.7 0.0 50.8* (0.000) 0.7 (0.553) Reading fairs 34.0 1.6 0.0 32.4* (0.000) 1.6 (0.336) Borrowing books from a book bank or a library 15.7 1.5 2.5 14.2* (0.000) −1.0 (0.531) Percentage of children who participated in: Reading camps 44.8 0.9 0.0 43.9* (0.000) 0.9 (0.526) Story time with community volunteers 42.5 0.8 0.0 41.7* (0.000) 0.8 (0.381) Tutoring and reading games with older children (reading buddies) 39.4 0.3 0.0 39.1* (0.000) 0.3 (0.553) Reading fairs 25.5 1.2 0.0 24.3* (0.000) 1.2 (0.336) Borrowing books from a book bank or a library 15.1 1.7 1.5 13.4* (0.000) 0.2 (0.914) Number of childrena 452 468 470 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a These are the number of children for whom a household survey was completed. * Difference in group means is statistically significant at the .05 level. In addition, the parents of 16 percent of children in Group A reported their children were invited to borrow books from a book bank or library, and that most of those children actually borrowed books or printed materials from a book bank or a library. Our results on participation in community reading activities suggest that the children in the evaluation were actually exposed to the interventions to which they were assigned, and that children in Group A are participating EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 22 in the reading activities offered in their communities through the community action component of the Leer Juntos, Aprender Juntos intervention. For example, the parents of 26 to 46 percent of the children in Group A reported these children participated in reading camps, story time and reading buddies activities, and reading fairs. In contrast, in Groups B and C, the parents of a very small percentage of children (0 to 1 percent) reported these children participated in the activities of the community component of the intervention. Finally, these results on awareness of and participation in the community component activities provide evidence that there was minimal crossover from families in Groups B and C during the reading activities that are offered only in communities of Group A as part of the intervention. Family/household members of the children in Group A also participated in reading activities offered in the community. About one-third of the household survey respondents in Group A reported that they participated in reading workshops for parents to help their children learn to read, and in 21 percent of the households in Group A at least one family/household member reported participating in reading activities in the community as a volunteer (Figure IV.3). Figure IV.3. Participation in reading workshops and volunteering activities (percentage of households) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. For all the measures of participation in reading activities in the community, we found that participation in Group A was statistically significantly higher than in Group B. This result is not surprising because the Leer Juntos, Aprender Juntos intervention offered the activities of the community component only in the communities of children and their families in Group A, and not in the communities in Group B. 45 43 39 26 1 0 1 0 0 0 1 0 0 10 20 30 40 50 60 70 80 90 100 Reading camps Story time with community volunteers Tutoring and reading games led by older children Reading fairs Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 44* 42* 39* 24* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 23 B. Early impacts In this section, we describe the literacy environments and instruction practices in the evaluation’s Phase I and II 2nd-grade classrooms, and the literacy environment in the homes of the students in the evaluation, based on teacher reports, live classroom observations, and a household survey. These data were collected during the second year of implementation of the evaluation (midline), about 12 to 18 months after randomization of schools, allowing for about one year of potential exposure to treatment differences. Because of this timeline, we would expect to see some early effects of the Leer Juntos, Aprender Juntos program in terms of the classroom environment and children’s participation in reading activities outside of school hours. In addition, instructional practices that could have been influenced by the teacher training might be expected to show differences between groups receiving the program and groups that are not. 1. Literacy materials in the classroom We found impacts of the Leer Juntos, Aprender Juntos intervention on the literacy environment in the classroom, as indicated by the available literacy materials: as shown in Figures IV.4 and IV.5, materials with text in Spanish or K’iche’ and materials to facilitate early reading instruction are available in more classrooms in the treatment groups (A and B) than in the prevailing practice group. Specifically, more classrooms in Group B (22 percent) than in Groups A and C (8 and 9 percent, respectively) have printed materials (such as books and posters that were not handmade by the teachers or the students) with text in K’iche’. In addition, more classrooms in Group B than C have noncommercial handmade materials with text in either Spanish or K’iche’, and printed materials with text appropriate to students’ age/grade (Figure IV.4). Figure IV.4. Available materials with text in the classroom (percentage of classrooms) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. 57 8 90 41 92 67 22 98 49 94 49 9 85 24 79 0 10 20 30 40 50 60 70 80 90 100 Printed materials with text in Spanish Printed materials with text in K’iche’ Noncommercial handmade materials with text in Spanish Noncommercial handmade materials with text in K’iche’ Printed materials with text appropriate for students’ age/grade Percentage Leer Juntos (A) Leer Juntos-school only (B) -14* 13* 25* 16* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 24 We also found impacts on the availability of materials to facilitate early reading instruction in the classroom: a complete alphabet in Spanish and familiar words written in K’iche’ are displayed for students to see in more classrooms in Group B than in Group C. (Figure IV.5). Figure IV.5. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. 2. Teachers’ approach to teaching reading In the in-school component of the Leer Juntos, Aprender Juntos intervention, teachers in Groups A and B were trained to use the Literacy Boost model strategies to provide instruction on five foundational reading skills (alphabet knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension) in conjunction with any other methods or approaches that they regularly employ to teach reading. We asked teachers, through the teacher survey, to report what methods they use to teach reading so that we could learn about how the training provided by Save the Children may have altered the mix of strategies currently in use. In particular, we asked teachers to report whether they use each of the following four methods to teach reading to their students in 2nd grade on a regular basis6 : • Syllabic: This method consists of teaching letter knowledge and phonological awareness. Teachers using this method focus on teaching letters and simple syllables, emphasizing the correspondence between the written syllables and their sounds, and on helping students use 6 To develop the list of four methods included in the survey, we conducted a pilot survey in 2013 with a group of teachers in Guatemala. In that pilot, we used an open-ended question to ask teachers what method/approach they used to teach reading to their students. The answers that most teachers reported during that pilot were the four methods we ended up including in the final version of the teacher survey. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 25 those simple syllables to form short words. After students learn to form short words, the teacher moves on to teach them how to form more-complex syllables and longer words. • Communicative-textual: Teachers using the communicative-textual method focus on teaching reading comprehension and supporting the development of communicative competencies in meaningful contexts by presenting images and text to students, explaining their meaning, and helping students communicate that meaning through oral and written language. • Phonological/global: With this method, teachers typically promote a range of foundational reading skills such as letter knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension. This method has four stages: (1) comprehension, in which teachers show images and objects to teach the meaning of words; (2) imitation, in which teachers guide students in copying short sentences and taking dictation, using the words learned in the previous stage; (3) foundation, in which teachers guide students in reading sentences that use the words and structures learned in the two previous stages; and (4) production, in which teachers ask students questions as a way to explain the meaning of the words and sentences that students learned to write and read. Teachers also use songs and short stories to help explain the meaning of the words and sentences. • Positioning: With this method, teachers focus on teaching letter knowledge using cards with letters and a game that resembles bingo and is based on letters instead of numbers. Some of the teachers who reported using the positioning method also reported that they learned about this method through colleagues or directly from the Guatemalan nongovernmental organization “Asociación de Padres de Familia San Miguel ASOSAM,” which provides teachers with teaching materials and professional development activities that include training in pedagogical methods. Teachers could also report that they use more than one of these methods, a method other than these, or that they do not use a method at all. All teachers in the three groups reported using at least one method to teach reading to their 2nd-grade students. A majority of teachers in Groups A and B reported using the syllabic method to teach reading to their 2nd-grade students (Figure IV.6). Also, more teachers in Group B than C (82 and 46 percent, respectively) reported using the syllabic method, and this difference is statistically significant. This means that more teachers in Group B (and Group A) than in Group C reported incorporating instruction on letter knowledge and phonological awareness—which are the focus of the syllabic method—into their reading teaching practice with 2nd-grade students. We found no statistically significant differences between groups in the percentage of teachers who reported they also use other methods7 . 7 The other methods reported include: (1) phonetic method, (2) onomatopoeic (echoic) method, (3) eclectic method, (4) integrative method, (5) word generating approach, (6) active approach, (7) reading aloud as a group, (8) repetition, (9) songs and games, and (10) Amigos en la Lectura. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 26 Figure IV.6. Teacher-reported approach to teaching reading (percentage of teachers) Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Approaches are not mutually exclusive. * Difference in group means is statistically significant at the .05 level. 3. Teacher-reported use of time We also asked teachers to report the amount of time they spent during their most recent full￾time work week on different activities related to teaching, such as planning and preparing lessons, meeting with their students’ parents, and teaching reading and writing. Contrary to expectations based on the program’s focus on reading instruction, we did not find statistically significant differences between groups in the amount of time teachers reported spending on activities related to their teaching work, with one exception: teachers in Group A reported spending, on average, 30 minutes longer meeting with their students’ parents than did teachers in Group B (Figure IV.7). Teachers in Group C also reported spending about 24 minutes longer meeting with their students’ parents than did teachers in Group B, however, this difference is not statistically significant. On average, teachers in each group reported they spent 3 to 4 hours during the week planning and preparing lessons, and 4.0 to 4.5 hours teaching reading and writing (Figure IV.7). 76 54 46 26 20 82 68 48 28 24 46 59 40 21 23 0 10 20 30 40 50 60 70 80 90 100 Syllabic Communicative-textual Phonological/global Positioning Other method/approach Percentage Leer Juntos (A) Leer Juntos-school only (B) 36* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 27 Figure IV.7. Teacher-reported number of hours in most recent work week spent in activities related to their job Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. 4. Observed practice To measure observed teaching practice, the evaluation team observed reading instruction given to 2nd-grade students during a typical language or reading class. For a period of about 50 minutes (on average, teachers in each group were observed for 48 minutes), observers recorded whether teachers spent time conducting different reading instruction activities and if they used instructional practices focused on teaching children five essential literacy skills: (1) letter knowledge, (2) vocabulary, (3) phonemics and phonological awareness, (4) fluency, and (5) reading comprehension. Contrary to expectations, we did not find any statistically significant difference between groups in teachers’ observed practice. In all groups, teachers spent the majority of the classroom observation period using written text to read aloud to or with students in a group setting and teaching new words and discussing meaning with students (Figure IV.8). We did not find any statistically significant difference between groups in the amount of time teachers spent on those two literacy instruction activities or in other literacy instruction activities such as work with the alphabet and written words and providing opportunities for students to practice reading by themselves (Figure IV.8). Teachers in the three groups also used language while providing reading instruction in similar ways: teachers in each group spent about 60 percent of the observation period providing instruction in Spanish only (31 minutes in Group A, 32 minutes in Group B, and 28 minutes in Group C, on average) and spent about 40 percent of the observation period providing instruction 4.0 0.9 4.3 1.9 0.3 3.4 0.4 4.5 2.0 0.4 3.4 0.8 3.9 1.7 0.3 0.0 1.0 2.0 3.0 4.0 5.0 Planning and preparing lessons Meeting with students' parents Teaching reading and writing Providing additional support to low performing students Activities to promote reading in the community outside school hours Hours Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 0.5* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 28 in both Spanish and K’iche’ (13 minutes in Group A and 14 minutes in Groups B and C, on average). Figure IV.8. Observed teachers’ time using literacy instruction activities (average number of minutes) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. We did not find statistically significant differences between groups in teachers’ use of literacy instruction practices to teach vocabulary, fluency, letter knowledge, phonemics and phonological awareness, and book reading and reading comprehension (Table IV.3).8 No teachers in any of the groups were observed using concepts about print. More teachers in the treatment groups than in than prevailing practice group were observed providing instruction on grammar, mechanics, or spelling, and on phonological awareness, and more teachers in Group B than in Groups A and C were observed encouraging students to use context clues for reading comprehension. However, none of those differences are statistically significant (Table IV.3).9 It is possible that we did not find any differences between groups in the observed classroom practices because one year of implementation of the intervention is not enough time to influence how teachers provide reading instruction. It is also possible that teachers in all groups were already applying the instruction strategies that were emphasized in the in-school component of the intervention, leaving little room for the intervention to make a difference.10 We have 8 The difference between Groups A and B in the percentage of classrooms where teachers were observed teaching fluency (50 percent in Group A and 32 percent in Group B) is not statistically significant at the .05 level, but it is at the .10 level (p-value = 0.076). 9 The difference between Groups B and C is statistically significant at the .10 level (p-value = 0.067). 10 In other words, if the impact of the intervention on teacher practices is very small, it is possible that with a sample of 50 teachers per group we did not have enough statistical power to detect that impact. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 29 explored possible explanations for why we found no differences in the instructional practices observed in the evaluation classrooms. For example, as presented in Table A.1 in Appendix A, we explored whether there are differences between groups in terms of the percentage of first-year teachers in the 2nd grade (that is, teachers who have been teaching in the current school for less than a year), but we did not find differences. We also examined teacher-reported and observed literacy instructional practices by subgroups of teachers defined by tenure in the school (who have taught at current school for (1) more than two years and (2) for two or fewer years), teacher experience (teachers with more than five years of teaching experience and teachers with five or less years of experience), and participation in the evaluation’s baseline and midline data collections (teachers who participated in both data collections, and teachers participating only in the midline data collection). We found that within those subgroups, as with the whole sample, teachers in the three groups are similar in terms of their use of literacy instructional practices (data not shown). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 30 Table IV.3. Observed teachers’ use of literacy instructional practices in the classroom Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of classrooms where teachers used the following practices to: Teach vocabulary Introduced or rehearsed vocabulary words and explained word meaning or elaborated on new concepts 67.9 65.9 62.7 2.0 (0.829) 3.2 (0.736) Teach fluency 50.2 32.2 32.9 18.0 (0.076) −0.7 (0.948) Teach letter knowledge Encouraged students to recognize letters, identify the names of letters, or distinguish upper and lower case 19.6 25.6 23.7 −6.0 (0.458) 2.0 (0.811) Teach phonemics and phonological awareness Provided instruction on phonological awareness 17.6 21.6 17.5 −4.0 (0.621) 4.1 (0.619) Prompted students to segment words into specific units (syllables or phonemes) 5.8 9.8 13.0 −4.0 (0.516) −3.2 (0.608) Provided instruction on grammar, mechanics, or spelling 48.3 40.3 30.7 8.0 (0.423) 9.5 (0.347) Provided instruction on word writing 74.7 80.7 73.5 −6.0 (0.486) 7.2 (0.413) Teach book reading and reading comprehension. Pre-reading or book preview 3.7 5.7 6.8 −2.0 (0.644) −1.1 (0.805) Encouraged predictions 2.0 2.0 −0.1 0.0 (1.000) 2.1 (0.384) Explored children’s interest in the story and facilitated connections with their lives 7.8 7.8 6.7 0.0 (1.000) 1.1 (0.851) Encouraged students to use context clues for reading comprehension 2.1 6.1 −0.1 −4.0 (0.225) 6.2 (0.067) Discussed text structure 0.0 4.0 −0.1 −4.0 (0.086) 4.1 (0.085) Discussed the characters in the text, who they are, their motivation and/or goals 12.2 16.2 7.9 −4.0 (0.547) 8.3 (0.222) Guided students in the use of reading comprehension strategies 14.2 10.2 8.0 4.0 (0.535) 2.2 (0.741) Total number of classroomsa 50 50 48 . . Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. None of the differences presented in this table were statistically significant at the 0.05 level. a These are the classrooms where we conducted an observation. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 31 5. The literacy environment and reading activities in the home We asked the household survey respondents to report, for each evaluation child, on who reads with the child at home, the types of reading activities with the child, and the reading￾related activities that the child does outside of school hours. Family/household members participate in reading activities with the child at home. In most of the households, a family member reads with the child at home. This family member can be the survey respondent or any other member of the family, such as a parent who did not respond to the survey or a sibling. As shown in Figure IV.9, about one-third of the children in each group are read to by an adult (the household survey respondent). This is consistent with the household survey respondents’ reports about their literacy skills: about one-third of survey respondents (34 percent in Group A, 39 percent in Group B, and 35 percent in Group C) reported they are able to read, which means that about two-thirds of household survey respondents in each group are not able to read (these data are not shown in this report). Because of that, we asked whether they look at pictures in books with the evaluation children. We also asked whether other family members read with the evaluation child/children living in the household, and found one statistically significant difference between groups: other family/household members read with a larger percentage of children in Group A than in Group B (Figure IV.9). Figure IV.9. Reading with the evaluation children (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. We inquired about who are the other family members who read with the child, and found an impact of the Leer Juntos, Aprender Juntos intervention on the reading activities that children do outside of school hours: as shown in Figure IV.10, siblings read with a larger percentage of evaluation children in Group A (64 percent) than in Groups B and C (55 percent). This result 30 65 81 34 63 73 31 60 72 0 10 20 30 40 50 60 70 80 90 100 Respondent reads with the child Respondent looks at books with the child Other family member reads with the child Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 8* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 32 suggests that the activities of the community action component of the intervention—the reading buddies activity, for example—are encouraging siblings to read to the children in the evaluation. Figure IV.10. Other family/household members who read with the child (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. For about one-quarter of the children in each group, household survey respondents reported reading textbooks or school notebooks with the evaluation children (23 percent in Group A, 29 percent in Group B, and 24 percent in Group C; these results are not presented in a table or figure). In all groups, on average, household survey respondents spent two days a week reading or looking at books with the evaluation child/children living in the household. Finally, in all groups, during the most recent day that survey respondents read or looked at books with the children, they spent on average six minutes reading books and eight to nine minutes looking at books (these results are not presented in a table or figure). There were no statistically significant differences between groups in the amount of time that the survey respondents spent reading or looking at books with the evaluation children. Parents of the children in the evaluation are interested in finding out how their children do at school. For the majority of the children in each group (at least 81 percent in each group), the household survey respondents reported that they (or another family/household member) have talked to their child’s teacher about the child’s progress in school at least once during the school year that the household survey was administered. We found statistically significant differences between groups on those reports: for a larger percentage of children in Groups A and C (88 and 87 percent, respectively) than in Group B (81 percent), household survey respondents reported that they have talked to the child’s teacher about their child’s progress in school at least once during the school year. 64 17 55 18 55 17 0 10 20 30 40 50 60 70 80 90 100 Sibling Other adult in the household Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 9* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 33 Parents’ expectations about school completion for their children are similar across groups. For about 60 percent of the children in each group, household survey respondents reported they would like their children to complete primary or elementary school (59 percent in Group A, 60 percent in Group B, and 58 percent in Group C). For about 20 percent of the children in each group, household survey respondents reported they would like their children to complete high school (20 percent in Groups A and C, and 22 percent in Group B). Finally, only for 3 to 5 percent of children in each group, household survey respondents reported they would like their children to complete a college education (4 percent in Group A, 3 percent in Group B, and 5 percent in Group C). We did not find any statistically significant differences between groups in the expectations of the family/household about school completion for their children. The percentage of children enrolled in 2nd grade is similar in all groups, but more children in Group A than in Group B had repeated 1st grade. We also asked caregivers of the children in the evaluation about the children’s enrollment in and attendance at school. As presented in Table IV.4, around 70 percent of the children in each group were enrolled in 2nd grade and had been absent from school for at least one day during the school year when the household survey was administered. We found only one statistically significant difference between groups in terms of the children’s enrollment in 1st grade: a larger percentage of children in Group A (31 percent) than in Group B (25 percent) were enrolled in 1st grade in the school year when the household survey was administered (the second year of program implementation, when most of the 1st-grade cohort should have progressed to 2nd grade). More children were attending the 1st grade in Group A than in Group B at the time the household survey was administered because more children in Group A (51 percent) than in Group B (45 percent) have repeated 1st grade (Table IV.4). However, the difference between Groups A and B in the percentage of children who have repeated 1st grade was not statistically significant.11 For all three groups, the repetition rates in the evaluation sample were higher than the repetition rate estimates recently reported by the Ministry of Education in Guatemala (Ministerio de Educación 2016): In 2015, the 1st grade repetition rate was 21.7 percent in the Department of El Quiché and 19.1 percent in all the departments in the country. 11 We found no statistically significant differences at the .05 level. However, the difference between Groups A and B in the percentage of students who have repeated 1st grade (51 percent in Group A and 45 percent in Group B) is statistically significant at the .10 level (the p-value from the test of difference between the group means is 0.062). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 34 Table IV.4. Children’s school attendance, as reported in the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children attending the following grades in the current school yeara 1st grade 30.5 25.0 23.8 5.5* (0.038) 1.2 (0.658) 2nd grade 66.7 69.8 69.2 −3.1 (0.290) 0.6 (0.852) Not enrolled in school 2.8 5.3 7.0 −2.5 (0.120) −1.7 (0.390) Percentage of children who have repeated a grade: Any grade 61.5 56.7 54.6 4.8 (0.164) 2.1 (0.533) Preschool 6.7 7.4 5.7 −0.7 (0.645) 1.6 (0.324) 1st grade 51.0 44.9 45.0 6.1 (0.062) −0.1 (0.977) Both preschool and 1st grade 3.8 4.3 3.8 −0.5 (0.678) 0.5 (0.704) Percentage of students who have been absent from school for at least one day in the current school yeara 71.7 70.7 67.7 1.0 (0.723) 3.0 (0.359) Number of childrenb 452 468 470 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a This refers to the 2014 school year for the Phase I cohort and to the 2015 school year for the Phase II cohort. b The number of children for whom a household survey was completed. * Difference in group means is statistically significant at the .05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 35 Leer Juntos, Aprender Juntos sought to increase availability of children’s books and the time scheduled for reading in the home, but the effects did not emerge by the time we conducted the household survey. A majority of the children in each group (87 percent in Group A, 89 percent in Group B, and 86 percent in Group C) have a scheduled time—set by their parents or caregivers—to spend in reading activities (Table IV.5). On average, the scheduled time for reading at home is three hours per week in each group. We found no statistically significant differences between groups in the scheduled time for reading at home. Table IV.5. Availability of books for children and scheduled time for reading at home Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children who have a scheduled time for reading at home 86.9 88.7 86.0 1.8 (0.429) 2.7 (0.268) Hours per week scheduled for reading at home (average) 2.9 3.0 2.7 −0.2 (0.503) 0.3 (0.143) Percentage of households in which books for children are available 85.8 84.6 82.2 1.2 (0.689) 2.4 (0.389) Number of books for children in the household (average) 2.6 2.7 2.4 −0.1 (0.645) 0.3 (0.185) Percentage of households with books for children in the following languages: Spanish only 57.1 59.3 54.0 −2.3 (0.529) 5.3 (0.137) K’iche’ only 2.1 1.5 1.3 0.5 (0.468) 0.2 (0.819) Spanish and K’iche’ 25.1 22.9 24.8 2.2 (0.459) −1.9 (0.542) Percentage of children using the books for children in the household with the following frequency: Never 14.8 16.7 18.6 −1.9 (0.520) −1.9 (0.496) One or two times per week 40.0 36.0 36.6 4.1 (0.191) −0.6 (0.857) Three or four times per week 16.7 17.5 17.3 −0.8 (0.754) 0.3 (0.908) Every day 28.5 29.8 27.6 −1.3 (0.642) 2.2 (0.454) Number of householdsa 431 445 444 . . Number of childrenb 452 468 470 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. None of the differences presented in this table were statistically significant at the 0.05 level. a The households that completed the household survey. b The number of children for whom a household survey was completed. As shown in Table IV.5, some books for children are available in a majority of the households in the evaluation (82 to 86 percent in each group). On average, households in each group have two to three books for children. On average, 60 percent of the households in each EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 36 group reported having books in Spanish only. About 25 percent of the households in each group reported having children’s books written in both, Spanish and K’iche, and only 2 percent of the households in each group reported having children’s books written in K’iche only. Households also reported the frequency with which the evaluation children use the children’s books available in their home. About two-fifths of the children in each group use the children’s books one or two times per week, and about one-third of the children in each group use the children’s books every day. There were no statistically significant differences between groups in the availability, language, and frequency of use of the books for children available in the home. Children in the evaluation spend time doing reading activities at home. Outside of school hours, the majority of children participate in reading activities such as reading books and doing homework, and about 40 percent of the children in each group spend time hearing or telling stories. In addition, more children in Group A than in Groups B and C participate in reading games in their community after school hours, which is not surprising because children in Group A are receiving the community reading activities of the intervention (Figure IV.11). Figure IV.11. Reading-related activities at home (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. Figure IV.12 shows that on average, children in the three groups read books in about three of the days of the week preceding the household survey, did homework on four days of that week, and heard or told stories on one day of that week. We found evidence of program impacts on the time children spend on reading activities outside of the school hours: on average, children in Group B spend more time during the week reading books (2.8 days) than children in Group C (2.5 days), and children in Group A participate in reading games in their communities after school one day a week, whereas children in groups B and C do not spend time on that activity. 79 94 42 38 80 94 44 2 75 96 39 0 0 10 20 30 40 50 60 70 80 90 100 Read books Do homework Hear or tell stories Reading games in the community after school Percentage Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 36* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 37 Figure IV.12. Number of days in previous week in which child did reading￾related activities at home (average number of days) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the .05 level. Children in the three evaluation groups are not substituting time reading for time on other activities that are not related to reading. As presented in Table IV.6, almost all of the children in each group (97 percent in Group A and 98 percent in Groups B and C) played inside or outside of their house while at home in the week before the household survey was administered, and did so almost every day of that week. Most children (at least 71 percent in each group) also listened to the radio or watched television and helped with household chores such as housework (cleaning, washing, or cooking), taking care of the family’s vegetable garden/corn field and/or livestock, and taking care of younger siblings. For a small percentage of children in each group (4 percent in Groups A and C and 5 percent in Group B) household survey respondents reported that the week before the administration of the household survey the children worked for a payment or wage (Table IV.6). We found one statistically significant difference between groups in the activities not related to reading that children do at home: a larger percentage of children in Group B (85 percent) than in Group A (78 percent) helped with housework in the week before administration of the household survey (Table IV.6). This finding suggests that the community reading activities that the children in Group A had access to may have led them to replace time on household chores with time spent participating in the reading activities offered by the program in their community. On average, children in the three groups listened to the radio or watched television on 4 to 5 days of that week, helped with housework and taking care of younger siblings on 4 days of that week, and helped with the family’s vegetable garden/corn field and animals on 3 days of that week. Table IV.6 also shows that on average, children in Group B helped with housework on 3 4 1 1 3 4 1 0 3 4 1 0 0 1 2 3 4 5 Read books Do homework Hear or tell stories Reading games in the community after school Days Leer Juntos (A) Leer Juntos-school only (B) Prevailing practice (C) 0.3* 1* EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 38 more of the days of the week before the administration of the household survey (4.3 days) than children in Group A (3.8 days). This result is consistent with the result presented earlier on the percentage of children in Groups A and B who helped with housework (cleaning, washing, or cooking). Table IV.6. Children’s activities at home that are not related to reading Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children who did the following in the week before the administration of the household survey: Play inside or outside the house 96.8 98.0 98.0 −1.2 (0.178) 0.1 (0.923) Listen to the radio or watched television 71.4 74.4 76.0 −3.0 (0.380) −1.6 (0.594) Help with housework 78.4 85.4 88.6 −7.0* (0.000) −3.1 (0.056) Help with vegetable garden/corn field and/or with livestock 73.8 71.4 72.7 2.4 (0.461) −1.3 (0.689) Take care of younger siblings 71.3 73.0 71.8 −1.7 (0.536) 1.2 (0.640) Work for pay or wage 3.8 5.3 4.2 −1.5 (0.236) 1.2 (0.302) Number of days (average) in the week before the administration of the household survey in which children did the following: Play inside or outside the house 6.3 6.4 6.3 −0.1 (0.358) 0.1 (0.514) Listen to the radio or watched television 4.0 4.3 4.5 −0.2 (0.228) −0.2 (0.335) Help with housework 3.8 4.3 4.4 −0.5* (0.002) −0.1 (0.711) Help with vegetable garden and/or with livestock 3.3 3.2 3.2 0.1 (0.463) 0.0 (0.948) Take care of younger siblings 3.8 4.0 3.9 −0.1 (0.469) 0.1 (0.713) Work for pay or wage 0.1 0.1 0.1 −0.0 (0.468) −0.0 (0.910) Number of childrena 452 468 470 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a The number of children for whom a household survey was completed. * Difference in group means is statistically significant at the .05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 39 V. SUMMARY OF FINDINGS AND NEXT STEPS Two main findings emerged from the second year of implementation of Leer Juntos, Aprender Juntos. First, evidence from the teacher survey, classroom observation, and school visits suggests there is a meaningful contrast between the treatment groups in the expected direction. Specifically, we found that: • Teachers in the treatment groups (Groups A and B) received the Leer Juntos, Aprender Juntos intervention training for which they were targeted. In the second year of the evaluation, at least 92 percent of the teachers in the treatment groups (A and B) reported participating in reading instruction professional activities through the Leer Juntos, Aprender Juntos intervention, and 100 percent of the teachers in the two treatment groups received at least one coaching visit in their classrooms every three months. • Reading activities (such as reading camps, story hour, peer assistance, and reading festivals) were conducted in the communities in Group A, and the children and their families in this group did participate in these community activities Second, we found evidence of early impacts of the intervention on teaching practices and the literacy environment in the school and at home: • More teachers in the treatment groups (A and B) than in Group C (the prevailing practice group that did not receive the intervention) reported using the syllabic method, which suggests that teachers in the treatment groups, as compared to their counterparts in the prevailing practice group, may be incorporating more instruction on letter knowledge and phonological awareness—which are the focus of the syllabic method—into their reading teaching practice. • Printed materials and other materials to facilitate literacy instruction are available in more classrooms in Groups A and B than in Group C. • More children in Group A than in Group B (and C) read with their siblings at home, suggesting that the activities of the community action component of the intervention such as the reading buddies activity, for example, are encouraging siblings to read to the children in the evaluation; and (4) fewer children in Group A than in Group B helped with housework in the week before administration of the household survey, which suggests that the community reading activities that the children in Group A had access to as part of the intervention might have reduced the amount of housework that they do. The differences between groups in terms of the literacy instructional practices that teachers were observed implementing in the classroom are still not statistically significant. However, the midline measurements were conducted early enough (at the end of the second year of implementation) that there is still time for greater impacts to be realized by the endline measurement that will occur in the third year of program implementation. More reports from this evaluation are forthcoming. The current report presents the main findings from the midline data collection efforts in Guatemala, and a separate report presents the midline findings from Peru. For the two countries, we will conduct final follow-up teacher surveys, classroom observations, and assessments of students’ reading comprehension skills in EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 40 the third year of participation in the evaluation (when the students will be attending 3rd grade). The final follow-up data collection will take place in 2015 for the Phase I cohort and in 2016 for the Phase II cohort in Guatemala and Peru. The findings from the final follow-up data collection of the Leer Juntos, Aprender Juntos evaluation will be presented in a final report for each country, including the findings from the estimation of impacts of the Leer Juntos, Aprender Juntos intervention approaches on 3rd-grade students’ reading skills outcomes in areas such as fluency, decoding, and comprehension. The final evaluation reports (for both Guatemala and Peru) will also present more detailed findings on implementation cost-effectiveness. For each component of the intervention, we will report on the services delivered; whether they were delivered as intended; and if not, why. For the intervention components with documented impact, we will measure the additional cost of implementing each of those components and assess the requirements for implementing them in the future. We will compare the magnitude of any positive effects with the costs of generating those improvements and will express the results in units that allow comparison with other options for investing scarce resources in improving educational outcomes. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 41 REFERENCES Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. “Remedying Education: Evidence from Two Randomized Experiments in India.” Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1235–1264. Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy, vol. 2, 2010, pp. 1–30. Bruns, B., and J. Luque. “Great Teachers: How to Raise Student Learning in Latin America and the Caribbean.” Washington, DC: World Bank, 2014. Comings, John. “Evidence-Based Reading Projects.” Version I. Washington, DC: U.S. Agency for International Development, December 2012. Friedlander, Elliott and Claude Goldenberg, Editors. “Literacy Boost in Rwanda: Impact Evaluation of a 2-year Randomized Control Trial.” Stanford, CA: Stanford University, 2016. Glazerman, Steven, Larissa Campuzano, Nancy Murray, and Camila Fernandez. “Final Leer Juntos, Aprender Juntos Evaluation Plan.” Washington, DC: Mathematica Policy Research, March 22, 2013. King, K.A., and M. Arnal. “Local and Global Dimensions of Language Revitalization.” In Handbook of Indigenous Language Revitalization in the Americas, edited by S. Coronel￾Molina and T. McCarty. London, UK: Taylor & Francis, Ltd., 2016. Lugo-Gil, Julieta, Kara Contreary, Nancy Murray, Steven Glazerman, Larissa Campuzano, Camila Fernandez, and Kate Place. “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Guatemala: Final baseline report.” Washington, DC: U.S. Agency for International Development, 2016. Available at https://pdf.usaid.gov/pdf_docs/PA00MMTD.pdf. Ministerio de Educación. Anuario Estadístico de la Educación 2015. Anuarios Estadísticos, Ciudad de Guatemala: Gobierno de Guatemala, Ministerio de Educación, July 2016. Available at http://estadistica.mineduc.gob.gt/anuario/. Accessed August 30, 2016. National Institute of Child Health and Human Development. Report of the National Reading Panel, Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction. NIH publication no. 00-4769, Washington, DC: U.S. Government Printing Office, 2000. Nieto, Ana Maria. “Modifications of Literacy Boost’s general parent workshop design and parents’ appraisals of the workshops in four communities in Guatemala: Implications for the revision of Literacy Boost’s Parent Workshops.” Cambridge, MA: Harvard Graduate School of Education, November 13, 2015. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 42 Rubio, F. E. “Educación bilingüe en Guatemala: Situación y desafíos.” Document prepared for the workshop Balance y Perspectivas de la Educación Intercultural Bilingüe, Banco Mundial-PROEIB Andes, June 2004. Save the Children. “Literacy Boost Toolkit: Teacher Training.” Washington, DC: Save the Children, 2012a. Save the Children. “Literacy Boost Toolkit: Community Action.” Washington, DC: Save the Children, 2012b. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Report, September 2012–December 2012 (Quarter 1).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2012. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2013– December 2013 (Quarters 2–5).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2013. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2014– December 2014 (Quarters 6–9).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2014. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2015– December 2015 (Quarters 10–13).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2015. Trapnell, Lucy A. “Some Key Issues in Intercultural Bilingual Education Teacher Training Programmes—As Seen from a Teacher Training Programme in the Peruvian Amazon Basin.” Comparative Education, vol. 39, no. 2, special no. 27, May 2003, pp. 165–183. UNESCO. “If You Don’t Understand, How Can You Learn?” Global Education Monitoring Report, Policy Paper 24, Paris, France: United Nations Educational, Scientific and Cultural Organization, 2016. UNESCO. “Education for All 2000–2015: Achievements and Challenges.” EFA Global Monitoring Report, Paris, France: United Nations Educational, Scientific and Cultural Organization, 2015. USAID. “Early Grade Reading and Teacher Training in the LAC Region: Influencing Factors, Lessons and Approaches.” Washington, DC: USAID, June 2011. APPENDIX: BACKGROUND CHARACTERISTICS OF EVALUATION SCHOOLS, TEACHERS, STUDENTS, AND HOUSEHOLDS IN GUATEMALA This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.1 In this appendix, we briefly describe the characteristics of the schools, teachers, and students that we expect to be constant over time or very unlikely to be affected by treatment, as a check on randomization12 and because some of those characteristics may affect the impacts of the intervention and were not measured at baseline. The baseline report (Lugo-Gil et al. 2016) did not show noteworthy differences in the background characteristics of schools, teachers, and students in the evaluation, and the following analysis reinforces that general finding. 1. Schools Schools in the three groups are similar in terms of location, number of classrooms in use, class size, and participation in other education and social programs. Schools in each group are similarly distributed across five municipalities in the Department of El Quiché: San Antonio Ilotenango, Santa Cruz del Quiché, Santo Tomás Chichicastenango, San Andrés Sajcabajá and Zacualpa. The predominant language in the Department of El Quiché is K’iche’. On average, in each group schools have six classrooms in use and have 19 to 20 students enrolled in 2nd grade. We found are no statistically significant differences between groups. The utilities and facilities in the evaluation schools are basic, but a majority of the schools have piped water supply (94 percent in Groups A and C, and 96 percent in Group B), electricity (78 percent in Group A, 80 percent in Group B, and 81 percent in Group C), working restrooms (94 percent in Groups A and C, and 100 percent in Group B) and hand-washing facilities (82 percent in Group A, 86 percent in Group B, and 76 percent in Group C) for children. In the second year of program implementation (midline), we obtained the same results in terms of the schools’ infrastructure characteristics that we did in the baseline year (Lugo-Gil et al. 2016): for the most part, the groups are similar in terms of the utilities and facilities available in the schools. However, as we did in the baseline, in the evaluation midline we found a few statistically significant differences between groups in the available utilities and facilities in the school. For example, a higher percentage of schools in Group B (54 percent) than in Groups A and C (40 and 24 percent, respectively) have potable drinking water. In addition, more schools in Group B (87 percent) than in Group C (59 percent) have functional toilets. Finally, more schools in Group B than in Group A have separate facilities for boys and girls (66 percent in Group A and 46 percent in Group B) and an outdoor recreational space (60 percent in Group A and 42 percent in Group B). We will control for these differences between groups in our analysis of the final impacts of the program. Schools also reported on their participation in regional and national education programs taking place at the school, which reinforced the notion that Leer Juntos, Aprender Juntos was not operating in a vacuum. About 50 percent of the schools in each group (50 percent in Group A, 58 percent in Group B, and 53 percent in Group C) reported they participate in the national reading program Leamos Juntos. Schools in the evaluation also participated in other programs, such as the national children’s nutrition program (22 percent in Group A, 18 percent in Group B, and 26 percent in Group C). We found no statistically significant differences between groups in the participation of schools in regional and national education programs. These results are similar to what we found in the first year of the evaluation (Lugo-Gil et al. 2016): at least one-fifth of the schools in each group (44 percent in Group A, 38 percent in Group B, and 22 percent in Group 12 If random assignment produced equivalent groups, then we would generally expect the schools in the three groups to be similar. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.2 C) reported participating in Leamos Juntos, with no statistically significant differences between groups in the participation of schools in regional and national programs focused on education, health, or nutrition. 2. Teachers We identified one teacher of 2nd-grade students from every evaluation school (selecting randomly if there was more than one) and interviewed the teachers to learn about their personal and professional backgrounds. We did not find statistically significant differences between groups in terms of teachers’ background characteristics (Table A.1). Table A.1. Teacher background characteristics at midline Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Female (percentage) 49.7 57.7 58.9 −8.0 (0.434) −1.2 (0.907) Age (average) 33.6 33.3 33.1 0.3 (0.862) 0.2 (0.905) K’iche’ ethnicity (percentage) 82.0 72.0 70.7 10.0 (0.121) 1.3 (0.838) Highest level of education (percentage) Degree in urban pedagogya 45.9 45.9 50.1 0.0 (1.000) −4.2 (0.639) Degree in rural pedagogya 16.2 20.2 12.0 −4.0 (0.419) 8.3 (0.103) Degree in intercultural bilingual educationb 5.8 15.8 8.7 −10.0 (0.104) 7.1 (0.255) Professoriate 18.2 10.2 12.1 8.0 (0.257) −1.9 (0.789) Years of teaching experience (average) 8.6 8.5 9.7 0.1 (0.906) −1.2 (0.218) Years teaching at current school (average) 6.0 4.5 6.3 1.5* (0.047) −1.8* (0.021) Years teaching 2nd grade (average) 4.0 3.2 3.8 0.8 (0.225) −0.6 (0.345) Percentage of first-year teachers in 2nd grade (in current school for less than a year) 5.8 9.8 6.7 −4.0 (0.465) 3.0 (0.584) Long-term contract (percentage) 69.9 65.9 79.4 4.0 (0.646) −0.6 (0.130) Number of teachersc 50 50 48 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Teachers with a degree in urban (rural) pedagogy have completed a secondary school-level pedagogy program and have practiced in schools in urban (rural) areas. b Teachers with a degree in intercultural bilingual education have the ability to understand, speak, and write in K’iche’ (or the indigenous language of the community where they teach). In addition, teachers with this degree have received training in pedagogical methods to teach bilingual and multicultural students. c These are the teachers who completed a survey. * Difference in group means is statistically significant at the .05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.3 However, we did find statistically significant differences between groups in the tenure of the teachers at the current school: on average, teachers in Groups A and C have taught at their current school for more years (6.0 years in each group) than teachers in Group B (4.5 years, on average). Because teachers who are new to the school in the current school year (2014 and 2015 for teachers in the Phase I and II cohorts, respectively) have not received as much intervention training as the teachers who have been teaching at the current school for at least one year, we explored whether the groups differed in terms of the percentage of first-year teachers in 2nd grade (teachers in current school for less than a year). We did not find statistically significant differences between groups in the percentage of first-year teachers in 2nd grade (Table A.1), so the amount of intervention training that 2nd-grade teachers have received to date should not be a factor in whether teachers’ instructional practices differ across groups. 3. Households As described in Chapter III, we administered a household survey to obtain data on household composition, socioeconomic status, children’s (students’) literacy practices and routines at home, and participation of the children and their families in reading-related activities in the community. The household survey was completed in 1,320 households for 1,390 children in the evaluation, mostly by mothers of evaluation children (Table A.2). Of the 1,320 households, 1,254 had one of the children in the evaluation sample. The other 66 households had more than one child from the evaluation sample—these children were siblings or relatives living in the same household (62 households had two evaluation sample children [124 children] and 4 households had three evaluation sample children [12 children]). In the households with more than one child in the evaluation, all of the children are, by the evaluation’s design, part of the same group. The response rate for the household survey (at the student level) is 93.5 percent. Table A.2. Respondents to the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of households in which survey respondent was the evaluation child’s: Mother 80.9 80.8 83.0 0.1 (0.478) -2.2 (0.352) Father 9.1 10.9 8.2 1.8 (0.478) 2.7 (0.352) Other caregiver 10.0 8.3 8.8 1.7 (0.478) −0.5 (0.352) Number of householdsa 431 445 444 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. None of the differences presented in this table were statistically significant at the 0.05 level. a These are the households that completed the household survey. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.4 As expected, households across the three groups are similar in many ways. We found no statistically significant difference in terms of household composition, parents’ educational background and current work status, and availability of assets, utilities, and commodities. This result is expected because the Leer Juntos, Aprender Juntos intervention is not designed to affect households’ composition and socioeconomic characteristics and because random assignment was used to produce equivalent groups. However, it is important to understand the households of the children in the evaluation because some of those characteristics may affect the impacts of the intervention and were not measured at baseline. In our examination of the impacts of Leer Juntos, Aprender Juntos that will be conducted in the final year of the project, we will take into account any chance differences between groups on households’ background characteristics. A majority of the households in each group are two-parent households. There are no statistically significant differences between groups in the percentage of two parent households (82 percent in Groups A and C, and 81 percent in Group B). As shown in Table A.3, in the three groups there are on average eight people living in the evaluation households, of whom four to five are children. We found one statistically significant difference between the groups: there were more children living in the households in Group A (5 children on average) than in the households in Group B (4 children on average). We found only two statistically significant differences in the educational background and work status of the adults in the household: a larger percentage of other adults in Group C than in Group B (38 and 30 percent, respectively) have never attended school or only attended preschool, and a larger percentage of fathers in Group B than in Group A (94 and 91 percent, respectively) have a paid job (Table A.3). Households in the three groups have similar incomes. Households reported a monthly income of about 1,100 quetzales, or US$145, on average. Average household income is higher in Group C than in Group B and in Group B than in Group A, but these differences between groups are not statistically significant (Table A.3). A potentially important determinant of education outcomes is the wealth of the family. As a proxy for wealth, we tried to measure household assets, such as the quality of the home. Through the household survey, we inquired about the materials used to build the houses where the children of the evaluation live. We found that in all groups, about two-thirds of the families/households (72 percent in Group A, 67 percent in Group B, and 66 percent in Group C) live in a house that has a floor made of raw earth, dirt, or sand. In addition, the roof in the houses of a majority of families/households (90 percent in Group A and 92 percent in each of Groups B and C) is built with basic materials such as hay, palm leaves, cardboard, wood scraps, and fiberglass.13 We did not find any statistically significant difference between groups in the construction materials of the houses of the families in the evaluation. 13 In contrast to rudimentary materials, other materials such as concrete, terra-cotta, and tile are considered finished/refined materials. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.5 Table A.3. Households’ socioeconomic status Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Number of household members (average) 7.7 7.5 7.6 0.2 (0.238) −0.1 (0.428) Number of children in the household (average) 4.6 4.3 4.4 0.3* (0.003) −0.2 (0.088) Number of rooms in the household (average) 3.0 3.1 3.0 −0.1 (0.798) 0.1 (0.297) Percentage of households where the following household members have never attended school or only attended preschool: Mother 69.0 63.0 66.6 6.0 (0.090) −3.6 (0.241) Father 53.7 47.9 48.4 5.8 (0.159) −0.5 (0.898) Other adults who spend time with evaluation child 24.5 30.2 38.0 −5.7 (0.113) −7.8* (0.044) Percentage of households where the following household members have a paid job: Mother 21.9 24.1 24.2 −2.1 (0.430) −0.1 (0.969) Father 90.5 94.3 95.0 −3.8* (0.024) −0.7 (0.627) Other adults who spend time with evaluation child 54.2 57.0 55.4 −2.7 (0.549) 1.5 (0.742) Household monthly income in US dollars 140.3 145.9 153.8 −5.6 (0.673) −7.9 (0.531) Number of householdsa 431 445 444 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a These are the households that completed the household survey. * Difference in group means is statistically significant at the .05 level. Households in the three groups are also similar in terms of their assets and the available utilities and commodities (Figure A.1). A majority of the families/households own land (92 percent in Groups A and C and 91 percent in Group B), and at least 71 percent of the households in each group have phone service and electricity. About 50 percent of households in each group have a television, and 2 to 4 percent of households in each group have a computer (Figure A.1). We found only two statistically significant differences between groups in the utilities and commodities available to the households: on average, more households in Group A than B (84 and 78 percent, respectively) have phone service, and more families/households in Group C than B (8 and 4 percent, respectively) own a car or truck (Figure A.1). Finally, almost all the children (students) in the evaluation sample walk to school (99 percent in Groups A and C and 98 percent in Group B), and the average length of the commute to school is 16 to 17 minutes in each group. There are no statistically significant differences between groups in the type or length of the commute to school. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.6 Figure A.1. Household assets, utilities, and commodities (percentage of households) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group is statistically significant at the .05 level. 4. Students We asked caregivers of the children in the evaluation about the children’s school attendance history. As presented in Table A.4, at the time of administration of the household survey, children in all groups had attended school for three years on average. About 50 percent of the children in each group had attended preschool before enrolling in 1st grade and had been enrolled in preschool for about 8 to 10 months. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.7 Table A.4. Children’s school attendance history, as reported in the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Number of years children in the evaluation have attended school (including preschool), including current school yeara (average) 3.0 3.1 3.0 −0.1 (0.597) 0.1 (0.456) Percentage of children who attended preschool before starting 1st grade 53.4 57.4 52.1 −4.0 (0.402) 5.3 (0.267) Number of years children attended preschool (average) 0.7 0.8 0.7 −0.1 (0.653) 0.1 (0.425) Number of childrenb 452 468 470 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. None of the differences presented in this table are statistically significant. a This refers to the 2014 school year for the Phase I cohort and to the 2015 school year for the Phase II cohort. b The number of children for whom a household survey was completed. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-000 Fax: (202) 216-3524 www.usaid.gov