Latin American and the Caribbean (LAC) Reading Evaluation Contract Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Midline Report July 2017 This publication was produced for review by the United States Agency for International Development under task order AID-OAA-M-12-00020. It was prepared by Julieta Lugo-Gil, Nancy Murray, Camila Fernandez, Steven Glazerman, Ivonne Padilla, and Larissa Campuzano. i Latin American and the Caribbean (LAC) Reading Evaluation Contract Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Midline Report Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH i ABSTRACT This report presents the results from the midline data collection of the Leer Juntos, Aprender Juntos (Read Together, Learn Together) evaluation in Peru. In this evaluation, we randomly assigned schools to three groups: Group A schools implemented the in-school and community action program components; Group B schools implemented only the in-school component; and Group C schools did not implement any of the components of the program. The findings from the midline data collection confirm that the program is being implemented according to plans. Teachers in the treatment groups (Groups A and B) received the teacher training and coaching for which they were targeted. Also, children and families living in the communities in Group A participated in the community-based reading activities for which they were targeted. Moreover, children and families in Groups B and C (which were not assigned to receive the community action component) experienced low levels of exposure to services similar to the community-based reading activities offered to Leer Juntos, Aprender Juntos program participants in Group A. The findings from the midline data collection also provide evidence of program impacts on intermediate outcomes, such as the characteristics of the reading environments and reading activities outside of school hours. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH iii ACKNOWLEDGMENTS The authors gratefully acknowledge numerous people and organizations that contributed to the implementation of all evaluation activities in Peru and the production of this report. The staff at the regional government of Apurímac and the Andahuaylas, Chincheros, and Huancarama provincial education units (known as UGELs, or Unidades de Gestión Educativa Local) provided a warm welcome and essential institutional support for the evaluation. Wilber Venegas, regional governor of Apurímac, and Fernando Acuña, education advisor to the regional governor and director of Education in the Andahuaylas UGEL, provided important support in coordinating evaluation activities in the Apurímac region. The UGEL directors, education specialists, and administrative staff—Wilfredo Siancas, José Porfirio Vivanco Hermoza, Washington Mendoza, Abraham Arenas, Neyser Tello, Kenny Monteagudo, and Celia Reynaga in Andahuaylas; Gary Huamán Tarco in Andahuaylas and Chincheros; Rolando Ochoa Talaverano, Ramiro Ortiz, Francisco Chipana, Luz Elva Villanueva Zevallos, and Tessy Geovanna Camiña Berrocal in Chincheros; and Santiago Benitez in Huancarama—shared key information on the schools and communities in the Apurímac region and offered continuous assistance during our data collection efforts. The staff from Save the Children and Kallpa provided us with key information on program implementation and valuable support in coordinating data collection activities. They include, Katy Anis, Chief of Party for Leer Juntos, Aprender Juntos; Brizza Zuazo, Leer Juntos, Aprender Juntos-Peru project manager; Marízabel Alagón, Save the Children/Peru’s coordinator of monitoring and information; José Carmen Reynaga and Eutropía Delgado, project coordinators, Kallpa; Francisco Zúñiga, Kallpa’s in-school actions coordinator, and Edison Palomino, Kallpa’s community actions coordinator. Kallpa’s technical and administrative project team in the Apurímac region also provided valuable information and assistance in the coordination of the evaluation activities. USAID/Washington and USAID/Peru staff guided and supported the evaluation design and its implementation. They include, from USAID/Washington, Barbara Knox-Seith, senior education advisor, and Michael Lisman, education advisor, Bureau for Latin America and the Caribbean; and while at USAID/Peru, Clifford Brown, deputy chief, Governance & Institutional Strengthening; Natalia Petroni, monitoring and evaluation specialist; Miriam Choy, development assistance specialist; Evelyn Rodriguez-Perez, chief, Health and Education Office; Marisol Pérez, office director, Governance and Institutional Strengthening; Jo Jean Elenes, acting chief, Health and Education Office; Patricia Perez, education advisor, Health and Education Office; and Brindis Ochoa, education project management specialist, Health and Education Office. We thank our local evaluation partner, Grupo de Análisis para el Desarrollo (GRADE) and its staff for high quality data collection in the evaluation schools and communities. Miguel Jaramillo ensured careful planning of evaluation activities in Peru and provided keen oversight of all data collection activities. Elsa Bardalez developed a collaborative relationship with staff at the UGELs and has been instrumental in obtaining local support for the evaluation in the Apurímac region. Juan José Diaz provided overall project management support, oversaw the production of the data sets, and ensured consistency of data programming. Gabriela Guerrero coordinated the design and pilot of the follow-up reading assessments and supervised the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH iv enumerator training. María Balarín coordinated the qualitative data collection on program implementation with assistance of Kristell Benavides. Mariel Bedoya, Carla Pajares, Licet Infantas Díaz, Julio Almonacid, Luciana de la Flor, and Milagros Maldonado coordinated and supervised essential data collection and data entry activities. We are also indebted to Sylvia Linan-Thompson for her thorough and incisive input at different stages of the evaluation. Her expertise in models of reading interventions in bilingual classrooms and ample experience with measurement of early-grade reading skills were essential in designing the evaluation and interpreting baseline findings. At Mathematica Policy Research, Galina Lapadatova and Seth Morgan, provided critical support in data cleaning and programming. We are enormously appreciative of all the school directors and teachers for opening the doors of their classrooms and allowing us to observe the valuable work they do with students. Last, but not least, we thank the students and parents who provided us with the important information we are able to share in this report. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH v CONTENTS ABSTRACT .................................................................................................................................................... i ACKNOWLEDGMENTS................................................................................................................................iii EXECUTIVE SUMMARY ......................................................................................................................... ES-i I. INTRODUCTION .................................................................................................................................... 1 II. The intervention, Prevailing Practice in Peru, and Early implementation overview ............................... 3 A. Three approaches to improving early-grade reading ..................................................................... 3 1. Leer Juntos, Aprender Juntos....................................................................................................... 3 2. Leer Juntos-school only................................................................................................................ 7 3. Prevailing practice......................................................................................................................... 7 B. Timeline of implementation of Leer Juntos, Aprender Juntos in Peru ........................................... 9 III. Evaluation questions and analytical approach ..................................................................................... 11 A. Research questions and data....................................................................................................... 11 B. Analytical approach ...................................................................................................................... 12 IV. Findings ................................................................................................................................................ 15 A. Implementation milestones, challenges, and differences between groups.................................. 15 1. Implementation milestones and challenges................................................................................ 15 2. Teacher-reported participation in professional development ..................................................... 17 3. Reading activities in the community ........................................................................................... 20 B. Early impacts ................................................................................................................................ 23 1. Literacy materials in the classroom ............................................................................................ 24 2. Teachers’ approach to teaching reading .................................................................................... 25 3. Teacher-reported use of time ..................................................................................................... 27 4. Observed practice....................................................................................................................... 28 5. The literacy environment and reading activities in the home ..................................................... 31 V. Summary of findings and Next Steps ................................................................................................... 39 References.................................................................................................................................................. 41 APPENDIX: BACKGROUND CHARACTERISTICS OF EVALUATION SCHOOLS, TEACHERS, STUDENTS, AND HOUSEHOLDS IN PERU .....................................................................A.1 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH vi TABLES IV.1. Teacher-reported participation in selected reading instruction professional development topics .............................................................................................................................................. 19 IV.2. Parents’ reports on children’s exposure to and participation in reading activities in the community...................................................................................................................................... 22 IV.3. Observed teachers’ use of literacy instructional practices in the classroom ................................. 30 IV.4. Children’s school attendance, as reported in the household survey ............................................. 33 IV.5. Availability of books for children and scheduled time for reading at home.................................... 34 IV.6. Children’s activities at home that are not related to reading.......................................................... 37 A.1. Teacher background characteristics at midline ............................................................................A.5 A.2. Respondents to the household survey..........................................................................................A.6 A.3. Households’ socioeconomic status...............................................................................................A.7 A.4. Children’s preschool attendance, as reported in the household survey .......................................A.9 FIGURES II.1. Timeline of Leer Juntos, Aprender Juntos activities 2012—2015 in Peru ..................................... 10 III.1. Leer Juntos, Aprender Juntos evaluation design........................................................................... 11 IV.1. Teachers’ participation in reading instruction professional development activities ....................... 17 IV.2. Percentage of households reporting that reading activities or games for children are offered in their community.............................................................................................................. 21 IV.3. Family members’ participation in reading workshops and volunteering activities (percentage of households) ........................................................................................................... 23 IV.4. Available materials with text in the classroom (percentage of classrooms) .................................. 24 IV.5. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) ........................................................................................................... 25 IV.6. Teacher-reported approach to teaching reading (percentage of teachers)................................... 27 IV.7. Teacher-reported number of hours in most recent work week spent in activities related to their job........................................................................................................................................... 28 IV.8. Observed teachers’ time using literacy instruction activities (average number of minutes) .......... 29 IV.9. Reading with the evaluation children (percentage of children)...................................................... 31 IV.10. Other family/household members who read with the child (percentage of children)..................... 32 IV.11. Reading-related activities at home (percentage of children) ......................................................... 35 A.1. Household assets, utilities, and commodities (percentage of households)..................................A.8 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH ES-i EXECUTIVE SUMMARY A. Introduction This report presents the results from the midline data collection of the Leer Juntos, Aprender Juntos (Read Together, Learn Together) evaluation in Peru. The findings confirm that the program is being implemented according to plans and provide evidence of its impacts on intermediate outcomes, such as the characteristics of the reading environments and reading activities outside of school hours. The Leer Juntos, Aprender Juntos evaluation includes 295 schools in Guatemala and Peru. The sample sizes were similar in the two countries, with Peru accounting for 49 percent of the schools and 42 percent of the students. The research team randomly assigned the schools to three groups—Groups A, B, and C—each taking a different approach to literacy instruction. Schools in Group A received an in-school intervention that includes teacher training and coaching, and a community-based intervention that relies on community volunteers to engage young students and promote a culture of reading outside of regular school instructional periods. The schools in Group B received only the in-school teacher training and coaching component. Schools in Group C did not receive the in-school component or the community-based intervention; rather, they received the prevailing practice approaches available to all schools in each country. For both Guatemala and Peru, we collected data at the beginning of program implementation in the first year of the evaluation (the baseline) and at the end of the second year after baseline (midline); we will collect data again at the end of the third year (endline). Our analysis of the data collected in the second year of the Leer Juntos, Aprender Juntos program implementation (midline) yielded two main findings: 1. The program is being implemented according to its original vision. 2. The program is having an impact on intermediate outcomes, such as the reading environments and reading activities outside of school hours. B. The implementation is following its original vision Our findings confirm that teachers in the treatment groups (Groups A and B) received the teacher training and coaching for which they were targeted. For instance, most teachers in the treatment groups (89 percent in Group A and 84 percent in Group B) reported that they participated in reading instruction professional activities during the second year of program implementation, whereas 53 percent of teachers in Group C did. In addition, most teachers in the treatment groups (at least 77 percent of teachers in Groups A and B) reported that they received training focused on teaching letter knowledge, fluency, reading comprehension, vocabulary, and phonological awareness. These topics are the focus of the teacher training component of the Leer Juntos, Aprender Juntos intervention. Children and families living in the communities in Group A are participating in the community-based reading activities for which they were targeted by Leer Juntos, Aprender Juntos. In addition, children and families in Groups B and C (which were not assigned to receive the community-based intervention) experienced low levels of exposure to services similar to the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH ES-ii ones offered by the community-based intervention component of Leer Juntos, Aprender Juntos. Of the households in Group A, 76 percent reported that reading activities or games for children are available in their community, and only 15 and 18 percent of households in Groups B and C, respectively, reported that those activities are available in their communities. Most children in Group A (56 to 64 percent) participated in the types of reading activities offered in the community as part of the Leer Juntos, Aprender Juntos program, such as reading camps, story time activities, peer assistance (reading buddies), and reading fairs, whereas a small percentage of the children in the other two groups (6 to 11 percent) participated in those activities. The parents of the children in Group A also participated in the community reading activities offered by the intervention: 59 percent of the parents in this group reported participating in reading workshops focused on helping them support their children in learning to read, and in 34 percent of the households in Group A, at least one adult reported volunteering to lead community reading activities offered as part of the intervention. C. The intervention is having impacts on intermediate outcomes The literacy environment in the classrooms of students in the two treatment groups (Groups A and B) is stronger than the typical environment provided in the prevailing practice classrooms (Group C). For instance, in the second year of program implementation, more classrooms in the treatment groups than in the prevailing practice group had materials to facilitate early reading instruction, such as printed materials (posters, banners, and other resources) with text in Quechua and Spanish and Quechua alphabets displayed prominently in the classroom. We also found that more teachers in Group B (receiving the in-school component of the intervention) than in the prevailing practice group (not receiving the intervention) were observed prompting students to segment words into units such as syllables or phonemes (a practice employed to teach phonemics and phonological awareness) and pre-reading or previewing a book (a practice used to teach book reading and reading comprehension). In addition, more teachers in Group B than in Group C reported using the phonological/global method, which suggests that teachers who received the intervention training and coaching, as compared with their counterparts in the prevailing practice group, may be incorporating into their teaching more instruction on a range of foundational reading skills. These skills include letter knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension, which are the focus of the phonological/global method. Our findings suggest that the community action component of the intervention is starting to have an effect on the reading activities that children participate in outside of school hours. More children in Group A than in Group B played reading games with other children in the community outside of the school hours, and a smaller percentage of children in Group A than in Group B helped with chores such as cooking, washing, and cleaning when they were at home. These findings suggest that the community reading activities that the children in Group A had access to as part of the intervention may have reduced the amount of housework that they did. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 1 I. INTRODUCTION In the past 15 years, countries in Latin America and the Caribbean (LAC) have made considerable progress on increasing primary school enrollment and completion among the very poor (United Nations Educational, Scientific and Cultural Organization [UNESCO] 2015). However, challenges persist in improving literacy, particularly for groups with significant language diversity (Benavides et al. 2010). For example, in Peru, 73 percent of the dual-language students1 attending 4th grade in 2015 did not achieve the Spanish reading comprehension level expected for their grade (Ministerio de Educación 2016). Intercultural bilingual education is increasingly used to help poor, marginalized children whose mother tongue differs from their country’s dominant language (Trapnell 2003; U.S. Agency for International Development [USAID] 2011). Intercultural bilingual approaches to education include formal school-based programs with instruction in the mother tongue as well as informal community-wide interventions (King and Arnal 2016). Previous research has shown that children should be taught in their mother tongue (UNESCO 2016), and models aimed at increasing instruction time in the classroom and reading practice time at home have shown promise (Banerjee et al. 2010, 2007; Bruns and Luque 2014). These models include community engagement, trained volunteers teaching low-performing students, and remediation through locally trained teachers. However, comprehensive reading programs that incorporate both language-transition instructional approaches in the classroom and supports for building mother tongue reading abilities outside the classroom have not been rigorously evaluated in the LAC region, and their cost-effectiveness is unknown. To address this need for rigorous evidence, USAID contracted with Mathematica Policy Research. The goal of this contract is to design and conduct an evaluation of the Leer Juntos, Aprender Juntos program—an approach to early-grade reading instruction in LAC communities with linguistically diverse populations. Save the Children developed the Leer Juntos, Aprender Juntos program based on its Literacy Boost model, which includes teacher training and community involvement. The Leer Juntos, Aprender Juntos evaluation includes 295 schools in Guatemala and Peru. The research team randomly assigned the schools in each country to one of three groups: Groups A, B, and C. Schools in Group A received two intervention components: (1) teacher training and coaching and (2) community-based activities aimed to promote reading. Schools in Group B received only the teacher training and coaching component, and schools in Group C did not receive either of the two intervention components. The sample sizes were similar in the two countries, with Peru accounting for 49 percent of the schools and 42 percent of the students. Because Save the Children planned to roll out the program in a phased manner, we also carried out the evaluation in phases. We began random assignment of eligible schools and baseline data collection in about half of the evaluation schools in each country in 2013 (Phase I), and in the other half of schools in 2014 (Phase II). We report results of the samples from both phases combined, in terms of years since baseline. For both phases, we collected data in the first year (baseline, in 2013 for Phase I and 2014 for Phase II), at the end of the second year after baseline 1 Students whose mother tongue is a native language other than Spanish (Ministerio de Educación 2016). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 2 (midline, in 2014 for Phase I and 2015 for Phase II), and at the end of the third year (endline, in 2015 for Phase I and 2016 for Phase II). In the current report, we present findings from the midline data collection in Peru, which included classroom observations and teacher and household surveys. Our findings reveal two major outcomes. First, we found that the two components of the Leer Juntos, Aprender Juntos intervention were implemented as intended. Specifically, we found that that in the second year of the program in Peru, at least 76 percent of the teachers in the treatment groups (A and B) reported participating in reading instruction professional activities through the Leer Juntos, Aprender Juntos intervention, and 100 percent of the teachers in the two treatment groups received at least one coaching visit in their classrooms every three months. In addition, intervention activities aimed at promoting reading were conducted in Group A’s communities, and the children and their families in this group did participate in these community activities. It is important to verify that the teachers and children in the two treatment groups actually receive the appropriate training, services, and activities through the intervention, so that we can attribute observed impacts to the intervention and not to other factors. Second, we found evidence of early impacts of the Leer Juntos, Aprender Juntos intervention in Peru: the literacy environment and the reading teaching practices in the classrooms of the children in the two treatment groups (Groups A and B) is improving. The final (endline) evaluation results will include the findings on the impacts of Leer Juntos, Aprender Juntos on the reading comprehension outcomes of students. These results will be available in 2017. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 3 II. THE INTERVENTION, PREVAILING PRACTICE IN PERU, AND EARLY IMPLEMENTATION OVERVIEW In this chapter, we describe the key elements of the Leer Juntos, Aprender Juntos program, present a summary of the prevailing early-grade reading instruction approach in Peru, and provide an overview of the first two years of implementation of the Leer Juntos, Aprender Juntos program in Peru (2013–2015). A. Three approaches to improving early-grade reading In the evaluation of the Leer Juntos, Aprender Juntos program, we examined three distinct approaches to improving early-grade reading in linguistically diverse communities: (1) an approach combining teacher training and coaching with a community component aimed at creating a culture of reading both in school and outside of school, (2) an approach that includes only teacher training and coaching, and (3) an approach that does not include either the teacher or community components included in the first two approaches, but rather represents the prevailing early-grade reading instruction approach in Peru. We provide more details about these approaches below. 1. Leer Juntos, Aprender Juntos Leer Juntos, Aprender Juntos (Group A, or Leer Juntos for ease of presentation) means Read Together, Learn Together. Save the Children, an international non-governmental organization focused on promoting children's rights and providing relief and support to children in developing countries, developed this program based on its experiences with its Literacy Boost program in El Salvador, Haiti, Ethiopia, Pakistan, and more recently, Rwanda (Friedlander and Goldenberg 2016), among other developing countries. Like Literacy Boost, Leer Juntos, Aprender Juntos combines in-school and community components. The in-school component includes teacher training and coaching. The community component relies on volunteers to engage with young students and promote a culture of reading outside of regular instructional periods at school. The in-school and community components of the Leer Juntos, Aprender Juntos program align with the elements of reading instruction that existing research (Comings 2012) has identified as effective when implemented simultaneously: (1) text and materials, (2) teacher training and support, (3) community and parental support, and (4) assessment and tracking. Therefore, in addition to targeting important changes in the classroom, the Leer Juntos, Aprender Juntos program engages community members and parents in the learning process. The community outreach component is a potentially low-cost, effective way to increase children’s time on task. The Leer Juntos, Aprender Juntos program adapted the teacher training and community action components of Literacy Boost to K’iche’- and Quechua-speaking populations in Guatemala and Peru, respectively. The program’s primary implementers—Save the Children in Guatemala and Kallpa, its local partner in Peru—trained teachers to teach and monitor students’ mastery of core reading skills. Save the Children and Kallpa also worked to strengthen parent and community involvement in building children’s reading abilities in their mother tongue and opportunities to practice. These efforts included creating and providing context-relevant reading EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 4 materials in their mother tongue and community activities that promote reading engagement and a culture of literacy. In-school component: teacher training and coaching. The in-school component of Leer Juntos, Aprender Juntos, which is based on the Literacy Boost model, follows the recommendations of the National Reading Panel for best approaches in reading instruction (National Institute of Child Health and Human Development 2000). The goal of this component is to train and coach teachers so they are better equipped to provide mother tongue and/or Spanish reading instruction focusing on five foundational reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension), to monitor students’ mastery of these skills, and to increase students’ access to and use of mother tongue reading materials. This component incorporates instruction in the mother tongue but adapts to the linguistic background of teachers and students. The school-based component entails six main activities, intended to increase class time on reading instruction and improve the quality of reading instruction: 1. Training trainers in reading instruction techniques 2. Training teachers in the five core skills of reading 3. Creating materials for enhanced, print-rich classrooms 4. Mentoring and coaching of teachers in classrooms 5. Ongoing formative assessments to track children’s progress in reading 6. Guidance in the adaptation of existing curricula to incorporate five core skills of reading and related reading instruction techniques into daily school activities. The teacher training activity (for more details, see the teacher training toolkit [Save the Children 2012a]) consists of nine modules focused on the following topics: 1. Introduction to reading development and instruction for young children 2. Formative assessment 3. Addressing language issues in the literacy classroom 4. Letter knowledge/alphabetic principle 5. Phonological awareness 6. Reading fluency 7. Vocabulary 8. Reading comprehension 9. Reflecting on and applying best practices for teaching reading comprehension and formative assessment EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 5 Teachers receive most of the training content at the beginning of the school year.2 They complete their training throughout the school year. For example, in the first year of Leer Juntos, Aprender Juntos implementation in Peru, teachers were trained on four modules (introduction to reading development, language issues, letter knowledge, and phonological awareness) over two days in May 2013. The remainder of the training was delivered in July (two workshops on fluency and vocabulary modules conducted over four days), October (one-day training on reading comprehension module) and December (one-day training on formative assessment and reflecting and practicing modules). In the second year of program implementation in Peru, teachers were trained on the first eight modules during five days in early March 2014, with one￾day make-up workshops held in late March, April, and July for teachers who could not attend to the early March workshop. Teachers received training on the last module (reflecting and practicing) during a one-day session in December. Following the first training session, the teachers in Peru received a classroom visit from the program’s technical staff (coaches) at least once (and in most cases, two or three times) every three months (Save the Children and Kallpa 2013, 2014, 2015). The purpose of the visits was to support teachers by observing their work, interacting in the classrooms, demonstrating teaching techniques, and suggesting improvements in the application of program teaching strategies. During the coaching visits, teachers requested help in preparing lessons plans (Save the Children and Kallpa 2013). In addition, coaches supported open discussion sessions (called Teacher Circles) with groups of teachers during the afternoons or evenings. These sessions focused on (1) strengthening and practicing the topics learned in the training, (2) conducting activities to develop the five core reading skills, (3) designing lesson plans that incorporate the five core reading skills, and (4) identifying language problems in different contexts and proposing strategies to develop literacy skills in the mother tongue. Community action component. The community action component of the program, also based on the Literacy Boost model, aims to strengthen parental and community involvement in building children’s reading abilities and increasing their opportunities to practice reading outside of school. (For more details, see the Literacy Boost community action toolkit [Save the Children 2012b].) This component is delivered through playful group activities, such as reading camps, reading buddy sessions (peer assistance), reading festivals, and reading contests led by community volunteers. The community action component includes the following main activities (Nieto 2015; Save the Children and Kallpa 2013, 2014, 2015): • Creating printed materials for book banks—collections of books and reading materials—in the children’s mother tongue and/or in Spanish to build portable libraries. In Peru, the program’s team selected, collected, adapted, and produced reading materials covering different genres (such as legends, poems, and stories) in Spanish and Quechua. 2 The school year in Peru begins in March and ends in December. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 6 • Promoting the use of book banks among community members. Each community has at least one book bank. In Peru, program staff indicated that children were enthusiastic about books with foldouts and books with colorful illustrations and large print. • Supporting the implementation of reading activities in the community. These activities were implemented during the school year (March through December) so that invitations for children and their families to participate in the activities could be distributed in the classrooms, and community volunteers could have time off during the school vacation period. The intervention’s community activities are as follows: • Story hour. This activity is usually conducted in the afternoons and in parallel with the distribution of books through community book banks. Story hour activities were conducted about once a week. In Peru, story hour has been a popular activity, because children can attend the activity with their parents and siblings. During story hour, community volunteers, parents, or any other community member tell stories about the community or act out local legends and myths. • Reading camps. In this activity, teachers read stories to children. This is a structured activity that took place outside of school hours. For this activity, the program team first develops a plan s for the reading camps and then shares it with the teachers so they can implement them in their communities. The purpose of the reading camps is to stimulate the development of all five core reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension), in the mother tongue or in Spanish, and incorporate storytelling and reading games to engage children. The reading camps also served as an opportunity for program staff to engage local youth to participate in the community volunteering activities. In Peru, reading camps have been implemented once a week in each school community in Group A. - Reading buddies (peer assistance). In this activity, older children (in grades 4 to 6) read together with the younger children (grades 1 to 3) to help them improve their reading skills. This activity was implemented one or two times per week in each Group A community in Peru, mostly within schools, to maximize the number of children participating in the activity. - Reading festivals. Children participate in games and activities in which they can practice, develop, and strengthen their reading skills. In Peru, reading festivals involved all the community education stakeholders: students, teachers, principals, parents, community volunteers (reading promoters), and local education authorities. In Peru, three reading festivals were conducted in three locations (on school premises as extracurricular activities) in November 2013, reaching participants from 23 Phase I school communities in Group A. Reading festivals were held also in November 2014 in 18 locations (over six days), reaching participants from 49 Group A school communities (Phase I and II). - School–community accountability meetings. Program staff meet with community leaders, government officials from the Apurímac region where the evaluation is being implemented, and authorities from local education administrations (UGELs, Unidades de Gestión Educativa Local) to discuss funding sources of reading activities, progress with the reading activities implemented in schools and the communities, resources used and their purpose, and future sustainability of the activities. In Peru, these meetings were EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 7 held in each community during the second half of 2015, the last year of intervention implementation. • Training volunteers to lead reading activities in the community. The purpose of this training is to enable volunteers to implement the community action activities (such as promoting the use of book banks, reading buddies, and story hour) with children in their communities. In training, volunteers experience each of the community reading activities and develop a weekly schedule of activities for their community to guide the rollout of implementation. In Peru, volunteers in phase I communities received training in August 2013. Identification of volunteers in phase II communities happened in March 2014, and volunteers from all communities (phase I and phase II) participated in training workshops in May 2014 and April 2015. Volunteers received visits from program staff twice a month in the first two years of the project. • Conducting reading awareness workshops with parents and community members. The purpose of these workshops is to promote awareness and understanding among parents and community members about how the community reading activities help children learn to read. The workshops also aim to raise awareness about how important it is for children to start learning to read in their mother tongue. In Peru, reading awareness workshops with parents were held about once every three months in each community in Group A. 2. Leer Juntos-school only The second approach considered in the evaluation is a variant of Leer Juntos, Aprender Juntos that includes the in-school component but not the community component. We refer to this variant as Leer Juntos, Aprender Juntos-school only (Group B, or Leer Juntos-school only for ease of presentation). Including this approach in the evaluation enables us to isolate the effects of the in-school from the effects of the community action component. 3. Prevailing practice The third approach is the prevailing early-grade reading approach currently being implemented by the Ministry of Education in the regions where the evaluation is taking place (Group C, or prevailing practice group). Here, we discuss the strategies currently being pursued in Peru. Information on the strategies being pursued in Guatemala are available in a separate midline report. Since the 1970s, the government of Peru has supported the dissemination of intercultural and bilingual education (IBE) services in schools. The Ministry of Education, through its General Office of Intercultural, Bilingual, and Rural Education (known by its acronym in Spanish, DIGEIBIR), prescribes that schools offering IBE services should have teachers who know and value the cultural heritage of their students and demonstrate oral and written language skills in the prevalent native language of the region as well as in Spanish, and who have been trained in IBE and implement and practice IBE approaches in the classroom (Ministerio de Educación 2013). In addition, IBE schools should have a curriculum and a pedagogical approach that considers knowledge, skills, history, and cultural heritage of the students as well as knowledge of other cultures, and should provide educational materials for different content areas both in the prevalent indigenous language of the region and in Spanish. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 8 The Ministry of Education in Peru advocates a communicative and textual approach (enfoque comunicativo y textual) aimed at educating students to understand, speak, read, and write in their mother tongue and in Spanish, and to use both languages in different communicative contexts within and outside their communities (Ministerio de Educación 2008). The DIGEIBIR recommends that IBE programs initiate instruction of literacy skills in the mother tongue but also provide oral instruction in Spanish. According to DIGEIBIR, children should first acquire literacy skills in their mother tongue and then gradually transition to learning those skills in Spanish. Therefore, in Peru (as in Guatemala) there is not a specific grade level at which instruction should transition fully from mother tongue to Spanish; rather, the transition should be gradual, and each grade level should increase the amount of instruction in Spanish. The expectation for bilingual programs is that in grades 4, 5, and 6, the amount of instruction provided in the mother tongue and in Spanish in bilingual education programs is about the same (Ministerio de Educación 2013). Two key initiatives from the Peruvian Ministry of Education to strengthen children’s literacy skills are the Redes Educativas Rurales (REDES) initiative and the Programa Estratégico Logros de Aprendizaje (PELA) program. REDES is a model of quality rural education that offers teacher support on classroom practices, teacher training, access to information technologies, curricular development for multigrade and single-teacher schools, educational materials in the mother tongue and in Spanish, and improvement in school infrastructure (Ministerio de Educación 2012). The PELA approach to educational policy change uses a performance-based budget strategy aimed at strengthening educational management and administration, teacher training, school infrastructure, and access to quality education (Vargas Dávila 2009). In 2015, the Ministry of Education began two new initiatives: Soporte Pedagógico (Pedagogical Support) targeting urban areas and Soporte Pedagógico Intercultural (SPI; Intercultural Pedagogical Support) aiming to serve bilingual schools located primarily in rural areas. The goal of these initiatives is to improve student learning in elementary schools by (1) providing training workshops, teachers’ peer-to-peer study groups, virtual educational support for teachers and principals; (2) implementing learning sessions designed to develop students’ communication, math, and social studies skills that require additional support; (3) executing efficient mechanisms for timely allocation and use of appropriate teaching materials and educational resources; and (4) building capacity of principals, teachers, and parents to ensure adequate learning conditions for students. In addition, these initiatives employ teachers who work after regular school hours (docentes fortaleza) to provide remedial support to students who are falling behind. While the two initiatives aim to improve the quality of education and student achievement, SPI emphasizes the implementation of the Ministry of Education’s bilingual intercultural education model with strong bilingual literacy acquisition (in mother tongue and in Spanish) and cultural diversity components. In addition to national-level programs to promote literacy among the Quechua-speaking population, schools in the control group can also be subject to local decisions about education priorities and practices. Both national and local efforts likely influence the curriculum in use in a particular school. Therefore, the control group in this evaluation must be understood to include exposure to practices and programs aimed at improving early-grade literacy. It is the effect of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 9 Leer Juntos, Aprender Juntos program above and beyond these coexisting efforts that we seek to discover in the present evaluation. B. Timeline of implementation of Leer Juntos, Aprender Juntos in Peru In Peru (as in Guatemala), the schools and communities in Group A (Leer Juntos group) received the in-school (teacher training and coaching) and community-based activities components of the intervention. Schools in Group B (Leer Juntos-school only group) received the teacher training and coaching component, but the community action component of the intervention was not implemented in this group. Neither component of the Leer Juntos, Aprender Juntos intervention was implemented in schools or communities in the prevailing practice group (Group C). Planning for the start of implementation of the Leer Juntos, Aprender Juntos program activities in Groups A and B in Peru began in fall 2012 with the development of a work plan and project strategy, hiring of program implementation staff, a training workshop for key implementation staff from Save the Children, and meetings of key stakeholders, for example, USAID, local education authorities, and relevant local government entities (Save the Children and Kallpa 2012, 2013). Preparing for rolling out the program in Peru continued in winter and early spring of 2013; with activities that included recruiting and orienting technical and project administration staff, developing an implementation monitoring and evaluation system, mapping elements of the teacher training with the national curriculum, translating project materials to Quechua, producing reading materials for local book banks, and recruiting schools to participate in the evaluation. Figure II.1 presents the timeline of Leer Juntos, Aprender Juntos activities in Peru from 2013 to 2016. The school year in Peru begins in March and ends in December. Random assignment for the Phase I cohort was conducted in April 2013. The Leer Juntos, Aprender Juntos intervention was rolled out in Peru in May 2013 with the first teacher training sessions for Phase I teachers in Groups A and B. The teacher training continued in 2013 with sessions conducted in July, October, and December. Data collection for the baseline of the Phase I cohort occurred in September and October 2013, about two to three months after teacher training began.3 Recruitment and training of the community volunteers happened in August 2013 in Group A Phase I communities, and implementation of the community-based intervention activities such as reading banks, story hour, reading buddies, and reading camps began in September 2013 (Figure II.1). For the Phase II cohort, random assignment was conducted in March 2014. The first teacher training sessions for Phase II teachers in the treatment groups were held also in March 2014 (Phase I teachers also participated in this training session), and teacher training for Phase II 3 While it would have been ideal to have the baseline data collected prior to the rollout of teacher training in Phase I schools, it was not possible to procure a local data collection firm in the time between reaching an agreement with stakeholders on the evaluation design (April 2013) and the roll out of the teacher training (May 2013). We submitted and agreed upon the evaluation design with stakeholders in April 2013, and only after that we were able to share the evaluation design in the terms of reference in the request for proposals from prospective data collectors. We also had to allow time for preparation and review of proposals, and once we had a data collection partner, GRADE, for them to recruit and train interviewers. For Phase II schools we were able to accomplish baseline data collection closer to the time of teacher training roll out. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 10 teachers continued with sessions held in April, July, and December. Data collection for the baseline of the Phase II cohort was conducted in April and May 2014.The first training of community volunteers for Phase II took place in May 2014, as did the start of implementation of the community action components in Group A (Figure II.1). Data collection for the midline of the Phase I cohort was conducted in November and December 2014. The midline data collection for the Phase II cohort was conducted from August through October 2015 (Figure II.1). Figure II.1. Timeline of Leer Juntos, Aprender Juntos activities 2012—2015 in Peru Note: The school year in Peru begins in March and ends in December. The months of school recess are blacked out. RA = random assignment of schools; BL = baseline data collection; TT = rollout of teacher training and support component; CA = rollout of community action component; ML = midline data collection; EA = end of intervention implementation activities; EL = endline data collection. The teacher training and support (coaching) activities, as well as training of teacher leaders and principals, continued throughout 2014 and 2015 for the Phase I and II cohorts in the treatment groups (A and B). Recruitment and training of community volunteers and community reading activities also continued in 2014 and 2015 for the Phase I and II communities in Group A. The implementation of the teacher training and support and community components of the intervention (Groups A and B) ended in December 2015 for the Phase I and II cohorts. The final data collection (endline) took place in September and October 2015 for the Phase I cohort and in August and September 2016 for the Phase II cohort (Figure II.1). A final report with the findings based on the endline data collection for the two cohorts will be available in the fall of 2017. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 11 III. EVALUATION QUESTIONS AND ANALYTICAL APPROACH The purpose of the Leer Juntos, Aprender Juntos evaluation is to determine the relative impact and cost-effectiveness of the three approaches: Leer Juntos (A), Leer Juntos-school only (B), and prevailing practice (C). As outlined in the Leer Juntos, Aprender Juntos evaluation plan (Glazerman et al. 2013), the primary outcomes of interest for the final impact analysis (available in 2017) will be early-grade reading skills, such as fluency and comprehension, but we will also examine other outcomes, such as improved classroom practices and culture of reading at home and in schools. To provide rigorous evidence on the relative impacts of the three alternative approaches, we randomly assigned schools to the three intervention groups (Figure III.1). The research team selected and assigned 74 schools to intervention groups in 2013 (Phase I) and 71 schools in 2014 (Phase II).4 Figure III.1. Leer Juntos, Aprender Juntos evaluation design Source: Leer Juntos, Aprender Juntos evaluation plan (Glazerman et al. 2013) A. Research questions and data The evaluation aims to answer two primary research questions: 1. What is the impact of the community action component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to an intervention that does not have the community action component? To answer this question, we compared outcomes for schools in Group A with those in Group B. 4 A similar number of schools were assigned in the same manner in Guatemala: 75 schools in Phase I and another 75 in Phase II. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 12 2. What is the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to prevailing practice? To answer this question, we compared outcomes for schools in Group B with those in Group C. After randomly assigning the 145 schools to the three groups, the evaluation team randomly selected one 1st grade classroom, one teacher, and 10 1st grade students from each school to include in the analysis sample. We used this approach to minimize data collection cost and burden while still obtaining data that were representative of the schools in the evaluation. The Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016) presents the findings from the examination of the baseline data, collected from a sample of students in 1st grade at baseline. This midline report focuses on the data from the second year of program implementation for both Phase I and II children, when most of the children in the evaluation would have progressed to 2nd grade. Therefore, it describes the evaluation sample after about one year of potential exposure to the intervention. We conducted random assignment in April 2013 for the Phase I cohort and in March 2014 for the Phase II cohort. Thus, the midline data collection took place about 15 to 19 months after randomization of schools. For the midline data collection, we observed schools and classrooms, interviewed teachers, and interviewed parents or main caregivers of the children in the evaluation. The school observation used a checklist that could be completed based on an environmental scan of the school premises. During a one-hour class period, the evaluation team observed and recorded information on teachers’ instructional practices, teachers’ and students’ language use in the classroom and time on task, and other school and classroom characteristics. The evaluation team also administered an in-person survey of teachers about their instructional practices; education and experience; participation in professional development activities; occupational needs; career expectations; and background characteristics, such as Quechua proficiency and income. Finally, the evaluation team administered an in-person survey of the main caregivers (usually the mother or father) of the children in the evaluation to learn about household composition, family socioeconomic status, household assets, children’s schooling background and routines at home, and the participation of children and families in reading activities offered in their communities. B. Analytical approach We examined differences between groups in measures at the school level (characteristics of schools, teachers, and classrooms) and at the student level (characteristics of students’ households). To assess the differences between groups in those measures, we conducted regression analyses that took into account the design of the random assignment of schools5 and the data cohort (Phases I and II). With those regressions, we computed regression-adjusted means of the relevant measures for each group (A, B, and C) and tested whether the differences in those adjusted means between Groups A and B, and between Groups B and C, are statistically different from zero. We used the 5 The design used stratified random assignment, in which we first grouped schools with similar characteristics (within phase) into strata and randomly assigned to the three groups within these strata. This approach increases the likelihood that certain observable characteristics are evenly balanced across the three groups. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 13 p-values from the tests of differences in group means to assess statistical significance of the differences in means. We defined the difference in means between groups as statistically significant whenever the p-value of the test was lower than 0.05. In determining the statistical significance of mean differences between groups in measures at the student level, we took into account the clustering of students within schools, school size, and student-level nonresponse. For more details on how we conducted the stratified randomization of schools; the sample selection; and the methods we used to estimate the differences between groups in measurable characteristics of schools, teachers, classrooms, and students, refer to the Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016). This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 15 IV. FINDINGS This chapter presents our findings on the differences in implementation of approaches to improve early-grade reading among the groups, and the early impacts of Leer Juntos, Aprender Juntos. These results are based on data collected in the second year of program implementation for both Phase I and II samples (midline) through classroom observations and teacher and household surveys. The midline data also allowed us to confirm the baseline finding that the experimental groups were well balanced, by including household information on the children in our sample. We examined differences between groups in the background characteristics of schools, teachers, and children in the second year of program implementation (at midline), and present those findings in Appendix A. We considered background characteristics that are constant over time or unlikely to be affected by treatment, because some of those characteristics could affect the impacts of the intervention and were not measured at baseline.6 As with our examination of the groups at baseline (Lugo-Gil et al. 2016), we found no differences between groups in terms of school infrastructure and teachers (Appendix A, Table A.1), nor did we find noteworthy differences between groups in terms of children’s participation in preschool or households’ socioeconomic characteristics (Appendix A, Figure A.1 and Tables A.2 through A.4). However, our endline analysis will control for child-level and school level variables that might influence student outcomes. A. Implementation milestones, challenges, and differences between groups In Chapter II, we describe how the Leer Juntos, Aprender Juntos program was planned. In this section, we present our examination of the program implementation milestones and challenges in Peru and the differences in available teacher training and community-based activities between the treatment groups and the nonintervention group in the second year of program implementation. The main goals of this analysis are to (1) assess whether the Leer Juntos, Aprender Juntos program was implemented according to plan and (2) understand what is happening in Group C (the counterfactual), including how the prevailing practice may have changed since baseline. 1. Implementation milestones and challenges The Leer Juntos, Aprender Juntos team achieved several milestones in the first two years of program implementation in Peru with the Phase I and II cohorts (2013–2015). As documented by the program staff in Peru, these implementation milestones include the following (Save the Children and Kallpa 2013, 2014, 2015): • Delivery of all training modules to Phase I and II teachers in 2013 and 2014, respectively (435 teachers received training by the end of 2015) 6 We were able to conduct only one household survey. For logistical and conceptual reasons, we chose to conduct the household survey at midline, to assess early effects of the community intervention as well as collect household socioeconomic and demographic characteristics that we may want to control for in the final impact analysis. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 16 • A minimum of two technical support visits every three months completed with each teacher from the Phase I cohort in 2013, and at least one technical support visit every three months completed with each teacher from the Phase I and II cohorts in 2014 and 2015; around 3,000 technical support visits to teachers and schools were completed from 2013 to 2015 • Establishment of one book bank in each of the 49 communities in Group A, with about 250 books and printed materials (of which about 80 titles were produced by Saved the Children specifically for use in the book banks) available to children in each community and thousands of loans made in the communities of group A every three months • Delivery of training to 171 community volunteers in 2013 and 2014 • Implementation of all community action component activities in each of the 49 communities in Group A, with about 1,000 reading camps, reading buddies, and story hour activities implemented in 2013 and in 2014; annual reading festivals conducted in all communities in Group A; and at least one reading awareness workshop conducted with parents every three months from October 2013 through the end of 2015 The Leer Juntos, Aprender Juntos team also experienced challenges while implementing the intervention. During the early stages (Phase I, first year), the main challenges were: • Engaging teachers and ensuring that they completed all the modules of the intervention training before the school year ended • Reluctance from some teachers to use Quechua as the language of literacy instruction • Difficulties in identifying, engaging, and retaining volunteers to lead the community-based reading activities As implementation of the intervention progressed, the evaluation team encountered further challenges (Save the Children and Kallpa 2013, 2014, 2015): • Scaling-up the implementation of the in-school component of the program to the entire evaluation sample in the second year of implementation (2014) when Phase II teachers entered the evaluation • Disruption of community-based activities during national and school holidays • High turnover among the volunteers leading the community-based reading activities • Kicking off some of the activities of the community action component—for example, the implementation of book banks and reading camp activities in Phase I communities in Group A started in September 2013, but implementation of reading buddies activities did not start until April 2014 • Hesitation from parents and community members to let their children participate in reading activities in Quechua • Members of a teacher union in one school issued radio announcements with the goal of discouraging participation of other schools in the evaluation and in Leer Juntos, Aprender Juntos program activities—for example, some of the announcements stated that the goal of the evaluation was to test people as if they were laboratory rats. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 17 2. Teacher-reported participation in professional development Results from the teacher survey suggest that teachers in the treatment groups (A and B) received from Leer Juntos, Aprender Juntos the training for which they were targeted. Because teachers in Groups A and B received the teacher training and coaching component of the intervention, we do not expect to see statistically significant differences between those groups in their participation in reading instruction professional development (PD) activities. However, because the teachers in the prevailing practice group (Group C) did not receive the intervention’s training and coaching, we expect to see statistically significant differences between Groups B and C in their participation in reading PD. Figure IV.1 shows teacher-reported participation in reading PD at midline. Figure IV.1 is the first of a series of figures we present in this chapter. The colored bars in each figure represent groups (Leer Juntos [Group A] is dark blue, Leer Juntos-school only [Group B] is medium blue, and prevailing practice [Group C] is light blue). The group average or mean is placed within each bar. When the difference between groups is statistically significant, the magnitude of the difference is presented alongside the Group B bar. Statistically significant differences between Groups A and B are on the top left side, and statistically significant differences between Groups B and C are on the top right. A large percentage of the teachers in the two treatment groups (89 percent in Group A and 84 percent in Group B) reported participating in PD during the current school year (the second year of program implementation), as illustrated in Figure IV.1. The difference from the prevailing practice group was large and statistically significant, although, most teachers in Group C schools (53 percent) reported participating in reading PD during the current school year. Figure IV.1. Teachers’ participation in reading instruction professional development activities Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 18 As expected, most Group A and Group B teachers reported getting their training through Save the Children, whereas most Group C teachers received their training from the Ministry of Education. More than a third of teachers in each of the intervention groups reported having received training from the Ministry of Education. There is minimal evidence that teachers in Group C (the prevailing practice group) received the intervention services intended for the teachers in the treatment groups: only 3 percent of the teachers in the prevailing practice group reported participation in reading PD provided by Save the Children through Leer Juntos, Aprender Juntos. When asked about the focus of their professional development, most teachers in the two Leer Juntos, Aprender Juntos treatment groups reported higher rates for targeted literacy skills than did teachers in Group C. As we show in Table IV.1, more than 76 percent of the teachers in Groups A and B reported receiving training on teaching letter knowledge, reading fluency, reading comprehension, vocabulary, and phonological awareness. In contrast, only 30 to 51 percent of the teachers in the prevailing practice group reported participating in PD focused on teaching specific literacy skills. The difference between Groups B and C is statistically significant for all types of PD focused on teaching specific literacy skills. This finding aligns with our expectations, because the intervention provided teacher training aimed at improving instruction of the five core literacy skills: letter knowledge, phonological awareness, reading fluency, vocabulary, and reading comprehension. Teachers also reported participation in PD activities focused on other topics related to reading instruction (Table IV.1): at least 62 percent of teachers in each of the treatment groups (A and B) reported participation in PD focused on teaching reading in Quechua or teaching bilingual students, making/adapting materials to teach reading, and creating a suitable classroom environment for learning to read. Around 73 to 76 percent of teachers in the intervention groups reported receiving training focused on using students’ evaluations to inform their practice. Seventy-one percent of teachers in Group A also participated in training focused on providing support to students and their parents for conducting literacy activities at home and the community. In comparison, only 10 to 43 percent of the teachers in Group C reported participating in those types of PD. The difference between Groups B and C is statistically significant for all of these PD activities focused on other topics related to reading instruction, with the exception of PD activities focused on instruction in curricular areas not related to reading (Table IV.1). Finally, teachers in the treatment groups reported that they frequently use what they learned. At least 61 percent of the teachers in each of Groups A and B reported using what they learned from the PD every day, and 81 to 84 percent of teachers in the treatment groups reported using what they learned at least twice a week. In contrast, 41 percent of the teachers in Group C reported using what they learned from the PD every day, and 53 percent of teachers in this group reported using what they learned at least twice a week (Table IV.1). This finding suggests that the intervention targeted the reading instruction topics that interest teachers and that teachers find most useful. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 19 Table IV.1. Teacher-reported participation in selected reading instruction professional development topics Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of teachers who reported participating in professional development (PD) activities focused on teaching: Letter knowledge 76.7 78.0 30.1 -1.3 (0.887) 47.8* (0.000) Reading fluency 78.7 77.5 44.9 1.2 (0.888) 32.6* (0.000) Reading comprehension 89.1 83.6 51.1 5.5 (0.495) 32.5* (0.000) Vocabulary 80.8 82.0 38.5 -1.1 (0.894) 43.5* (0.000) Phonological awareness 77.1 80.0 29.9 −2.9 (0.741) 50.0* (0.000) Percentage of teachers who reported participating in other PD focused on: Teaching reading in Quechua or teaching bilingual students 66.7 62.8 30.2 3.9 (0.710) 32.6* (0.003) Making/adapting materials to teach reading 68.4 66.9 43.0 1.5 (0.885) 23.9* (0.023) Creating a suitable classroom environment for learning reading 70.8 70.7 33.8 0.1 (0.994) 37.0* (0.000) Teaching instructions for reading books 71.1 64.1 31.6 7.0 (0.446) 32.5* (0.001) Teaching classroom management 50.5 59.6 32.3 -9.1 (0.372) 34.8* (0.001) Using students’ evaluations 72.5 75.6 40.8 −3.0 (0.756) 34.8* (0.001) Other curricular areas 48.0 48.6 43.2 -0.7 (0.947) 5.4 (0.594) Managing multiple-grade classrooms 27.3 29.4 9.8 -2.1 (0.798) 19.6* (0.020) Providing support for literacy activities at home 70.7 48.5 30.3 22.2* (0.024) 18.2 (0.065) Conducting reading activities with parents and community members 64.5 40.3 19.9 24.2* (0.009) 20.4* (0.029) Percentage of teachers who reported: Using what they learned from the reading PD daily 66.6 61.4 40.8 5.2 (0.601) 20.6* (0.044) Using what they learned from the reading PD at least twice a week 80.8 83.7 53.2 -2.9 (0.729) 30.5* (0.001) Number of teachersa 49 46 48 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Teachers who completed a survey. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 20 3. Reading activities in the community Through the household survey, we asked parents and caregivers of the children in the evaluation about the reading-related activities available to their children and families in the community and their participation in those activities. The three main goals for collecting these data were to (1) learn whether the families in Group A were aware of the services available from the community action component of the intervention, (2) learn whether families in Group A have used these services, and (3) identify community reading activities available to children and families in Groups B and C. Some respondents in Group A could report that these services are not available to them if they were not aware of the services or the services were not offered as intended. Similarly, when families in Groups B and C reported that community reading activities were available to them (any value greater than 0 percent), their response could indicate either a prevailing level of services already offered in the community or that members of these two groups (Groups B and C) “crossed over” and received services that were not targeted for them. Children in Group A participate in the reading activities offered in the community. Three-quarters of the households in Group A (76 percent) reported that their community offers reading activities or games for children, suggesting that not all the evaluation families in the communities in Group A were aware of the activities that the intervention offered (Figure IV.2). In Groups B and C, 15 and 18 percent of the households, respectively, reported that that their community offers reading activities or games for children. Those percentages are relatively low; however, it seems that other providers (different from the Leer Juntos, Aprender Juntos intervention) might be offering reading activities in the communities of households in Groups B and C, or that families in Groups B and C may have been aware of the services offered by the intervention in the communities in Group A. The difference between Groups A and B is large and statistically significant (61 percentage points), which suggests that the community reading activities experienced by households in Group A represent a meaningful contrast from the activities available in the communities where the program did not offer the community component (Figure IV.2). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 21 Figure IV.2. Percentage of households reporting that reading activities or games for children are offered in their community Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. Looking at the specific reading activities (shown in Table IV.2), the household survey data suggest that children in the evaluation were offered the activities that corresponded to the group to which they were assigned. When asked whether their children were invited to participate in reading camps, story hour, reading buddies activities, and reading fairs, the parents/caregivers of at least 60 percent of the children in Group A reported that they were indeed invited to participate in those activities. In comparison, parents/caregivers of a smaller percentage of children in Groups B and C reported their children were invited to participate in those community activities. In addition, the parents/caregivers of 46 percent of children in Group A reported their children were invited to borrow books from a book bank or library, but more than 50 percent said that children in that group actually borrowed books or printed materials. 7 7 In all groups, the percentage of households that reported they borrowed a book is larger than the percentage of households that reported they were “invited” to borrow a book. It is likely that families borrow books if they are available without waiting to receive a formal invitation to do so. The survey questions were not structured in such a way as to be able to determine this, however. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 22 Table IV.2. Parents’ reports on children’s exposure to and participation in reading activities in the community Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children invited to participate in: Reading camps 61.9 10.7 13.6 51.1* (0.000) -2.9 (0.499) Story time with community volunteers 59.5 10.4 11.8 49.1* (0.000) -1.4 (0.737) Tutoring and reading games with older children (reading buddies) 64.6 8.6 13.1 56.0* (0.000) -4.5 (0.238) Reading fairs 67.0 11.1 11.2 55.9* (0.000) -0.2 (0.962) Borrowing books from a book bank or a library 45.5 22.3 15.5 23.2* (0.000) 6.7 (0.133) Percentage of children who participated in: Reading camps 58.2 8.7 11.4 49.5* (0.000) -2.7 (0.505) Story time with community volunteers 55.6 8.6 10.1 47.1* (0.000) -1.5 (0.690) Tutoring and reading games with older children (reading buddies) 60.9 6.4 11.2 54.4* (0.000) -4.8 (0.185) Reading fairs 63.5 9.4 9.5 54.1* (0.000) -0.2 (0.967) Borrowing books from a book bank or a library 55.3 36.1 25.2 19.3* (0.001) 10.9 (0.055) Number of childrena 366 338 333 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Number of children for whom a household survey was completed. * Difference in group means is statistically significant at the 0.05 level. Our results on actual participation (Table IV.2), as opposed to what we found regarding invitations to participate, confirm the finding that most children in the evaluation were exposed to the community activities that corresponded to the groups to which they were assigned. That is, the findings confirm that the intervention was implemented as planned: most of the children in Group A were exposed to the community reading activities of the Leer Juntos, Aprender Juntos intervention, and most of the children in Groups B and C were not. In the month before the household survey was administered, 56 to 64 percent of the children in Group A participated in reading camps outside school hours, story time activities in which volunteers read aloud to the children, activities in which older children provided tutoring and played reading games with the evaluation children (reading buddies), and reading fairs/festivals in which they read with other children in the community. Six to 11 percent of the children in Groups B and C reported participation in reading activities and games in their communities in the previous month. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 23 Family/household members of the children in the evaluation also participated in community reading activities. Findings on participation of adult family members in community activities related to reading also confirmed that (1) families in Group A are receiving the program services that they are supposed to receive as part of Leer Juntos, Aprender Juntos, and (2) there was minimal prevailing availability of these activities or crossover from families in Groups B and C during the study. Fifty-nine percent of the household survey respondents in Group A reported that they participated in reading workshops for parents to help their children learn to read. In 34 percent of the households in Group A, at least one family/household member reported participation in any of the reading activities in the community as a volunteer (Figure IV.3). These levels differ from the corresponding values for Group B and C by statistically significant margins of more than 20 percentage points. As with many of these measures of program implementation, the treatment contrast is not zero, nor is it as large as 100 to 0. However, the contrast between Group A (assigned to receive the community action component of the intervention) and Groups B and C (not assigned to the community action component of the intervention) is meaningful given the large statistical significant differences in participation between those groups. Figure IV.3. Family members’ participation in reading workshops and volunteering activities (percentage of households) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. B. Early impacts In this section, we describe the literacy environments and instruction practices in the study’s Phase I and II 2nd grade classrooms, and the literacy environment in the homes of the students in the evaluation. We base these results on data from teacher reports, live classroom observations, and a household survey. We collected these data during the second year of implementation of the evaluation (midline), about 15 to 19 months after randomization of schools, allowing for about EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 24 one year of potential exposure to treatment differences. Therefore, we would expect to see some early effects of the Leer Juntos, Aprender Juntos program in terms of the classroom environment and children’s participation in reading activities outside of school hours. In addition, instructional practices that could have been influenced by the teacher training might be expected to show differences between groups receiving the program and groups that are not. 1. Literacy materials in the classroom We observed that in the second year of implementation of Leer Juntos, Aprender Juntos in Peru, the program was operating in an environment in which literacy materials were already universally available in Spanish, but we found evidence that the program helped increase the availability of literacy materials in the mother tongue (Quechua) in classrooms in the treatment groups (Groups A and B). As we show in Figure IV.4, printed materials (such as books and posters), noncommercial handmade materials with text in Spanish, and printed materials with text that is appropriate for the children’s age or grade are available in almost all classrooms in the three groups (at least 92 percent of classrooms in each group). We found that more classrooms in Groups A and B than in Group C have printed materials and noncommercial handmade materials with text in Quechua. However, only the difference between Groups B and C in printed materials with text in Quechua is statistically significant. Figure IV.4. Available materials with text in the classroom (percentage of classrooms) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 25 We also found that in the second year of program implementation, more classrooms in Group A than in Groups B and C were equipped with a reading area providing reading materials for students, and complete alphabets in Spanish and in Quechua were displayed in more classrooms in Groups A and B than in Group C. (Figure IV.5). The statistically significant differences between Groups A and B in the availability of a reading area, and between Groups B and C in the availability of complete alphabets in Spanish and in Quechua, provide evidence of early impacts of the intervention on the availability of materials to facilitate early reading instruction in the classroom. Figure IV.5. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. 2. Teachers’ approach to teaching reading In the in-school component of the Leer Juntos, Aprender Juntos intervention, teachers in Groups A and B were trained to use the Literacy Boost model strategies to provide instruction on five foundational reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension) in conjunction with any other methods or approaches that they regularly employ to teach reading. We asked teachers to report through the teacher survey the methods they use to teach reading, so we could examine how the training provided by Save the Children may have altered the mix of strategies currently in use. In particular, we asked teachers to report whether they regularly use each of the following three methods8 to teach reading to their students in 2nd grade9 : 8 In Peru, the interviewer administering the teacher survey provided a description of each of the three methods. 9 To develop the list of four methods included in the survey, we conducted a pilot survey in 2013 with a group of teachers in Peru. We used an open-ended question to ask teachers what method or approach they used to teach reading to their students. The final version of the teacher survey includes the four methods that teachers reported most. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 26 • Syllabic: This method consists of teaching letter knowledge and phonological awareness. Teachers using this method focus on teaching letters and simple syllables, emphasizing the correspondence between the written syllables and their sounds, and helping students use those simple syllables to form short words. Once students learn to form short words, the teacher moves on to teach how to form more complex syllables and longer words. • Communicative-textual: The Ministry of Education in Peru is currently promoting this approach to reading instruction. Teachers using the communicative-textual method focus on teaching reading comprehension and supporting the development of communicative competencies in meaningful contexts by presenting images and text to students, explaining their meaning, and helping students communicate that meaning through oral and written language. • Phonological/global: With this method, teachers typically promote a range of foundational reading skills such as letter knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension. This method has four stages: (1) comprehension, in which teachers show images and objects to teach the meaning of words; (2) imitation, in which teachers guide students in copying short sentences and taking dictation, using the words learned in the previous stage; (3) foundation, in which teachers guide students in reading sentences that use the words and structures learned in the two previous stages; and (4) production, in which teachers ask students questions as a way to explain the meaning of the words and sentences that students learned to write and read. Teachers also use songs and short stories to help explain the meaning of the words and sentences. Teachers could also report that they use more than one of these methods, they use a method other than these, or that they do not use a method at all. All of the three major approaches were already common among teachers in the region being studied, but the Leer Juntos, Aprender Juntos intervention increased the percentage of teachers who reported using the phonological-global approach, which most closely corresponds to the program’s emphasis on the five foundational reading skills (Figure IV.6). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 27 Figure IV.6. Teacher-reported approach to teaching reading (percentage of teachers) Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Approaches are not mutually exclusive. * Difference in group means is statistically significant at the 0.05 level. Nearly half of Group C (prevailing practice) teachers reported using syllabic (49 percent) and phonological/global approaches (46 percent), with three-quarters (78 percent) also using communicative-textual methods. For the two treatment groups, these numbers were higher, but only the difference between Groups B and C in the percentage using the phonological/global method (71 in Group B versus 46 percent in Group C) is statistically significant. We found no statistically significant differences between groups in the percentage of teachers who reported that they also use other methods.10 3. Teacher-reported use of time We also asked teachers to report the amount of time they spent during their most recent full￾time work week on activities related to teaching, such as planning and preparing lessons, meeting with their students’ parents, and teaching reading and writing. Contrary to expectations based on the program’s focus on reading instruction, we did not find any statistically significant differences between the groups in the amount of time teachers reported spending on teaching reading and writing and on other activities related to their teaching work (Figure IV.7). On average, teachers in each group reported they spent seven to eight hours during the week teaching reading and writing and planning and preparing lessons (Figure IV.7). Teachers in all groups reported spending two to three hours in their most recent full-time work 10 Other methods reported include the paraphrasing method, the Dolorier method, mapping a story, and the alphabetical method. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 28 week meeting with their students’ parents, providing additional support to low-performing students, and conducting activities to promote reading in the community outside of school hours. Figure IV.7. Teacher-reported number of hours in most recent work week spent in activities related to their job Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: None of the differences between groups presented in this table is statistically significant at the 0.05 level. 4. Observed practice To measure observed teaching practice, the evaluation team observed reading instruction given to 2nd grade students during a typical language or reading class. For about 40 minutes, on average, observers recorded whether teachers spent time conducting reading instruction activities and whether they used instructional practices focused on teaching children five essential literacy skills: (1) letter knowledge, (2) vocabulary, (3) phonemics and phonological awareness, (4) fluency, and (5) reading comprehension. We did not find statistically significant impacts on the amount of time teachers spent on literacy instruction activities (Figure IV.8), but we did find a mix of significant and non￾significant differences in the types of literacy instruction activities that we observed in the classroom (Table IV.3). Two differences that are statistically significant favored the use of literacy practices by teachers in schools participating in the Leer Juntos-school only group (Group B) over those in prevailing practice schools (Group C): prompting students to segment words into units such as syllables or phonemes (25 percent in Group B and 7 percent in Group C) and pre-reading or previewing a book (68 percent in Group B and 47 percent in Group C). Other differences between Groups B and C were not statistically significant, although the overall pattern favored Group B over Group C. The exceptions were that more teachers in Group C than in Group B were observed (1) exploring children’s interest in the story and facilitating connections with their lives (42 percent in Group B and 43 percent in Group C, but this EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 29 difference is not statistically significant), and (2) discussing text structure (7 percent in Group B and 23 percent in Group C). Only the difference in discussing text structure with students is statistically significant. Figure IV.8. Observed teachers’ time using literacy instruction activities (average number of minutes) Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. Note: None of the differences between groups presented in this table is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 30 Table IV.3. Observed teachers’ use of literacy instructional practices in the classroom Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of classrooms in which teachers use the following practices to teach: Vocabulary Introduce or rehearse vocabulary words and explain word meaning or elaborate on new concepts 48.5 34.7 42.4 13.9 (0.204) -7.8 (0.482) Fluency 96.2 93.2 88.9 3.0 (0.572) 4.3 (0.429) Letter knowledge. Encourage students to recognize letters, identify the names of letters, or distinguish upper and lower case 36.6 38.7 21.2 −2.0 (0.834) 17.4 (0.079) Phonemics and phonological awareness Provide instruction on phonological awareness 30.1 33.5 21.5 −3.5 (0.708) 12.0 (0.204) Prompt students to segment words into specific units (syllables or phonemes) 12.2 25.3 6.7 −13.1 (0.086) 18.6* (0.017) Provide instruction on grammar, mechanics, or spelling 32.7 30.5 35.9 2.2 (0.793) -5.5 (0.530) Provide instruction on word writing 65.2 77.6 68.8 −12.3 (0.169) 8.8 (0.329) Book reading and reading comprehension Pre-read or preview book 50.8 67.7 46.8 −16.9 (0.074) 20.8* (0.030) Encourage predictions 36.9 32.8 29.6 4.1 (0.637) 3.2 (0.716) Explore children’s interest in the story and facilitate connections with their lives 37.6 42.3 43.3 -4.7 (0.646) -1.0 (0.920) Use concepts about print 8.3 10.2 16.7 -1.9 (0.782) -6.5 (0.359) Encourage students to use context clues for reading comprehension 24.2 27.1 17.3 −2.9 (0.736) 9.8 (0.259) Discuss text structure 18.5 6.5 22.9 12.0 (0.099) -16.4* (0.026) Discuss the characters in the text, who they are, their motivation and/or goals 45.6 55.3 47.5 −9.7 (0.336) 7.7 (0.448) Guide students in the use of reading comprehension strategies 75.2 81.6 79.3 -6.4 (0.429) 2.2 (0.783) Total number of classroomsa 49 46 48 . . Source: Leer Juntos, Aprender Juntos Classroom Observation—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Classrooms in which we conducted an observation. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 31 5. The literacy environment and reading activities in the home We asked the household survey respondents to identify, for each evaluation child, who reads with the child at home, the types of reading activities with the child, and the reading-related activities that the child does outside of school hours. Most children in the evaluation communities already read at home with a parent or other family members. There was little room for the intervention to make a difference, because most children attending schools in the prevailing practice group already have someone who looks at books with them (95 percent) or reads books to them (74 percent), as shown in Figure IV.9. This finding is consistent with the household survey respondents’ reports about their literacy skills: about three-quarters of survey respondents (76 percent in Groups A and C, and 73 percent in Group B) reported they are able to read (these data are not shown in this report). Figure IV.9. Reading with the evaluation children (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: None of the differences between groups presented in this table is statistically significant at the 0.05 level. The other family members who read with the children in all groups are siblings or other adults living in the household, such as aunts, uncles, and grandparents (Figure IV.10). We did not find any statistically significant differences between groups in the percentage of children who read with their parents or other family members at home, despite the intention of reading buddies programs offered in Group A communities to encourage more siblings and other volunteers to read to children.11 It is possible that we did not find impacts of the community activities on the reading that parents do at home with the children because most parents in all groups are highly involved in reading and looking at books with the children (as shown in Figure 11 As we indicate in Chapter II, the reading buddies activity did not start in Peru until April 2014. Therefore, children in the Phase I communities in Group A had about six months of exposure to the activity to by the time the household survey was administered (as opposed to one year of exposure had the activity started in September 2013). It is possible that this period was not long enough for the effects of the activity to manifest. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 32 IV.9) and are literate (about three-quarters of the household survey respondents in each group). Therefore, there is little room for the intervention to create an effect. Figure IV.10. Other family/household members who read with the child (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: None of the differences between groups presented in this table is statistically significant at the 0.05 level. For about 50 percent of the children in each group, household survey respondents reported reading story books and/or textbooks with the evaluation children (56 percent in Group A, 48 percent in Group B, and 54 percent in Group C). In all groups, on average, household survey respondents read or looked at books with the evaluation child/children living in the household on three days in a typical week. Finally, in all groups, during the most recent day that survey respondents read or looked at books with the children, they spent on average 30 minutes reading or looking at books. There were no statistically significant differences between groups in the time survey respondents spent reading or looking at books with the evaluation children. Parents of the children in the evaluation are interested in finding out how their children do at school. For most of the children in each group (at least 97 percent in each group), the household survey respondents reported that they (or another family/household member) have talked to their child’s teacher about the child’s progress in school at least once during the school year that the household survey was administered. We did not find statistically significant differences between groups on those reports. Parents’ expectations about school completion for their children are similar across groups. For about 50 percent of the children in each group, household survey respondents reported that they would like their children to complete a technical degree or some post-secondary education (51 percent in Group, 50 percent in Group B, and 45 percent in Group C), and for about the other half of the children in each group, household survey respondents reported that they would like their children to complete a college education or higher degree (43 percent in Group A and 48 percent in Groups B and C). Only the difference between Groups A and B in the parents’ expectations about their children graduating from college is statistically significant. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 33 The percentage of children enrolled in 2nd grade or who had repeated 1st grade is similar in all groups, but more children in Group B than in Group C had been absent from school for at least one day during the current school year. There was very little grade repetition among the children in the Peru sample, so again, there was little room for Leer Juntos, Aprender Juntos to make a difference. More than 98 percent of the children in each group were enrolled in 2nd grade, and only 2 to 4 percent of the children in each group had repeated the 1st grade (Table IV.4). This repetition rate is slightly higher than the average 1st grade repetition rate in Peru of 1.7 percent (World Bank 2016).12 First grade repetition rates in Peru are much lower than in other LAC countries—such as Guatemala, the other country in the Leer Juntos, Aprender Juntos evaluation—because in Peru, advancement from 1st to 2nd grade is guaranteed unless special circumstances, such as an explicit request from parents, warrant retaining a child in the 1st grade (Ministerio de Educación 2006). We found only one statistically significant difference between groups in terms of the children’s school attendance: a larger percentage of children in Group B (51 percent) than in Group C (41 percent) had been absent from school for at least one day during the school year in which the household survey was administered. Table IV.4. Children’s school attendance, as reported in the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children attending the following grades in the current school yeara: 1st grade 0.9 1.5 1.6 -0.7 (0.510) -0.0 (0.971) 2nd grade 98.9 98.2 98.4 07 (0.480) -0.2 (0.857) Not enrolled in school 0.2 0.0 0.0 0.2 (0.192) −0.0 (0.976) Percentage of children who have repeated 1st grade 2.9 3.7 2.1 -0.7 (0.558) 1.6 (0.187) Percentage of students who had been absent from school for at least one day in the current school yeara 45.3 50.7 41.1 -5.4 (0.234) 9.6* (0.031) Number of days absent in the current school year (average) 1.4 1.4 1.4 0.0 (0.946) −0.0 (0.981) Number of childrenb 366 338 333 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a The 2014 school year for the Phase I cohort and to the 2015 school year for the Phase II cohort. The percentages for the Leer Juntos-school only group do not sum to 100 percent, because one child in that group was reported as attending the 3rd grade in the current school year. b Number of children for whom a household survey was completed. * Difference in group means is statistically significant at the 0.05 level. 12 This figure is from the World Bank’s World DataBank, in which 2013 is the most recent year that the repetition rate is reported for Peru. Repetition rates in 2014 and 2015 may be higher than 1.7 percent. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 34 Leer Juntos, Aprender Juntos aimed to increase availability of children’s books and the time scheduled for reading in the home, but the effects did not emerge by the time we conducted the household survey. For most children in each group (73 percent in Group A, 68 percent in Group B, and 70 percent in Group C), parents or caregivers schedule a time for reading activities at home (Table IV.5). On average, the scheduled time for reading at home is three hours per week in each group. We found no statistically significant differences between groups in the scheduled time for reading at home. Table IV.5. Availability of books for children and scheduled time for reading at home Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children who have a scheduled time for reading at home 72.7 68.3 70.1 4.4 (0.324) -1.8 (0.699) Hours per week scheduled for reading at home (average) 3.2 3.3 3.0 −0.1 (0.752) 0.3 (0.337) Percentage of households in which books for children are available 99.2 98.6 97.0 0.6 (0.482) 1.5 (0.192) Number of books for children in the household (average) 6.3 5.5 6.5 0.8 (0.093) -1.0 (0.067) Percentage of households with books for children in the following languages: Spanish only 56.0 56.5 49.8 −0.5 (0.916) 6.7 (0.190) Quechua only 0.2 0.4 0.3 -0.2 (0.603) 0.1 (0.887) Spanish and Quechua 43.0 41.8 47.0 1.3 (0.755) −5.2 (0.265) Percentage of children using the books for children in the household with the following frequency: Never 1.4 1.8 3.1 −0.4 (0.673) −1.3 (0.295) One or two times per week 31.0 32.2 21.8 -1.2 (0.776) 10.4* (0.012) Three or four times per week 38.1 40.8 41.8 −2.7 (0.490) -0.9 (0.824) Every day 29.5 25.2 33.3 4.4 (0.313) -8.2 (0.057) Number of householdsa 364 334 329 . . Number of childrenb 366 338 333 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Households that completed the household survey. b Number of children for whom a household survey was completed. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 35 As we show in Table IV.5, some books for children are available most of the households in the evaluation (97 to 99 percent in each group). On average, households in each group have six to seven books for children, and the language of the books is Spanish only in 50 to 57 percent of the households in each group. About 40 percent of the households in each group reported having children’s books written in either Spanish or in Quechua. Households also reported the frequency with which the evaluation children use the children’s books available in their home. In each group, 22 to 32 percent of the children use the children’s books one or two times per week, and 25 to 30 percent of the children in each group use the children’s books every day. We found one statistically significant difference between groups in frequency of use of the books for children available in the home: more children in Group B than C use the children’s books one or two times per week. More children in Group C than in Group B use the children’s books every day, but this difference is not statistically significant.13 Children in the three groups engage in reading activities outside of school hours. At home or in the community, most children in the evaluation (at least 81 percent in each group) participate in reading activities such as reading books, doing homework, and hearing or telling stories. Slightly more children in group A (99 percent) than in group B (97 percent) spend time doing homework, and this small difference is statistically significant (Figure IV.11). In addition, more children in Group A (72 percent) than in Groups B and C (15 and 18 percent, respectively) play reading games with other children in their community. This finding is not surprising, because children in Group A are receiving the community reading activities of the Leer Juntos, Aprender Juntos intervention. Figure IV.11. Reading-related activities at home (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. 13 This difference is not statistically significant at the 0.05 level, but it is at the 0.10 level (p-value = 0.057). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 36 We also asked parents about the time children spent in the week preceding the household survey on the activities included in Figure IV.11 (results not shown in figure). Children in the three groups did homework on three to four days of the week preceding the household survey, and heard or told stories on two days of that week. Children in Group A played reading games with other children in the communities one day of that week, whereas children in groups B and C did not spend time on that activity in that week. Children in the evaluation are not substituting time reading for time on other activities that are not related to reading. As we present in Table IV.6, all children in each group played inside or outside of their homes in the week before the household survey was administered, and did so almost every day of that week. Most children (at least 91 percent in each group) also listened to the radio or watched television and helped with housework chores (such as cleaning, washing, or cooking). We found two statistically significant differences between groups in the activities not related to children reading at home. First, a smaller percentage of children in Group B (92 percent) than in Group C (97 percent) listened to the radio or watched television while at home in the week before the household survey was administered, and children in Group B did that activity on fewer days (five days), on average, than children in group C (six days). Second, more children in Group B (95 percent) than in Group A (91 percent) helped with housework, and children in Group B did that activity on more days (more than four days, on average) than children in Group A (fewer than four days, on average). This second finding suggests that the community component intervention activities that the children in Group A had access to may have led them to replace time on household chores with time spent participating in the reading activities offered by the program in their community. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 37 Table IV.6. Children’s activities at home that are not related to reading Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children who in the week before the administration of the household survey: Played inside or outside the house 99.8 100.0 100.0 −0.2 (0.192) 0.0 (0.983) Listened to the radio or watched television 94.7 91.5 96.9 3.2 (0.109) −5.3* (0.003) Helped with housework 91.2 94.6 94.5 −3.4* (0.049) 0.1 (0.949) Helped with vegetable garden/corn field and/or with livestock 79.3 77.1 77.4 2.1 (0.541) −0.2 (0.953) Cared for younger siblings 53.5 50.1 52.5 3.4 (0.358) -2.4 (0.557) Worked for pay or wage 7.2 7.1 6.5 0.1 (0.961) 0.6 (0.828) Number of days (average) in the week before the administration of the household survey in which children: Played inside or outside the house 6.4 6.5 6.5 −0.1 (0.465) 0.0 (0.926) Listened to the radio or watched television 5.5 5.4 5.8 0.1 (0.577) −0.4* (0.024) Helped with housework 3.6 4.2 4.2 −0.6* (0.002) 0.1 (0.787) Helped with vegetable garden and/or with livestock 3.1 3.2 3.2 -0.1 (0.540) 0.1 (0.775) Cared for younger siblings 2.8 2.8 3.0 0.0 (0.930) -0.2 (0.493) Worked for pay or wage 0.1 0.1 0.1 0.1 (0.065) −0.0 (0.212) Number of childrena 366 338 333 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Number of children for whom a household survey was completed. * Difference in group means is statistically significant at the 0.05 level. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 39 V. SUMMARY OF FINDINGS AND NEXT STEPS Two main findings emerged from this report. First, we found evidence of a meaningful contrast between the treatment groups in the expected direction. Specifically, we found that: • Teachers in the treatment groups (Groups A and B) received the Leer Juntos, Aprender Juntos intervention training for which they were targeted. Most of the teachers in Groups A and B reported that they received reading instruction professional development from Save the Children, and they reported higher rates of participation in reading instruction professional development activities focused on targeted literacy skills than did teachers in the prevailing practice group (Group C). The differences between the treatment groups and the prevailing practice group in participation in reading instruction professional development activities are all statistically significant. • The children and families in Group A received services to promote community reading activities (such as reading camps, story hour, peer assistance, and reading festivals) as intended as part of the Leer Juntos, Aprender Juntos program, and there was minimal prevailing availability of these community reading activities or crossover during the evaluation from children and families in Groups B and C (the groups that were not assigned to receive the community reading activities of the intervention). Second, we found evidence of early impacts of the intervention on teaching practices and the literacy environment in the school and at home: • More teachers in Group B (who received the in-school component of the intervention) than in Group C (the prevailing practice schools in which teachers did not receive any of the components of the Leer Juntos, Aprender Juntos intervention) were observed prompting students to segment words into units such as syllables or phonemes (a practice employed to teach phonemics and phonological awareness) and pre-reading or previewing a book (a practice used to teach book reading and reading comprehension). • More teachers in Group B than in Group C reported using the phonological/global method, which suggests that teachers in the treatment group receiving the in-school component of the intervention, as compared with their counterparts in the prevailing practice group, may be incorporating into their teaching more instruction on a range of foundational reading skills— such as letter knowledge, phonological awareness, decoding, vocabulary, fluency, and comprehension, which are the focus of the phonological/global method. • Printed materials and other materials to facilitate literacy instruction are available in more classrooms in Group B than in Group C. • More children in Group A than in Group B (and C) participate in reading games in the community outside of the school hours, and a smaller percentage of children in Group A than in Group B help with chores such as cooking, washing, and cleaning when they are home, which could translate in more time spent in other activities such as reading. More reports from this evaluation are forthcoming. The current report presents the main findings from the midline data collection efforts in Peru, and a separate report presents the midline findings from Guatemala. For the two countries, we will conduct final follow-up teacher EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 40 surveys, classroom observations, and assessments of students’ reading comprehension skills in the third year of participation in the evaluation (when the students attend 3rd grade). The final follow-up data collection took place in 2015 for the Phase I cohort and in 2016 for the Phase II cohort in Guatemala and Peru. Final reports will present findings from the final follow-up data collection of the Leer Juntos, Aprender Juntos evaluation, including the findings from the estimation of impacts of the Leer Juntos, Aprender Juntos intervention approaches on 3rd grade students’ reading skills outcomes in areas such as fluency, decoding, and comprehension. The final evaluation reports for Guatemala and Peru will be available in 2017. The final evaluation reports (for both Guatemala and Peru) will also present more detailed findings on implementation cost-effectiveness. For each component of the intervention, we will report on the services delivered; whether they were delivered as intended; and if not, why. For the intervention components with documented impact, we will measure the additional cost of implementing each of those components and assess the requirements for implementing them in the future. We will compare the magnitude of any positive effects with the costs of generating those improvements and will express the results in units that allow comparison with other options for investing scarce resources in improving educational outcomes. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 41 REFERENCES Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. “Remedying Education: Evidence from Two Randomized Experiments in India.” Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1235–1264. Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy, vol. 2, 2010, pp. 1–30. Benavides, Martín, Magrith Mena, and Carmen Ponce. “Estado de la Niñez Indígena en el Perú.” Lima, Perú: Fondo de las Naciones Unidas para la Infancia (UNICEF) e Instituto Nacional de Estadística e Informática (INEI), 2010. Bruns, Barbara, and Javier Luque. “Great Teachers: How to Raise Student Learning in Latin America and the Caribbean.” Washington, DC: World Bank, 2014. Comings, John. “Evidence-Based Reading Projects.” Version I. Washington, DC: U.S. Agency for International Development, December 2012. Friedlander, Elliott and Claude Goldenberg, Editors. “Literacy Boost in Rwanda: Impact Evaluation of a 2-year Randomized Control Trial.” Stanford, CA: Stanford University, 2016. Glazerman, Steven, Larissa Campuzano, Nancy Murray, and Camila Fernandez. “Final Leer Juntos, Aprender Juntos Evaluation Plan.” Washington, DC: Mathematica Policy Research, March 22, 2013. King, K. A., and M. Arnal. “Local and Global Dimensions of Language Revitalization.” In Handbook of Indigenous Language Revitalization in the Americas, edited by S. Coronel￾Molina and T. McCarty. London, UK: Taylor & Francis, Ltd., 2016. Lugo-Gil, Julieta, Kate Place, Nancy Murray, Steven Glazerman, Larissa Campuzano, Camila Fernandez. “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Final baseline report.” Washington, DC: U.S. Agency for International Development, 2016. Available at https://pdf.usaid.gov/pdf_docs/PA00MMTF.pdf. Ministerio de Educación. “Diseño Curricular Nacional de Educación Básica Regular: Proceso de Articulación.” Lima, Perú: Dirección General de Educación Básica Regular, 2006. Ministerio de Educación. “Diseño Curricular Nacional de Educación Básica Regular.” Lima, Perú: Dirección General de Educación Básica Regular, 2008. Ministerio de Educación. “Programa Nacional de Redes Educativas Rurales.” Lima, Perú: Ministerio de Educación, 2012. Ministerio de Educación. “Hacia una Educación Intercultural Bilingüe de Calidad. Propuesta Pedagógica.” Lima, Perú: Dirección General de Educación Intercultural Bilingüe y Rural, 2013. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 42 Ministerio de Educación. “Resultados de la Evaluación Censal de Estudiantes 2015 (ECE 2015): Cuarto grado de primaria EIB.” Lima, Perú: Oficina de Medición de la Calidad de los Aprendizajes, Ministerio de Educación, 2016. National Institute of Child Health and Human Development. “Report of the National Reading Panel, Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction.” NIH publication no. 00-4769, Washington, DC: U.S. Government Printing Office, 2000. Nieto, Ana Maria. “Modifications of Literacy Boost’s general parent workshop design and parents’ appraisals of the workshops in four communities in Guatemala: Implications for the revision of Literacy Boost’s Parent Workshops.” Cambridge, MA: Harvard Graduate School of Education, November 13, 2015. Save the Children. “Literacy Boost Toolkit: Teacher Training.” Washington, DC: Save the Children, 2012a. Save the Children. “Literacy Boost Toolkit: Community Action.” Washington, DC: Save the Children, 2012b. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Report, September 2012–December 2012 (Quarter 1).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2012. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2013– December 2013 (Quarters 2–5).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2013. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2014– December 2014 (Quarters 6–9).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2014. Save the Children and Kallpa. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2015– December 2015 (Quarters 10–13).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2015. Trapnell, Lucy A. “Some Key Issues in Intercultural Bilingual Education Teacher Training Programmes—As Seen from a Teacher Training Programme in the Peruvian Amazon Basin.” Comparative Education, vol. 39, no. 2, special no. 27, May 2003, pp. 165–183. UNESCO. “Education for All 2000–2015: Achievements and Challenges.” EFA Global Monitoring Report, Paris, France: United Nations Educational, Scientific and Cultural Organization, 2015. UNESCO. “If You Don’t Understand, How Can You Learn?” Global Education Monitoring Report, Policy Paper 24, Paris, France: United Nations Educational, Scientific and Cultural Organization, 2016. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH 43 USAID. “Early Grade Reading and Teacher Training in the LAC Region: Influencing Factors, Lessons and Approaches.” Washington, DC: USAID, June 2011. Vargas Dávila, José Luis. “El Programa Presupuestal Estratégico Logros de Aprendizaje: Avances, Tensiones, Perspectivas.” Lima, Perú: Tarea Asociación de Publicaciones Educativas, 2009. World Bank. “Education Statistics-All Indicators. Peru.” Washington, DC: World Bank. Available at http://databank.worldbank.org/data/views/variableselection/selectvariables.aspx ?source=education-statistics-~-all-indicators#. Accessed November 16, 2016. This page has been left blank for double-sided copying. APPENDIX: BACKGROUND CHARACTERISTICS OF EVALUATION SCHOOLS, TEACHERS, STUDENTS, AND HOUSEHOLDS IN PERU This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.3 In this appendix, we briefly describe characteristics of the schools, teachers, and students that we expect to be constant over time or unlikely to be affected by treatment, as a check on randomization14 and because some of those characteristics may affect the impacts of the intervention and were not measured at baseline. The baseline report (Lugo-Gil et al. 2016) did not show noteworthy differences in the background characteristics of schools, teachers, and students in the evaluation, and the following analysis reinforces that general finding. 1. Schools Schools in the three groups are similar in location, number of classrooms in use, class size, and participation in other education and social programs. Schools in each group are similarly distributed across three local education administrations (UGELs) in the Apurímac department: Andahuaylas, Huancarama, and Chincheros. The predominant language in the Apurímac department is Quechua. On average, in each group, schools use six elementary school classrooms and have 9 to 10 students enrolled in 2nd grade. We found no statistically significant differences between groups. The utilities and facilities in the evaluation schools are basic, but most of the schools have a piped water supply (90 percent in Group A, 85 percent in Group B, and 87 in Group C), electricity (98 percent in all groups), and working restrooms (94 percent in Group A, 93 percent in Group B, and 92 in Group C) and handwashing facilities (84 percent in Group A, 82 percent in Group B, and 83 percent in Group C). In the second year of program implementation (midline), we obtained the same results for school infrastructure characteristics that we did in the baseline year (Lugo-Gil et al. 2016). For the most part, the groups are similar in terms of the utilities and facilities available in the schools. However, as we found in the baseline, the evaluation midline indicated only a few statistically significant differences between groups in the available utilities and facilities in the school. For example, a higher percentage of schools in Group B (95 percent) than in Group A (80 percent) have computers for students’ use. In addition, more schools in Group C (25 percent) than in Group B (9 percent) have a library or room with reading materials. We will control for these differences between groups in our analysis of the final impacts of the program (available in 2017). Schools also reported on their participation in regional and national education programs taking place at the school, which reinforced the notion that Leer Juntos, Aprender Juntos was not operating in a vacuum. About 70 percent of the schools in each group (72 percent in Group A, 71 percent in Group B, and 70 percent in Group C) reported that they participate in the Ministry of Education’s Una Laptop por Niño (One Laptop per Child) program. Schools in the evaluation also participated in other programs, such as the national Strategic Learning Outcomes Program, or PELA, as it is known in Spanish (24 percent in Group A, 22 percent in Group B, and 33 percent in Group C) and the National Rural Education Networks Program (REDES; 18 percent in Groups A and C, and 21 percent in Group B). All schools in the three groups reported 14 If random assignment produced equivalent groups, then we would generally expect the schools in the three groups to be similar in observable characteristics. Note that balance or lack of balance on observables does not prove that random assignment worked or did not work. It can provide face validity to the notion that randomization produced balance on unobservable characteristics. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.4 participating in the national children’s nutrition program (known as Qali Warma) and received educational materials from the Ministry of Education. About one-third of the schools in each group (37 percent in Group A, 36 percent in Group B, and 29 percent in Group C) reported receiving pedagogical support from the Ministry of Education. These results are similar to what we found in the first year of the evaluation (Lugo-Gil et al. 2016): about two-thirds of the schools in each group reported participating in the One Laptop per Child program and most schools in each group (at least 83 percent in each group) reported participating in the national children’s nutrition program and receiving educational materials from the Ministry of Education, and we did not find any statistically significant differences between groups in the participation of schools in regional and national programs focused on education, health, or nutrition. 2. Teachers We identified one teacher of 2nd grade students from every evaluation school (selecting randomly if there were more than one) and interviewed the teachers about their personal and professional backgrounds. We did not find statistically significant differences between groups in terms of teachers’ background characteristics, such as gender, age, ethnicity, education level, years of teaching experience, and having a long-term contract (Table A.1). Because teachers new to the school in the current school year (2014 and 2015 for teachers in the Phase I and II cohorts, respectively) have not received as much intervention training as teachers who have been teaching at the current school for at least one year, we explored whether the groups differed in the percentage of first-year teachers in 2nd grade (teachers in current school for less than a year). We did not find statistically significant differences between groups in the percentage of first-year teachers in 2nd grade (Table A.1). Therefore, the amount of intervention training that 2nd grade teachers have received to date should not be a factor in whether teachers’ instructional practices differ across groups. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.5 Table A.1. Teacher background characteristics at midline Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Female (percentage) 57.9 61.6 63.7 −3.6 (0.714) −2.1 (0.831) Age (average) 44.1 42.6 42.7 1.4 (0.402) −0.1 (0.968) Mother tongue (percentage) Spanish only 28.6 21.6 27.1 7.0 (0.452) −5.5 (0.558) Quechua only 30.3 28.2 29.3 2.1 (0.836) −1.1 (0.915) Both Spanish and Quechua 41.1 50.2 43.6 −9.1 (0.364) 6.6 (0.515) Highest level of education (percentage) Incomplete degree (associate, technical, or college) 0.0 4.2 6.3 −4.2 (0.286) −2.1 (0.589) Associate or technical degree 63.4 57.1 56.1 6.3 (0.516) 1.0 (0.917) College degree 32.4 30.3 31.4 2.1 (0.813) −1.1 (0.903) Master’s or doctorate degree 4.2 8.4 6.2 −4.3 (0.419) 2.2 (0.677) Years of teaching experience (average) 18.6 16.7 18.0 2.0 (0.297) −1.3 (0.501) Years teaching at current school (average) 8.2 7.5 8.4 0.7 (0.651) −0.9 (0.581) Years teaching 2nd grade (average) 4.9 6.3 5.3 −1.4 (0.112) 1.0 (0.265) Percentage of first-year teachers in 2nd grade (in current school for less than a year) 24.8 20.5 22.8 4.3 (0.625) −2.3 (0.801) Long-term contract (percentage) 70.7 67.9 72.3 2.7 (0.785) −4.4 (0.667) Number of teachersa 49 46 48 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. None of the differences presented in this table is statistically significant at the 0.05 level. a Teachers who completed a survey. 3. Households As we describe in Chapter III, we administered a household survey to obtain data on household composition, socioeconomic status, children’s literacy practices and routines at home, and children and families’ participation in reading-related activities in the community. The household survey was completed in 1,027 households for 1,037 children in the evaluation, mostly by mothers of evaluation children (Table A.2). Ten of the 1,027 households had two children in the evaluation sample that were siblings or relatives living in the same EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.6 household. In the households with more than one child in the evaluation, all of the children are, by the evaluation’s design, part of the same group. The response rate for the household survey (at the student level) is 97 percent. As expected, households across the three groups are similar in many ways. We did not find noteworthy differences in terms of household composition, parents’ educational background and current work status, and availability of assets, utilities, and commodities. This result is expected because the Leer Juntos, Aprender Juntos intervention is not designed to affect households’ composition and socioeconomic characteristics and because random assignment was used to produce equivalent groups. However, it is important to understand the households of the children in the evaluation, because some of those characteristics may affect the impacts of the intervention and were not measured at baseline. When we examine the impacts of Leer Juntos, Aprender Juntos in the final year of the project (2017), we will take into account any chance differences between groups on households’ background characteristics. Table A.2. Respondents to the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of households in which survey respondent was the evaluation child’s: Mother 86.0 88.3 87.1 −2.3 (0.447) 1.2 (0.682) Father 10.3 11.1 10.1 −0.8 (0.777) 0.9 (0.727) Other caregiver 3.7 0.6 2.8 3.1* (0.010) −2.2 (0.064) Number of householdsa 364 334 329 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Households that completed the household survey. * Difference in group means is statistically significant at the 0.05 level. Most of the households in each group are two-parent households. There are no statistically significant differences between groups in the percentage of two-parent households (79 percent in Group A, 84 percent in Group B, and 83 percent in group C). As Table A.3 shows, in the three groups, there are on average five to six people living in the evaluation households, three of whom are children. We found one statistically significant difference between the groups: there were more people living in the households in Group B (6 people, on average) than in the households in Group C (5 people, on average). We found only two statistically significant differences in the educational background and work status of adults in the household: a larger percentage of the mothers in Group C than in Group B (7 and 3 percent, respectively) have completed some (or all) of a post-secondary degree (technical or college), and a larger percentage of mothers in Groups A and C (27 and 28 percent, respectively) than in Group B (18 percent) have a paying job (Table EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.7 A.3). Households in the three groups have similar incomes. Households reported a monthly income between 526 and 603 soles in each group, or US$163 to US$187, on average. Average household income is higher in Group C than in Group B, and in Group A than in Group B, but these differences between groups are not statistically significant (Table A.3). Table A.3. Households’ socioeconomic status Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Number of household members (average) 5.4 5.5 5.2 −0.1 (0.439) 0.3* (0.040) Number of children in the household (average) 3.1 3.2 2.9 −0.1 (0.625) 0.3* (0.019) Number of rooms in the household (average) 3.0 3.1 3.1 −0.0 (0.815) −0.1 (0.536) Percentage of households in which the following household members have never attended school or attended only preschool: Mother 18.1 15.8 14.2 2.4 (0.380) 1.5 (0.601) Father 2.6 1.5 2.2 1.1 (0.383) −0.7 (0.615) Other adults who spend time with evaluation child 12.1 8.0 1.2 4.0 (0.367) 6.9 (0.139) Percentage of households in which mothers completed some (or all) of a post-secondary degree (technical or college) 2.5 3.2 6.9 −0.7 (0.559) −3.7* (0.015) Percentage of households in which the following household members have a paying job: Mother 27.2 17.8 27.7 9.4* (0.013) −10.1* (0.012) Father 99.9 99.3 98.8 0.6 (0.215) 0.5 (0.284) Other adults who spend time with evaluation child 43.5 52.0 48.2 −8.5 (0.303) 3.8 (0.622) Household monthly income in U.S. dollars 174.5 163.1 187.0 11.4 (0.610) −23.9 (0.376) Number of householdsa 364 334 329 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Households that completed the household survey. * Difference in group means is statistically significant at the 0.05 level. A potentially important determinant of education outcomes is the wealth of the family. As a proxy for wealth, we tried to measure household assets, such as the quality of the home. Through the household survey, we inquired about the materials used to build the houses where the children of the evaluation live. We found that in all groups, around 90 percent of the families/households (91 percent in Group A, 92 percent in Group B, and 89 percent in Group C) live in a house that has a floor made of raw earth, dirt, or sand. In addition, the roofs of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.8 houses of most families/households are built with basic materials such as hay, palm leaves, cardboard, wood scraps, and fiberglass (in contrast to basic materials, other materials such as concrete, terra cotta, and tile are considered finished/refined materials). We found two statistically significant differences between groups in the construction materials of the houses of the families in the evaluation: more families in Group B (25 percent) than C (16 percent) live in a house with a roof built with finished/refined materials, and more families in Groups A and C (82 and 83 percent, respectively) than in Group B (73 percent) live in a house with a roof built with basic materials. Households in the three groups are also similar in terms of their assets and the available utilities and commodities (Figure A.1). Most families/households own land (89 percent in Groups A and B and 92 percent in Group C) and have electricity (92 percent in Group A and 89 percent in Groups C and B), and at least 75 percent of the households in each group have phone service and a television (Figure A.1). We found only one statistically significant difference between groups in the utilities and commodities available to the households: on average, more households in Group C than B (9 and 3 percent, respectively) own a computer (Figure A.1). Finally, almost all the children (students) in the evaluation sample walk to school (94 percent in Groups A and C and 95 percent in Group B), and the average length of the commute to school is 16 to 19 minutes in each group. There are no statistically significant differences between groups in the type or length of the commute to school. Figure A.1. Household assets, utilities, and commodities (percentage of households) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN PERU: MIDLINE REPORT MATHEMATICA POLICY RESEARCH A.9 4. Students We asked caregivers of the children in the evaluation about the children’s preschool attendance. As Table A.4 presents, at the time of administration of the household survey, almost all of the children in each group had attended preschool before enrolling in 1st grade and had been enrolled in preschool for about three years. We found a statistically significant difference between groups: children in Groups A and C had attended preschool for about 10 weeks longer than children in Group B. Table A.4. Children’s preschool attendance, as reported in the household survey Blank cell Leer Juntos (A) Leer Juntos￾school only (B) Prevailing practice (C) A–B B–C Percentage of children who attended preschool before starting 1st grade 98.8 98.9 99.5 −0.1 (0.938) −0.6 (0.540) Number of years children attended preschool (average) 2.9 2.7 2.9 0.2* (0.013) −0.2* (0.002) Number of childrena 366 338 333 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Column A–B presents differences in the regression-adjusted group means between the Leer Juntos group and the Leer Juntos-school only group, and Column B–C presents differences in the regression-adjusted group means between the Leer Juntos-school only group and the prevailing practice group. The p-values from tests of differences between group means are presented in parentheses. a Number of children for whom a household survey was completed. * Difference in group means is statistically significant at the 0.05 level. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-000 Fax: (202) 216-3524 www.usaid.gov