Evaluation of Leer Juntos, Aprender Juntos Early-Grade Reading Intervention in Guatemala Final Report October 2019 This report was produced for review by the United States Agency for International Development under task order AID-OAA-M-12-00020. It was prepared by Julieta Lugo-Gil, Nancy Murray, Camila Fernandez, Steven Glazerman, and Larissa Campuzano of Mathematica. Photo Credit: Save the Children. Evaluation of Leer Juntos, Aprender Juntos Early-Grade Reading Intervention in Guatemala Final Report Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH i ABSTRACT This report estimated the impact of the two main components of Leer Juntos, Aprender Juntos—an early-grade reading instruction program for linguistically diverse communities. Save the Children developed the program based on its Literacy Boost model and implemented it in the K’iche’-speaking region of Guatemala. The program’s in-school component aims to train and coach teachers to improve reading instruction in the early grades. The community action component aims to strengthen parental and community involvement and increase children’s opportunities to practice reading outside school. We randomly assigned schools to one of three evaluation groups: Group A schools implemented both program components; Group B schools implemented only the in-school component; and Group C schools did not implement the program. To assess the impact of the community action component, we compared Group A with Group B. To assess the impact of the in-school component, we compared Group B with Group C. Within each school, we followed a group of children from 1st grade to 3rd grade. We found no impacts of either component of the Leer Juntos, Aprender Juntos program on children’s reading skills in Guatemala. This finding does not mean that the children in these schools did not learn. Rather, it shows that, on average, children in the schools in each evaluation group made similar progress with their reading skills. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH iii ACKNOWLEDGMENTS The authors gratefully acknowledge numerous people and organizations that contributed to the implementation of all evaluation activities in Guatemala and to the production of this report. Luisa Fernanda Müller Duran and Maria José del Valle from the Evaluation Division of the Guatemala Ministry of Education, Juárez and Associates’ Regina da Costa, Justo Magzul, and Fernando E. Rubio, and Cristina Perdomo from the U.S. Agency for International Development (USAID)/Reforma Educativa en el Aula, all shared invaluable insights and measurement tools for assessing emerging literacy and early-grade reading in Spanish and K’iche’ in Guatemala. The State Department of Education in El Quiché (DIDEDUC El Quiché) provided a warm welcome and essential departmental support for the evaluation. Hember Herrera, departmental director of education, and Marcelino Cabrera, assistant director of intercultural bilingual education, offered continuous assistance during our data collection efforts in schools and communities in the Department of El Quiché. The staff from Save the Children provided us with key information on program implementation and valuable support in coordinating data collection activities. They include Katy Anis, chief of party for Leer Juntos, Aprender Juntos, and from Save the Children/Guatemala, Lilia Cifuentes, education project manager, and Ana Quixtán, monitoring and evaluation coordinator. Save the Children’s technical and administrative project team in El Quiché also provided valuable information and assistance in the coordination of the evaluation activities. USAID/Washington and USAID/Guatemala staff guided and supported the evaluation design and its implementation. They include, from USAID/Washington, Barbara Knox-Seith, senior education advisor, and Michael Lisman, education advisor, Bureau for Latin America and the Caribbean; from USAID/Guatemala, Laura Villegas Lalinde, project management specialist for education, Health and Education Office; and while at USAID/Guatemala, Juan Luis Córdova, education officer, Health and Education Office. We thank our local data collection partner, DMC Consultores, and its staff for high quality data collection in the evaluation schools and communities. Marco Oliva provided overall project management support in Guatemala, oversaw data entry, and ensured consistency of data programming. Otto Rego contributed thoughtful technical support, skilled training for our field staff, and generous collaboration in building a local team in the Department of El Quiché. Guillermo Duarte ensured careful planning of evaluation activities in Guatemala, provided keen oversight of all data collection activities, and developed a collaborative relationship with staff at the State Department of Education in the Department of El Quiché. We are also indebted to Sylvia Linan-Thompson for her thorough and incisive input at different stages of the evaluation. Her expertise in models of reading interventions in bilingual classrooms and ample experience with measurement of early-grade reading skills were essential in designing the evaluation and interpreting baseline findings. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH iv At Mathematica Policy Research, Galina Lapadatova, Seth Morgan, Ivonne Padilla, Sarah Chan, Maura Butler, and Colleen Fitts provided critical support in data cleaning, programming, editing, and presentation. We are enormously appreciative of all the school directors and teachers who opened their classrooms and allowed us to observe the valuable work they do with students. Last, but not least, we thank the students and parents who provided us with the important information we are able to share in this report. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH v CONTENTS ABSTRACT ................................................................................................................................................. i ACKNOWLEDGMENTS ...............................................................................................................................iii ACRONYMS ................................................................................................................................................xi EXECUTIVE SUMMARY ........................................................................................................................ ES-1 A. Introduction ............................................................................................................................. ES-1 B. Evaluation questions and design............................................................................................ ES-2 C. Summary of findings ............................................................................................................... ES-3 D. Conclusions, limitations, lessons learned, and recommendations......................................... ES-4 I. INTRODUCTION .................................................................................................................................. 1 A. Early grade literacy education in linguistically diverse contexts ................................................... 1 B. The intervention and evaluation ................................................................................................... 1 C. Overview of Leer Juntos, Aprender Juntos evaluation................................................................. 8 II. EVALUATION DESIGN AND DATA..................................................................................................... 9 A. Evaluation questions .................................................................................................................... 9 B. Evaluation design ......................................................................................................................... 9 C. Leer Juntos, Aprender Juntos logic framework .......................................................................... 11 D. Types of data collected for the evaluation.................................................................................. 12 E. Analytic approach ....................................................................................................................... 15 F. Characteristics of the sample at endline .................................................................................... 17 G. Baseline equivalence of the analysis sample............................................................................. 22 III. WAS LEER JUNTOS, APRENDER JUNTOS IMPLEMENTED AS INTENDED?.............................. 25 A. School participation in education interventions .......................................................................... 25 B. Implementation challenges in Guatemala .................................................................................. 26 C. Implementation of the in-school component............................................................................... 27 D. Implementation of the community action component ................................................................. 31 IV. WHAT ARE THE IMPACTS ON INSTRUCTIONAL PRACTICES AND LITERACY ENVIRONMENT? ............................................................................................................................... 35 A. Classroom instruction ................................................................................................................. 35 B. Reading environment and behaviors.......................................................................................... 41 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH vi V. WHAT WERE THE IMPACTS ON CHILDREN’S EARLY-GRADE READING SKILLS? ................... 49 A. How we measured early-grade reading skills............................................................................. 49 B. Impacts on reading skills ............................................................................................................ 50 C. Impacts by gender ...................................................................................................................... 55 VI. CONCLUSIONS, LIMITATIONS, LESSONS LEARNED, AND RECOMMENDATIONS ................... 59 A. Conclusions ................................................................................................................................ 59 B. Limitations................................................................................................................................... 62 C. Lessons learned ......................................................................................................................... 63 D. Recommendations...................................................................................................................... 64 REFERENCES............................................................................................................................................ 67 APPENDIX A: TIMELINE OF EVALUATION ACTIVITIES IN GUATEMALA, 2013–2016 .......................A.1 APPENDIX B: QUALITATIVE DATA COLLECTION ................................................................................B.1 APPENDIX C: CONTROL VARIABLES IN CHILD-LEVEL REGRESSION ANALYSES .........................C.1 APPENDIX D: SCHOOL INFRASTRUCTURE CHARACTERISTICS......................................................D.1 APPENDIX E: CLASSROOM AND HOUSEHOLD ENVIRONMENTS.....................................................E.1 APPENDIX F: STANDARD DEVIATIONS OF CHILDREN’S READING SKILLS MEASURES...............F.1 APPENDIX G: PROGRAM IMPACTS IN EFFECT SIZES ...................................................................... G.1 APPENDIX H: SENSITIVITY ANALYSES................................................................................................H.1 APPENDIX I: IMPACTS OF THE PROGRAM COMPONENTS FOR BOYS AND GIRLS IN TERMS OF EFFECT SIZES............................................................................................................... I.1 APPENDIX J: DATA COLLECTION INSTRUMENTS ............................................................................. J.1 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH vii TABLES II.1. Response rates to data collections in the evaluation 14 II.2. Characteristics of evaluation schools in the final follow-up year 18 II.3. Background characteristics of the teachers of the evaluation students in the final follow￾up year 19 II.4. Experience and tenure characteristics of the teachers of the evaluation students in the final follow-up year 20 II.5. Teacher-reported language skills (percentages of teachers) 20 II.6. Grade and school at the time of the final follow-up data collection (percentage of students) 22 II.7. Baseline characteristics of children in the evaluation sample 22 III.1. School participation in education and social programs, reported by principals or other school administrators 26 III.2. Parents’ reports on children’s exposure to and participation in reading activities in the community 33 IV.1. Teacher-reported methods and strategies to teach reading 38 IV.2. Observed instructional practices to teach literacy skills 40 IV.3. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) 42 IV.4. Characteristics of the books observed in the classrooms (percentage of classrooms) 43 IV.5. Availability of books for children at home 44 IV.6. Dedicated time for reading at home 46 IV.7. Children’s activities at home that are not related to reading 48 V.1. Impacts on the literacy outcomes of girls and boys in the evaluation 56 B.1. Respondents, methods, topics of inquiry, and locations for the qualitative data collection B.1 C.1. Control variables included in the analyses of the reading skills of 3rd graders in the evaluation C.1 D.1. Available utilities in evaluation schools (percentages of schools) D.1 D.2. Other facilities available in schools D.2 D.3. Physical and health hazards observed in schools D.3 E.1. Members, services, and municipalities in the Educational Network in the Department of El Quiché E.2 E.2. Teacher-reported participation in reading instruction professional development (PD) activities in the endline year (percentages of teachers) E.4 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH viii E.3. Teacher-reported participation in reading instruction professional development (PD) activities in the year before endline survey administration (percentages of teachers) E.5 E.4. Teacher-reported participation in reading instruction professional development (PD) activities focused on specific topics in the endline year E.7 E.5. Students’ use of different types of text, as reported by teachers E.8 E.6. Number of minutes students spend on the following activities during a typical language and reading class, as reported by teachers E.9 E.7. Teacher-reported needs and challenges (percentages of teachers) E.10 E.8. Observed general instructional practices E.11 E.9. Observed teachers’ use of language in the classroom E.12 E.10. Observed students’ use of language in the classroom E.12 E.11. Books and materials observed in the classroom E.13 F.1. Standard deviations of children’s reading skills outcome measures F.1 G.1. Impacts on literacy outcomes in standard deviation units G.1 H.1. Differences between treatment groups in children’s literacy outcomes, accounting for nonresponse to the 3rd grade assessment H.1 H.2. Differences between treatment groups in children’s literacy outcomes, based on sample with complete data H.2 H.3. Impacts on the literacy outcomes of children in Phase I and Phase II schools H.3 H.4. Impacts of the full intervention H.4 I.1. Community component effects on literacy outcomes for girls and boys, in standard deviation units I.1 I.2. In-school component effects on literacy outcomes for girls and boys, in standard deviation units I.2 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ix FIGURES II.1. Leer Juntos, Aprender Juntos evaluation design 11 II.2. Leer Juntos, Aprender Juntos logic framework 12 II.3. The Department of El Quiché in Guatemala 17 IV.1. Teacher-reported participation in reading instruction training and coaching in the endline year 36 IV.2. Observed allocation of instruction time in a typical language or reading class 39 IV.3. Percentages of evaluation children whose parents or other household members read with them 45 IV.4. Reading-related activities at home (percentage of children) 47 V.1. Similar number of pseudo-words read correctly in the decoding task (of 50) for each treatment group 51 V.2. Similar number of words read correctly in the fluency task (of 112) for each treatment group 52 V.3. No positive program impacts on decoding and fluency accuracy scores 53 V.4. Similar number of questions answered correctly in the reading comprehension task (of 15) for each treatment group 54 V.5. Similar percentage of children who achieved a basic level of reading comprehension skills for each treatment group 54 A.1. Timeline of Leer Juntos, Aprender Juntos activities, 2013–2016 A.4 E.1. Rates of participation in professional development (PD) activities focused on specific literacy skills E.6 This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH xi ACRONYMS CTA Coordinadores Técnicos Administrativos DDE Departmental Directorate of Education DIDEDUC El Quiché Department of Education in El Quiché DIGEBI Dirección General de Educación Bilingüe Intercultural DIGEDUCA Ministry of Education in Guatemala DMC DMC Consultores EDUVIDA Programa Educación para la Vida y el Trabajo EGRA Early Grade Reading Assessment FUNSEPA Fundación Sergio Paiz Andrade in Guatemala KOIKA Korea International Cooperation Agency LAC Latin America and the Caribbean LAC Reads Latin America and the Caribbean Reads NEUBI Nueva Escuela Unitaria Bilingüe NGO non-governmental organization NEUBI Nueva Escuela Unitaria Bilingüe PADEP Programa Académico de Desarrollo Profesional PD Professional development activities PROFFE Proceso de Formación de Funcionarios de la Educación PRODESSA Proyecto de Desarrollo Santiago USAID U.S. Agency for International Development This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-1 EXECUTIVE SUMMARY A. Introduction This report presents results of the impact evaluation of Leer Juntos, Aprender Juntos, a program funded by the U.S. Agency for International Development (USAID) to improve early￾grade reading instruction in communities with linguistically diverse populations. Save the Children developed the program based on its Literacy Boost model, which includes teacher training and community involvement, and implemented it in the K’iche’-speaking region of Guatemala and the Quechua-speaking region of Apurímac in Peru. Mathematica designed a rigorous evaluation of Leer Juntos, Aprender Juntos in both Guatemala and Peru. This report focuses on the impact evaluation findings for Guatemala. A separate report presents the impact evaluation findings for Peru (Lugo-Gil et al. 2021a). The Leer Juntos, Aprender Juntos intervention has two main components: (1) the in-school component and (2) the community action component. In-school component: teacher training and coaching. The in-school component of Leer Juntos, Aprender Juntos was based on the Literacy Boost model. The goal of this component was to train and coach teachers to be better equipped for mother tongue and/or Spanish reading instruction in the early primary grades. This component entailed six main activities intended to increase class time on reading instruction and improve the quality of reading instruction: 1. Training trainers in reading instruction techniques 2. Training teachers in the five core skills of reading 3. Creating materials for print-rich classrooms 4. Mentoring and coaching teachers in reading instruction practices 5. Training teachers in conducting formative assessments to track progress of children’s reading skills 6. Guiding teachers to incorporate five core skills of reading and related reading instruction techniques into daily school activities Teacher training consisted of nine modules on a range of reading-instruction topics, including introduction to reading acquisition and instruction in the early primary grades, use of formative reading assessments, and five core reading skills. Following the teacher training, the program’s technical staff (coaches) visited classrooms at least once (in most cases, two or three times) every three months. These visits aimed to support teachers by observing their work, demonstrating teaching techniques, and suggesting improvements in the use of reading instruction strategies. Coaches also moderated teacher learning groups to strengthen teachers’ practical application of topics discussed in the training workshops, which included conducting activities to develop the five core reading skills, designing lesson plans incorporating core reading skills, and planning strategies to develop literacy skills in students’ mother tongue. The training and coaches did not dictate or favor one language of instruction over the other. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-2 Community action component. The community action component of the program, also based on the Literacy Boost model, aimed to strengthen parental and community involvement in building children’s reading abilities and increasing their opportunities to practice reading outside regular class time. This component was intended to be delivered primarily by trained community volunteers and included engaging group activities. The community action component included the following main activities: ● Creating printed materials in the children’s mother tongue, Spanish, or both, to build portable libraries known as book banks ● Promoting the use of book banks among community members ● Conducting reading activities in the community such as story hours, reading camps, and reading festivals or fairs ● Coordinating peer assistance through reading buddies ● Conducting school–community accountability meetings and reading awareness workshops with parents and community members B. Evaluation questions and design The evaluation was implemented as a randomized controlled trial. The evaluation team randomly assigned schools to one of three evaluation groups: Group A schools implemented both components of Leer Juntos, Aprender Juntos, Group B schools implemented only the in￾school component of the program, and Group C schools did not implement the program. (We refer to Group C schools as prevailing practice.) The evaluation aimed to answer the following two primary evaluation questions: • What is the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to prevailing practice? This question contrasts schools in Group B to those in Group C. • What is the impact of the community action component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to an intervention that does not have the community action component? This question contrasts schools in Group A to those in Group B. In addition to these two primary questions, the evaluation addresses a question about implementation: • Were the program components implemented as intended? This question relates to whether each program component’s services were offered as originally intended, whether participants took part, and whether these program components had intermediate impacts on teaching and the availability and use of reading materials. It also considers any barriers or challenges to effective implementation. In Guatemala, we recruited 150 schools. Within each school, we followed a group of children from 1st grade through the end of 3rd grade. The implementation of Leer Juntos, Aprender Juntos in Guatemala began in May 2013 and continued through March 2016. Because EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-3 Save the Children planned to roll out the program in phases, we also carried out the evaluation in two phases. We randomly assigned eligible schools and collected baseline data in the first half of the evaluation schools in 2013 (Phase I), and then in the other half of schools in 2014 (Phase II). We conducted two follow-up data collections after the baseline: one at the end of the second year of implementation (midline, in 2014 for Phase I and 2015 for Phase II) and another at the end of the third year of implementation (endline, in 2015 for Phase I and 2016 for Phase II). Implementation of the program began in May 2013 for Phase I schools and January 2014 for Phase II schools. We present results for students pooled across the two phases because we did not find any significant pattern of differences between phases. As part of the evaluation, we also conducted an implementation evaluation. This involved data from several sources, described in Chapter II, including school observations, classroom observations, teacher surveys, focus groups, and interviews. The observations took place at baseline (first year of the evaluation) and midline (second year). Teacher surveys took place at baseline, midline, and endline, as described in Chapter II. C. Summary of findings Leer Juntos, Aprender Juntos was implemented with fidelity, but we did not find evidence of impacts on reading outcomes from either the in-school or the community component of the program. We did not find any meaningful pattern of statistically significant differences in children’s reading skills between the group that implemented both components of the program and the group that implemented only the in-school component of the program. Nor did we find any meaningful pattern of such differences between the group that implemented the in-school component compared to the group that implemented prevailing practices. While the treatment groups were similar across several dimensions of reading environment within classrooms, schools receiving only the in-school component had more materials to support the teaching of reading. We did find evidence of some favorable effects on the reading environment within classrooms, in particular with respect to the materials for reading instruction in K’iche’ and Spanish that were available in the classrooms. However, those effects did not translate into impacts of any of the program’s components on children’s reading skills in the 3rd grade. The lack of impacts on reading skills by the end of third grade may be because the program had not fully matured or been in place long enough. Program staff reported in interviews that the first year of implementation was a learning year for them, requiring adjustments to the Guatemalan context. Similarly, teachers need time to learn and be able to apply new instructional practices, so it is possible that the program’s implementation period was not enough time for them to fully apply what they learned in the program’s training. Program staff and teachers also reported that most teachers in the early grades did not have adequate K’iche’ language skills to fully incorporate the teaching strategies in mother tongue that they learned through the program’s training. Another possible explanation is that the amount of training the program provided to teachers was not enough to generate impacts on students’ reading skills. In the last year of the study, only about half of the 3rd grade teachers surveyed in intervention schools reported having received training from the program that year (50 percent in Group A and 48 percent in Group B). In addition, the implementation team faced challenges with EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-4 the length of the nine training modules, and the mandate from El Quiche’s Departmental Directorate of Education that teacher trainings should not interfere with the required 180 days of class per year. D. Conclusions, limitations, lessons learned, and recommendations 1. Conclusions Leer Juntos, Aprender Juntos was implemented mostly as intended, although there were some challenges. Teachers in the schools assigned to implement the in-school component did receive training and individualized coaching sessions. Beginning teachers found that the Leer Juntos, Aprender Juntos teaching strategies and pedagogic activities were particularly helpful, and teachers in general reported receiving innovative project materials for use in the classroom in Spanish and mother tongue (K’iche’). However, take-up of reading instruction strategies and activities was particularly challenging for teachers who lacked mother tongue skills, and for those who taught in multi-grade classrooms. Additionally, some teachers felt they did not receive as much feedback and support in the coaching sessions as they were expecting, and there was high teacher turnover and rotation of teachers to upper grades. The activities that were part of the community action component were delivered mostly as intended in the schools that were assigned to receive that component. These activities included access to book banks, story hour, reading camps, and reading fairs. Challenges in the implementation of this component included difficulty recruiting and retaining volunteers to lead the community activities, seasonal fluctuation in attendance to the community activities, and geographic barriers to children’s participation in after-school reading activities. The in-school component of Leer Juntos, Aprender Juntos improved teachers’ participation in training and coaching and some aspects of the classroom literacy environment. Teachers in schools that implemented only the in-school component of Leer Juntos, Aprender Juntos participated in the training and coaching activities offered by the program, and a greater percentage of teachers in those intervention schools participated in professional development activities focused on reading instruction than teachers in the prevailing practice schools. In interviews, teachers said that the program’s training helped them implement instruction practices focused on developing students’ foundational reading skills, such as understanding the relationship between letters and sounds, identifying letters, and learning new words. Also, more classrooms in the intervention schools implementing the in-school component only than in prevailing practice schools displayed a complete alphabet written in K’iche’ and familiar words in Spanish and K’iche’. The favorable intermediate effects of the in-school component of Leer Juntos, Aprender Juntos did not translate into impacts on observed literacy instruction or on children’s reading skills. We did not find evidence of positive program effects of the in-school component on the instructional practices observed in the classroom, such as the instructional practices teachers use to teach five foundational literacy skills or the way teachers spent the time during the language or reading lesson that we observed. Additionally, we did not find statistically significant positive impacts of the in-school component on children’s reading skills three years after the program started. This finding does not mean that the children did not learn. Rather, it EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-5 shows that, on average, children whose schools were offered the in-school component of Leer Juntos, Aprender Juntos made similar progress in reading as children in the control group. The community action component of Leer Juntos, Aprender Juntos had some of the intermediate impacts predicted by the program logic model, but it did not have impacts on students’ reading skills. The community action component did not increase the amount of time children spent reading by themselves or with parents, but it did increase the likelihood that a sibling would read to them. However, this intermediate impact did not translate into favorable impacts on reading skills: students in schools implementing the full Leer Juntos, Aprender Juntos intervention had similar reading skills to students in schools that implemented only the in￾school component. The reason for the finding of null impacts of the two program components on children’s reading outcomes is unclear. One possible explanation is that the contrast between schools implementing the in-school component and prevailing practice schools (the schools in the control group) may not have been as distinct as anticipated because the program was implemented along with other reading programs. For example, the Leer Juntos, Aprender Juntos program was implemented in a context where several other initiatives—at the national and regional levels in the Department of El Quiché—had been launched to improve early-grade reading. Another possible explanation is that teachers lacked the K’iche’ language skills necessary to implement the program as intended. 2. Limitations The baseline of the evaluation was not a “true” baseline. The baseline assessment of students’ early literacy skills was administered about three months after random assignment, which was about two months after the rollout of the program’s teacher training activities. It was not feasible to measure baseline student outcomes before random assignment because several evaluation activities needed to be completed before data collectors could begin baseline data collection; these included identifying a local partner that could assist in recruiting and training field workers, implementing the data collection plan, and supervising data collection activities. As a result of the late baseline, the children’s skills that we captured in the baseline assessment (when students in the evaluation were in first grade) reflect the ability of the students after up to three months of potential exposure to different conditions caused by assignment of schools to different intervention groups. Despite this limitation, it is still appropriate to consider the evaluation’s baseline assessment as the reference point for the evaluation. Under most circumstances, a late baseline will still be useful for estimating the impact of that intervention as long as the impact on skill development is slow in the early period of program implementation (Schochet 2010). That is the case in the current evaluation because the teachers in the groups receiving the program (Groups A and B) had only been practicing their newly acquired skills for at most two months when the baseline data collection took place, and literacy acquisition in children is a process that happens gradually over many months or even years. The evaluation design assumes that the impacts of in-school and community action components of the intervention are additive. Specifically, the evaluation design assumes that EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-6 the effect of the community action component on its own relative to prevailing practice is equal to the sum of the impact of the combined in-school and community action components relative to the in-school component on its own. This assumption could be violated if implementing the in￾school component makes it easier or harder to simultaneously do community activities. We did not find clear evidence of such a violation, but it should be taken into account by any policymakers who wish to apply the lessons of this evaluation to a future implementation of community action on its own. Findings from this evaluation may not necessarily apply to other regions. It is always tempting to generalize from the experience of one or two evaluations to other contexts, but caution is warranted. In Guatemala, the evaluation included schools from just a few municipalities in El Quiche, which is just one of 22 departments in Guatemala. While a parallel evaluation was conducted in the Apurímac region of Peru, it too included schools from a single area of the country. In both countries, communities were selected with the following criteria in mind: the locations had to be within reasonable driving distance from each other to facilitate intervention and evaluation activities. They had to contain a high percentage of families that spoke one language besides Spanish, in this case K’iche’ (in Guatemala) or Quechua (in Peru). Results could differ if an approach like Leer Juntos, Aprender Juntos were taken in other communities with different characteristics and challenges than those of the communities included in this evaluation. 3. Lessons learned In Guatemala, the national early grade reading program seems as effective as a program implemented with technical assistance of a non-governmental implementer. in K’iche’. Teachers most commonly used Spanish for early grade instruction, while only one third of the students in the sample demonstrated proficiency in communicating in Spanish at baseline, perhaps preventing them from benefiting from the strategies teachers learned from Leer Juntos, Aprender Juntos training and coaching. Relying on volunteer work has substantial risks for fidelity of implementation. In the Department of El Quiché in Guatemala, where the program and evaluation took place, volunteer work is not common practice, and individuals want to be paid for their time working for the program. Therefore, it was difficult to recruit and retain volunteers, and to find volunteers that had the skills to lead the program activities consistently. This lack of skilled volunteers may have reduced the effectiveness of the community action component. 4. Recommendations Implementers and funders should consider the local context and assess the programmatic needs in that context before investing in designing, implementing, and evaluating new programs. In the evaluation’s region in Guatemala, it was difficult to recruit and retain volunteers with the skills to lead the program activities consistently. Therefore, the feasibility of using volunteers should be tested in the local context before implementing the program itself or an impact evaluation of a program that relies on such a strategy. It is also important to understand the prevailing practices to be able to establish whether any new program is distinct from programs already in place locally. A clear contrast between the intervention and EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH ES-7 the control group is critical to any evaluation that seeks to determine the effectiveness of a program, which in turn informs whether the program is worthy of further investment. This evaluation did not yield enough evidence to suggest that, as implemented with this population, the program’s results are commensurate with the level of investment required to sustain it. To improve future implementation in multilingual contexts of Leer Juntos, Aprender Juntos and other early grade reading interventions, program implementers, regional and national education authorities, and donors should find teachers that are proficient in their students’ mother tongue. Teachers’ proficiency in their students’ mother tongue is key to meeting the learning needs of children whose dominant language upon school entry is different from the school’s main language of instruction. To support the acquisition of children’s foundational literacy skills in this context, and be able to apply fully the Leer Juntos, Aprender Juntos’ teaching strategies, teachers need to be proficient in their students’ mother tongue (K’iche’ in the evaluation’s region in Guatemala). If teachers are not able to use reading instruction strategies in a language that is accessible to students, gains in teachers’ instructional skills resulting from teacher training and coaching may not translate into improvements in children’s reading outcomes. The initiative could meet this objective through pre-service training, in-service training, or alternative methods of recruitment and screening. If a pool of teachers who are proficient in their students’ mother tongue is not available, program implementers and donors should consider providing additional training to teachers on their students’ mother tongue and on the use of instruction strategies in that language prior to implementing a program that relies on those teachers’ skills. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 1 I. INTRODUCTION A. Early grade literacy education in linguistically diverse contexts The past decade has seen outstanding progress toward the Millennium Development Goals of universal primary school completion and gender parity in education, both around the world and in the Latin America and Caribbean (LAC) region, in particular. Yet deficits related to learning and access to education for poor and marginalized populations persist, particularly for groups whose mother tongue is not the same as the societal or official language of their country or region (Benavides et al. 2010; Gove and Cvelich 2011; Kim et al. 2016). In LAC countries with linguistically diverse populations, such as Peru and Guatemala, children enter school with a wide variation in ability to understand and speak the country’s societal or official language (Cueto et al. 2012). In the case of Peru and Guatemala, the societal or official language is Spanish. In these countries, some children have no knowledge of Spanish, whereas others speak and understand it fluently, even if they live and attend school in a region where the predominant language is the same as their mother tongue. Therefore, the challenge for educational programs aimed at improving reading skills is to work with a range of instructional approaches in a context of significant linguistic diversity. Such programs must focus on helping children develop the underlying skills that will ultimately result in improved reading skills. Studies have demonstrated that programs involving instructional practices and other supports in children’s mother tongue improve academic outcomes in some developing countries (Chesterfield and Abreu-Combs 2011; Crouch et al. 2009; Friedlander and Goldenberg 2016; Hernandez-Zavala et al. 2006; Patrinos and Velez 2009; Piper et al. 2016). However, no studies based on a rigorous evaluation design have been implemented in the LAC region. Moreover, existing studies have not focused on assessing the effectiveness of comprehensive reading programs that incorporate both transitional instructional approaches in the classroom and mother tongue supports for building reading abilities outside the classroom. Models aimed at increasing instruction time in the classroom and reading practice time at home (such as community engagement models, volunteer training to teach low-performing students, and remediation provided by locally trained teachers) have shown promise (Banerjee et al. 2010, 2007; Bruns and Luque 2014). Again, such models have not been rigorously evaluated in the LAC region, and their cost-effectiveness is unknown. B. The intervention and evaluation To address this need for rigorous evidence, the U.S. Agency for International Development (USAID) contracted with Mathematica Policy Research as an independent evaluator to design and implement rigorous evaluations of the agency’s investments in reading. The first evaluation funded under this contract was the nearly five-year evaluation of the USAID-funded Leer Juntos, Aprender Juntos program—an approach to early-grade reading instruction in LAC communities with linguistically diverse populations. Save the Children developed Leer Juntos, Aprender Juntos based on its Literacy Boost model, which includes teacher training and community involvement, and implemented the program in the K’iche’-speaking region of Guatemala and the Quechua-speaking Apurimac region of Peru. Mathematica worked with Save the Children to design a rigorous evaluation of Leer Juntos, Aprender Juntos in both Guatemala and Peru. This EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 2 report focuses on the impact evaluation findings for Guatemala, and a separate report presents the impact evaluation findings for Peru (Lugo-Gil et al. 2021a; Lugo-Gil et al.2021). 1. Leer Juntos, Aprender Juntos Leer Juntos, Aprender Juntos means Read Together, Learn Together. Save the Children, an international nongovernmental organization focused on promoting children’s rights and providing relief and support to children in developing countries, developed this program based on its Literacy Boost model. Save the Children has implemented Literacy Boost in at least 24 countries (Save the Children 2013a), including El Salvador, Haiti, Ethiopia, Pakistan, and more recently, Rwanda (Friedlander and Goldenberg 2016). Literacy Boost has three main components: (1) an in-school component that includes teacher training and coaching, (2) a community component that relies on volunteers to engage with young students and promote a culture of reading outside of regular instructional periods at school, and (3) a formative assessment component to track children’s progress in reading. The assessment component of Literacy Boost was incorporated as part of the in-school component of Leer Juntos Aprender Juntos. The combined in-school and community components represent the full Leer Juntos, Aprender Juntos intervention. The in-school and community components of the Leer Juntos, Aprender Juntos program align with the elements of reading instruction that existing research (Comings 2012) has identified as effective when implemented simultaneously: (1) text and materials, (2) teacher training and support, (3) community and parental support, and (4) assessment and tracking. Therefore, in addition to targeting important changes in the classroom, the Leer Juntos, Aprender Juntos program sought to engage community members and parents in the learning process. The community outreach component is a potentially low-cost, effective way to increase children’s time on task. The Leer Juntos, Aprender Juntos program adapted the teacher training and community action components of Literacy Boost to K’iche’- and Quechua-speaking populations1 in Guatemala and Peru, respectively. The program’s primary implementers—Save the Children in Guatemala and Kallpa, its local partner in Peru—trained teachers to teach and monitor students’ mastery of core reading skills. Save the Children also worked to strengthen parent and community involvement in building children’s reading abilities and opportunities to practice reading skills in their mother tongue. Those efforts included creating and providing context￾relevant reading materials in children’s mother tongue and conducting community activities that promoted reading engagement and a culture of literacy. Additionally, Save the Children implemented and assessed the component of formative assessment of the program, and was in regular contact with other teams implementing Literacy Boost in other countries to share experiences and learning. The implementation team was made up of Save the Children’s staff in Washington, DC, Guatemala City, and the regional office in the Department of El Quiché. Save the Children’s 1 In Guatemala, the program was implemented in five municipalities in the Department of El Quiché: San Antonio Ilotenango, Santa Cruz del Quiché, Santo Tomás Chichicastenango, San Andrés Sajcabajá, and Zacualpa. The Department of El Quiché is located northwest of the country’s capital, Guatemala City. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 3 staff in Guatemala (Save the Children/Guatemala, for brevity) recruited the regional team of specialists for the implementation of the program. The implementation of Leer Juntos, Aprender Juntos in Guatemala began in May 2013 and continued through March 2016. Program implementation covered three academic years, during which Save the Children provided the program to two cohorts of 1st grade students. In the first quarter of 2013, the local team recruited and hired local specialists2 and administrative staff, engaged with regional stakeholders, mapped the Literacy Boost approach to the national curriculum, and aligned reading instruction goals with the five core reading skills from the Literacy Boost framework. Subsequently, the team rolled out the training of trainers and conducted teacher training workshops. Planning for the implementation of Leer Juntos, Aprender Juntos in Guatemala began in fall 2012 with the development of a work plan and project strategy, recruitment and training of program implementation staff, meeting with stakeholders, development of a program monitoring and evaluation system, and preparation of program materials (Save the Children 2012, 2013). Save the Children rolled out the program in two phases. Implementation in Phase I began in 2013 and included half of the schools assigned to receive the full intervention (the teacher training and coaching component and the community action component, and half of the schools assigned to receive the teacher training and coaching component. In 2014, Save the Children continued implementing the program with Phase I schools and began implementing it with the remainder of the schools assigned to the groups where the full intervention and the teacher training and coaching component were offered. In-school component: teacher training and coaching. The in-school component of Leer Juntos, Aprender Juntos, which is based on the Literacy Boost model, followed the recommendations of the National Reading Panel for best approaches in reading instruction (National Reading Panel 2000). The goal of this component was to train and coach teachers so they become equipped to provide mother tongue and/or Spanish reading instruction focusing on five foundational reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension), to monitor students’ mastery of these skills, and to increase students’ access to and use of reading materials in the mother tongue. Teachers were taught to incorporate instruction in the mother tongue and adapt this instruction to the linguistic background of the students in a particular school. This school-based component entailed six main activities through which it is expected to increase allocation of class time on reading instruction and improve the quality of reading instruction: 1. Training trainers in reading instruction techniques in both Spanish and mother tongue 2. Training teachers in the five core skills of reading: alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension 3. Creation of materials for enhanced, print-rich classrooms in both Spanish and mother tongue 4. Mentoring and coaching teachers in reading instruction practices 2 Técnicos de Leer Juntos, Aprender Juntos. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 4 5. Training teachers in conducting formative assessments to track progress of children’s reading skills 6. Guiding teachers to incorporate five core skills of reading and related reading instruction techniques into daily school activities The teacher training activities (for more details, see the teacher training toolkit [Save the Children 2012a]) consisted of nine modules focused on the following topics: 1. Introduction to reading development and instruction for young children 2. Formative assessment 3. Addressing language issues in the literacy classroom 4. Letter knowledge/alphabetic principles 5. Phonological awareness 6. Reading fluency 7. Vocabulary 8. Reading comprehension 9. Reflecting on and applying best practices for teaching reading comprehension and formative assessment Teachers in 1st, 2nd, and 3rd grades in the intervention schools (schools implementing full intervention and the teacher training and coaching component) received most of the training content at the beginning of the school year.3 The remainder of their training was provided throughout the school year (Save the Children 2013, 2014, 2015). For example, in the first year of implementation of Leer Juntos, Aprender Juntos in Guatemala, teachers in Phase I schools were trained on the first five modules (introduction to reading development, formative assessment, language issues, letter knowledge, and phonological awareness) over two days (four to six hours each day) in June 2013. The remainder of the training was delivered in one-day sessions held in July (on fluency and vocabulary modules) and August (training on reading comprehension and reflecting and practicing modules). In the second year of implementation, teachers in Phase I and Phase II schools were trained on the first five modules during two days in January 2014, with the remainder of the training provided in one-day sessions held in March (covering vocabulary and fluency modules), May (covering the reading comprehension module), and September (covering the last module on reflecting and practicing). In 2015, the third year of implementation, teacher training was provided in three sessions held in the first quarter of the year, and two sessions held during the summer. After the teacher training, coaches visited the classrooms at least once, and in most cases, twice every three months (Save the Children 2013, 2014, 2015). The purpose of the visits was to directly support teachers by observing their work, interacting in the classrooms and/or demonstrating teaching techniques, and making recommendations and suggestions to teachers 3 The school year in Guatemala begins in January and ends in October. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 5 for improving the application of the teaching strategies taught in the program’s training. Coaches also supported “Teacher Circles,” which are sessions held during recess or in the afternoons or evenings that focus on (1) strengthening and practicing the topics learned in the training, (2) conducting activities to promote the development of the five core reading skills, (3) assisting teachers with designing lesson plans that incorporate the five core reading skills, and (4) identifying language problems that arise in different contexts and proposing strategies to promote the development of literacy skills in the students’ mother tongue. For more details on the implementation of the teacher training and coaching component in the evaluation schools, such as frequency of activities, please consult the Leer Juntos, Aprender Juntos midline report (Lugo￾Gil et al. 2017b). Community action component. The community action component of the program, also based on the Literacy Boost model, aimed to strengthen parental and community involvement in building children’s reading abilities and increasing their opportunities to practice reading outside of school. (For more details, see the Literacy Boost community action tool kit [Save the Children 2012b].) This component drew on the many ways that parents, older siblings, and community members can support children in becoming readers. Community activities were designed to be enjoyable, to engage students, and to empower adults even if they themselves are not fully literate. Parents and community members who do not read can engage children in oral language through storytelling and still contribute to children’s listening skills, oral comprehension, and vocabulary development. This component is delivered through group activities, such as reading camps, reading buddy sessions (peer assistance), reading festivals, and reading contests led by community volunteers. Specifically, the community action component included the following main activities (Nieto 2015; Save the Children 2013, 2014, 2015): 1. Creating printed materials in the children’s mother tongue, Spanish, or both, to build portable libraries known as book banks. In Guatemala, the program’s team also selected and purchased books from book fairs and other appropriate sources to contribute to the community book banks collections. 2. Promoting the use of book banks among community members. Each community had at least one book bank. 3. Conducting reading activities in the community such as story hours, reading camps, and reading festivals or fairs. These activities were implemented during the school year (January through October) so that invitations for children and their families to participate in the activities could be distributed in the classrooms, and community volunteers could have time off during school vacation. The intervention’s community activities were as follows: a. Story hour. This activity was conducted in the afternoons and in parallel with the distribution of books through the community book banks. These activities were conducted in each community at least once every three months. During story hour, community volunteers told traditional stories such as how corn was first planted, or stories about the community. b. Reading camps. In this activity, volunteers read stories to children. This structured activity took place outside of school hours weekly in each school community in Group A. The purpose of the reading camps was to stimulate the development of all five core EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 6 reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension) in the mother tongue or in Spanish and to incorporate storytelling and reading games to engage children. c. Reading buddies (peer assistance). In this activity, older children (in grades 4 to 6) read with the younger children (grades 1 to 3) to help them improve their reading skills. This activity was implemented once or twice per week in each Group A community, mostly within the schools to maximize the number of children participating. d. Reading festivals or fairs. Children participated in games and activities in which they could practice, develop, and strengthen their reading skills. In Guatemala, volunteers organized reading fairs in coordination with community leaders and some school principals and teachers and implemented the fairs in each community at least once each year of program implementation (2013–2015). e. School–community accountability meetings. Local education leaders and councils met with community authorities (such as members of the school board and the community development councils) to discuss the funding sources of reading activities, the progress achieved with the reading activities implemented in schools and the communities, the resources used and their purpose, and the future sustainability of the activities. In Guatemala, school–community accountability meetings were held at least once in each community in 2013 and 2014, the first two years of program implementation for Phase I schools and communities and the first year of implementation for Phase II schools and communities. 4. Coordinating peer assistance through reading buddies. The purpose of this training was to enable volunteers to implement the community action activities (such as promoting the use of book banks, reading camps, reading buddies, and story hour) with children in their communities. In the training, volunteers learned how to lead the community reading activities and developed a weekly schedule of activities for their community to guide the rollout of implementation. 5. Conducting school–community accountability meetings and reading awareness workshops with parents and community members. The purpose of these workshops was to promote awareness and understanding among parents and community members about how the community reading activities help children learn to read. The workshops also aimed to raise awareness about how important it is for children to start learning to read in their mother tongue and to have a dedicated place (“reading corner”) for reading at home. 2. Prevailing practice in Guatemala The prevailing practice in Guatemala was potentially influenced by national, departmental, and local forces. The ministry’s approach to the implementation of the national early-grade curriculum emphasizes instruction on foundational reading skills (for example, alphabet knowledge and decoding), leading to reading fluency and reading comprehension. According to the standards of the national curriculum, students are expected to learn to decode familiar and unfamiliar words in 1st grade, to develop reading fluency in 2nd grade, and to read with comprehension and use reading as a tool to acquire new knowledge in 3rd grade (Cotto and Quiñonez 2014). However, the practices implemented in the evaluation schools could have EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 7 reflected priorities and strategies, discussed below, that were prevalent at more local levels of education governance. Since the 1980s, the government of Guatemala, through the Ministry of Education’s DIGEBI (Dirección General de Educación Bilingüe Intercultural), has designed and implemented initiatives that aim to deliver bilingual education services in students’ mother tongue and in Spanish. The DIGEBI prescribes that children whose mother tongue is not Spanish enter bilingual education programs as early as possible, preferably in preschool or kindergarten (if preschool and/or kindergarten are available) or in 1st grade (Rubio 2004). According to DIGEBI, children should first acquire literacy skills in their mother tongue and then gradually transition to learning those skills in Spanish. Therefore, in Guatemala, there is not a specific grade level at which instruction should transition fully from the mother tongue to Spanish; rather, the transition should be gradual, and at each grade level, the amount of instruction in Spanish should be increased. One of the most recent and largest education initiatives in Guatemala is the national Leamos Juntos (We Read Together) program, which was established in 2012 and is implemented by the Guatemalan Ministry of Education. The primary objective of the program is to promote reading skills and cultivate, in bilingual and monolingual students, the habit of reading through various activities in the school, with family, and in the community. To meet this objective, the Ministry of Education provides teacher training, develops and distributes reading materials in Spanish and in the languages of the numerous ethnic groups in the country, establishes community alliances, and encourages parents to visit communal libraries with their children and practice reading at home (Ministerio de Educación 2012). The Leamos Juntos reading program implements five strategies at the school level: (1) enriching the classroom literacy environment, (2) providing a dedicated area in the classroom for reading, (3) providing libraries or reading spaces throughout the schools, (4) conducting schoolwide activities to promote reading, (5) daily reading periods, and (6) guided reading and writing practice. In addition to the school-based strategies, the national reading program also included communitywide strategies to promote reading, such as Cuentacuentos (oral storytelling), Radiocuentos (one-hour radio broadcasted story reading), and reading and writing contests. Therefore, the main goals and overarching strategies of the Leamos Juntos reading program are similar to the Leer Juntos, Aprender Juntos program in several ways, but the programs are implemented in different ways. Among the schools participating in the evaluation, participation in Leamos Juntos was similar across the three groups of schools implementing each of the approaches that we are comparing in this evaluation: about 66 to 70 percent of the schools in each treatment group participated in Leamos Juntos (see Table III.1 for more details on evaluation schools’ participation in these programs). While we conducted the evaluation of Leer Juntos, Aprender Juntos, other initiatives were taking place in Guatemala that focused on improving reading achievement—for example, USAID’s Leer y Aprender initiative (Lifelong Learning program)4 , which has one component targeting children in elementary grades and another targeting youth. The component targeting children focuses on improving teaching practices and increasing time dedicated to teaching 4 See http://www.usaidlea.org/home.html. The initiative was being implemented in five regions, including El Quiché, where the current evaluation was conducted. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 8 reading in schools, providing reading materials to students, increasing time learning to read in a Mayan mother tongue, and conducting assessments to measure students’ progress in learning to read. Another example is USAID’s Reforma Educativa en el Aula (Education Reform in the Classroom) (REAULA),5 an initiative aimed at strengthening the capacity of the Ministry of Education in Guatemala to improve the quality of the education in the country. This initiative was implemented by Juárez and Associates from 2009 to 2014, and consisted of providing technical assistance to the Ministry of Education focused on (1) improving the quality of teaching, (2) improving the learning environment in the classroom, (3) promoting learning in mother tongue and a second language, (4) increasing access to education, and (5) promoting participation of parents and the community in general in the students’ learning process. In addition, there were two other regional efforts to promote reading and improve student reading outcomes. In 2013, the Departmental Directorate of Education (DIDEDUC), organized a network of organizations implementing education initiatives in the Department of El Quiché, known as Red Educativa. The network included a committee dedicated to improving the delivery of trainings and materials to promote and strengthen reading instruction, supporting schools on specific needs, supplying books and other reading materials focused on reading instruction in primary grades, instituting a compulsory 30-minute-per-day reading practice period in the early primary grades, meeting with parents to promote school enrollment and prevent dropouts, and collaborating with other initiatives to provide support and materials for teaching reading, math, and bilingual education. In 2015, DIDEDUC also initiated nine lines of action to implement a reading model for bilingual intercultural environments designed by DIGEBI and REAULA. In coordination with the CTA’s (Coordinadores Técnicos Administrativos), DEDEDUC trained 622 teachers in the western highlands on this model for bilingual literacy instruction6 . We provide more details on regional reading initiatives in Chapter III. C. Overview of Leer Juntos, Aprender Juntos evaluation As noted in the introduction, the Leer Juntos, Aprender Juntos evaluation was conducted to contribute to understanding of what works in LAC to improve early grade reading. Knowing the most cost-effective approach to improving reading skill acquisition can guide policy makers and educators in their efforts to promote child literacy. Specifically, this evaluation sought to understand whether the Leer Juntos, Aprender Juntos program was more cost effective than prevailing practice, and whether the community action component of the program was cost effective approach to increase learning. It did so by comparing the impacts on learners and teachers of the full program, including the in-school and community components, the in-school program only, and the usual education services provided to schools (which we refer to as prevailing practice). In order to have a better understanding of the program in different contexts, separate but parallel impact evaluations of Leer Juntos, Aprender Juntos were conducted in Peru and Guatemala. 5 See http://www.reaula.org/index.php. 6 The model was implemented in El Quiché, Huhuetenango, San Marcos, Totonicapán, and Quetzaltenango. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 9 II. EVALUATION DESIGN AND DATA As described in Chapter I, the purpose of this evaluation is to determine the relative impact and cost effectiveness of the in-school and community action components of the Leer Juntos, Aprender Juntos program. This chapter details the evaluation design, the program’s logic framework, the types of data we collected, the analytic methods, and the evaluation sample. A. Evaluation questions The evaluation sought to answer the following two primary questions about program impacts: ● What is the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to prevailing practice? This question contrasts schools in Group B to those in Group C. ● What is the impact of the community action component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to an intervention that does not have the community action component? This question contrasts schools in Group A to those in Group B. In addition to these two primary questions, the evaluation addresses a question about implementation: ● Were the program components implemented as intended? This question relates to whether each program component’s services were offered as originally intended, whether participants took part, and whether these program components had intermediate impacts on teaching and the availability and use of reading materials. It also considers any barriers or challenges to effective implementation. B. Evaluation design In Guatemala, we recruited 150 schools.7 Within each school, we followed a group of children from 1st grade through the end of 3rd grade. The evaluation was implemented as a randomized controlled trial, in which the evaluation team randomly assigned schools to the three treatment groups (we explain the assignment process to treatment groups in detail in Chapter II): 1. Group A: These schools implemented the full intervention (the teacher training and coaching component and the community action component). We refer to this group as Leer Juntos, Aprender Juntos, or Leer Juntos for ease of presentation. 2. Group B: These schools implemented only the in-school component of the intervention. We refer to this group as Leer Juntos–school only. Including this group in the evaluation enabled us to isolate the effects of the in-school component from the effects of the community action component. 7 In Peru, we recruited 145 schools. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 10 3. Group C: These schools did not implement any of the components provided by Leer Juntos, Aprender Juntos but instead implemented the prevailing reading instruction approach and supports in the early grades in each country, in this case, Guatemala. We refer to this group as the prevailing practice group. To provide rigorous evidence on the relative impacts of the in-school and community action components of the program to improve early-grade reading, we used a randomized controlled trial design in which we randomly assigned schools to the three intervention groups (Figure II.1). We conducted random assignment at the school level, because the three approaches to improve early-grade reading are implemented at this level. Aligned with program rollout, the evaluation was carried out in two phases. We randomly assigned 75 schools to treatment groups in 2013 (Phase I) and 75 schools in 2014 (Phase II) in Guatemala.8 After random assignment, the evaluation team randomly selected one 1st grade classroom, one teacher, and 10 1st grade students from each school to serve as the analysis sample for the evaluation. We sampled this way to minimize data collection cost and burden while still obtaining data that were representative of the schools and students in the evaluation. We conducted two follow-up data collections after the baseline: one at the end of the second year of implementation (midline, in 2014 for Phase I and 2015 for Phase II) and another at the end of the third year of implementation (endline, in 2015 for Phase I and 2016 for Phase II). Implementation of the program began in May 2013 for Phase I schools and March 2014 for Phase II schools. For a more detailed timeline of the data collection and program implementation activities, see the Figure A.1 in Appendix A. We present results for students pooled across the two phases because we did not find any significant pattern of differences (see Appendix H, Table H.3) between phases. This approach is the same we have taken in earlier reports for Guatemala and Peru (Lugo-Gil et al. 2016a, 2016b, 2017a, 2017b). 8 In Peru, the evaluation team assigned a similar number of schools in the same manner: 74 schools in Phase I and another 71 in Phase II. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 11 Figure II.1. Leer Juntos, Aprender Juntos evaluation design Source: Leer Juntos, Aprender Juntos evaluation plan (Glazerman et al. 2013). The primary outcomes of interest for the impact analysis presented in this report are early￾grade reading skills, such as fluency and comprehension, but we also examined other outcomes, such as improved classroom practices and culture of reading at home and in schools. We answer the first question of the evaluation, about the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to prevailing practice, by comparing outcomes for schools in Group B with those in Group C. And by comparing the outcomes for schools in Group A with those in Group B, we answer the second evaluation question, which is to assess the community action component of Leer Juntos, Aprender Juntos on early-grade reading and other outcomes relative to an intervention that does not have the community action component. To understand the implementation context and address the program implementation question, we adopted a mixed-methods approach integrating survey data, interviews, focus groups, and a qualitative review of program documents. These data came from implementers, teachers, and parents of students. C. Leer Juntos, Aprender Juntos logic framework Figure II.2 illustrates the logic framework for the program. As we describe in Chapter I, Leer Juntos, Aprender Juntos has two components: (1) a teacher training and coaching component that was implemented within schools (Groups A and B), and (2) a community action component that was implemented outside of school hours with children and their families within the community (Group A). We expect the program to have effects in three key areas: (1) the instructional practices that teachers implement in their classrooms, (2) children’s access to and use of instructional materials in their mother tongue and in Spanish, and (3) participation in community reading activities after school hours, among other intermediate outcomes. The expected final outcomes for the program are children’s improved foundational and reading comprehension skills. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 12 Figure II.2. Leer Juntos, Aprender Juntos logic framework Source: Leer Juntos, Aprender Juntos Performance Management Plan (Save the Children 2013). D. Types of data collected for the evaluation We partnered with DMC Consultores (DMC) to collect data for the evaluation in Guatemala. We collected data from households, schools, classrooms, teachers, and students, following the expected intermediate and final outcomes of Leer Juntos, Aprender Juntos that the program’s logic framework indicates (Figure II.2). The survey instruments, data documentation, and data files used for this evaluation are available online upon request as restricted use data files at USAID’s Development Data Library website (https://data.usaid.gov/). We describe the evaluation’s data collection activities in more detail in this section of the report. 1. Data collection in schools and households The evaluation team collected data at baseline and again at the midline and endline points of program implementation. Data collection for the baseline occurred when the children participating in the evaluation were in 1st grade. The midline (first follow-up) data collection took place during the second year of program implementation for children in both Phase I and II, when most of the children in the evaluation should have progressed to 2nd grade and have had about one to one and a half years of potential exposure to the intervention. The endline (final follow-up) data collection took place three years after the baseline, when most of the children in the evaluation should have progressed to 3rd grade and have had two or more years of potential exposure to the intervention. See Appendix A, Figure A.1 for a detailed timeline of Leer Juntos, Aprender Juntos activities in Guatemala. At the baseline, midline, and endline data collections, we observed schools and classrooms and interviewed teachers. The school observation used a checklist that the evaluation team could EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 13 complete based on an environmental scan of the school premises. In the classroom observation, the evaluation team observed and recorded information on teachers’ instructional practices, teachers’ and students’ language use in the classroom and time on task, and other school and classroom characteristics during a one-hour class period. The evaluation team also administered an in-person survey of teachers about their instructional practices; education and experience; participation in professional development activities; occupational needs; career expectations; and background characteristics, such as proficiency in K’iche’ and income. For more information about the development, administration, and contents of the instruments for the school and classroom observations, and the teacher survey, refer to the Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016a). In addition to the observations and teacher surveys administered in schools, the evaluation team administered individual reading assessments to children in the baseline and endline data collections. During the baseline data collection, when evaluation children were in the 1st grade, the evaluation team assessed children’s oral language proficiency and emergent literacy skills (letter identification, emergent writing, emergent reading, phonemic awareness, pseudo-word decoding, and passage comprehension skills) in Spanish and K’iche’. The Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016a) presents a detailed description of the process to develop those assessments and of their contents, and reports on the findings from the examination of the baseline data. In the endline data collection, the individual assessments administered to children focused on reading skills in Spanish. This assessment was administered when most of the children in the evaluation attended the 3rd grade.9 The endline reading skills assessment was adapted from REAULA’s materials for curriculum-based reading assessments (USAID REAULA 2013). The endline assessments were conducted in Spanish, because Spanish is the language of literacy instruction in most (90 percent in each treatment group) of the 3rd grade classrooms we observed at the endline (see Appendix E, Table E.9 and the baseline report [Lugo-Gil et al. 2016a]). In the midline data collection, we observed the evaluation’s schools and administered a teacher survey and conducted a household survey. This survey was administered in person to the main caregivers (usually the mother or father) of the children in the evaluation at their homes. The purpose of this survey was to learn about the household composition, family socioeconomic status, household assets, children’s schooling background and routines at home, and children’s and families’ participation in reading activities offered in their communities. Findings from the midline data relevant to understanding intermediate outcomes are also presented in Chapter IV of this report. 2. Response rates Table II.1 shows that the overall response rates to each round of data collection were high, close to 90 percent or higher for every round of data collection (baseline, midline, and endline). Of the 150 schools that were randomly assigned to treatment groups, one school assigned to Group C closed and one school assigned to the same group refused to participate in the midline 9 In the final follow-up, we assessed the reading skills of the children in the sample even if they had not progressed to the 3rd grade at the time of data collection. That is, we followed up with children in the evaluation in whatever grade they were in. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 14 data collection. The latter school allowed data collectors to enter the school for the endline data collection. Therefore, the impact analysis presented in this report includes 149 schools (50 in Group A, 50 in Group B, and 49 in Group C). The number of children in the sample who were randomized into treatment groups is 1,480, and the number who completed a reading skills assessment in the 3rd grade is 1,338. The low levels of attrition across all three groups give us confidence that the evaluation sample continues to be representative across all treatment groups at endline and that we’ve maintained our experimental design. Table II.1. Response rates to data collections in the evaluation Data collection round Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) All Baseline data collection (2013 in Phase I and 2014 in Phase II) School infrastructure observation, classroom observation, and teacher survey 100 100 100 100 Emergent literacy skills assessment 95.6 97.4 97.4 96.8 Midline data collection (2014 in Phase I and 2015 in Phase II) School infrastructure observation, classroom observation, and teacher survey 100 100 96.0 98.7 Household survey 93.8 94.0 94.0 93.9 Endline data collection (2015 in Phase I and 2016 in Phase II) School infrastructure observation, classroom observation, and teacher survey 100 100 98.0 99.3 Reading skills assessment 91.1 92.0 88.2 90.4 Number of children who were randomized into treatment groups 482 498 500 1,480 Number of children who completed endline assessments 439 458 441 1,338 Source: Leer Juntos, Aprender Juntos School Infrastructure Observation, Classroom Observation, Teacher Survey, Household Survey, Emergent Literacy Skills Assessment, and Reading Skills Assessment—Baseline 2013 and 2014, Midline 2014 and 2015, and Final Follow-ups 2015 and 2016. 3. Qualitative data collection The evaluation team also collected qualitative data from stakeholders to understand how the school and community components were implemented, the facilitators and barriers to implementation, and teachers’ and parents’ perceptions of the Leer Juntos, Aprender Juntos program. We partnered with DMC to collect data for the evaluation. Qualitative data collection took place during 2014 and 2015 (see Appendix A, Figure A.1). The data are used to understand how program services were delivered, how services were received by stakeholders, and what services were received in each experimental group. They also illustrate some of the challenges to implementation. Mathematica and DMC, in collaboration with the Departmental Directorate of Education (DDE), the Administrative Technical Coordinators of the target municipalities, and Save the Children staff, recruited a purposive sample of parents of students in the treatment groups (Groups A and B), 1st through 3rd grade teachers in treatment and control schools, community volunteers, and local and implementation specialists from Save the Children to participate in focus groups and interviews. Using a convenience sampling approach, the team identified and selected participants that were easily accessible and could provide rich information relevant to the goals of the evaluation. Participant inclusion criteria were flexible, and took into EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 15 account potential constraints and barriers to participation in interviews and focus groups like distance, access to transportation, and work schedules. Participant inclusion criteria were: • For parents, to have a child enrolled in first, second or third grade, in a school assigned to Group A or B (phase I and phase II schools included) • For teachers, to teach first, second and third grade, in a school assigned to Group A, B, or C (phase I and phase II schools included) regardless of employment contract type • For volunteers, to be an active volunteer for the community action component DMC and Mathematica staff carried out interviews and focus groups with stakeholders at the Departmental Education Directorate (DDE), Save the Children staff, community volunteers, parents, and teachers from Phase I and Phase II schools located in the five municipalities (San Antonio Ilotenango, Santa Cruz del Quiché, Santo Tomás Chichicastenango, San Andrés Sajcabajá, and Zacualpa) of the Department of El Quiché. All interviews and focus groups were audio-recorded and transcribed verbatim for analysis. Field staff wrote interpretative summaries synthesizing key themes that emerged during the focus groups which were later used to triangulate findings from the thematic analysis of verbatim transcripts. The team faced two main challenges with the recruitment of participants for qualitative data collection in Guatemala. The first challenge was to ensure that participants who granted consent would actually attend the focus group meetings on the scheduled dates. To mitigate the risk of low attendance, DMC staff conducted several rounds of calls to participants to remind them of the focus group meeting date, time and place. To the extent possible, focus groups took place at central locations in larger municipalities which were more accessible by public transportation. The second challenge was that some parents who agreed to participate were fluent primarily in K’iche. DMC included staff fluent in K’iche as part of the qualitative data collection team to ensure that focus groups were conducted in the language in which parents where most comfortable speaking. (See Appendix B, Table B.1 for a summary of respondent types, methods, topics of inquiry, and location for all qualitative data collection activities, and Appendix J for the qualitative instruments used in these data collection activities.) E. Analytic approach We examined differences between treatment groups in measures at the school level (characteristics of schools, teachers, and classrooms) and at the child level (emergent literacy and reading skills and home literacy environment). To assess the differences between treatment groups in those measures, we conducted regression analyses that accounted for the design of the random assignment of schools and the data cohort (Phases I and II). The design used stratified random assignment, in which we first grouped schools with similar characteristics (within phase) into strata and, within each stratum, randomly assigned schools to the three groups. This approach increases the likelihood that certain observable and unobservable characteristics are evenly balanced across the three treatment groups. For more details on how we conducted the stratified randomization of schools and selected the sample, refer to the Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016a). With those regressions, we computed regression-adjusted means of the relevant measures for each treatment group (A, B, and C) and tested whether the differences in those adjusted EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 16 means between Groups A and B, and between Groups B and C, are statistically different from zero. We also computed effect sizes, which are the differences in adjusted means between two treatment groups expressed in standard deviation units. We used the p-values from the tests of differences in group means to assess statistical significance of the differences in means. We defined the difference in means between treatment groups as statistically significant whenever the p-value of the test was lower than 0.05. In determining the statistical significance of mean differences between treatment groups in measures at the child level, we accounted for the clustering of children within schools. For analyses of children’s reading skills measures in the 3rd grade, we defined the analytic sample as the sample of children who completed a reading skills assessment in the final follow￾up (endline) data collection (“respondents”). Therefore, our strategy to estimate program impacts follows an intent-to-treat approach.10 Our regression analyses of children’s reading skills measures in the 3rd grade also control for several variables at the child, household, and school levels measured at baseline. We included these variables in our analyses to account for any initial, observed differences between treatment groups and to increase the precision of our impact estimates. These control variables include (1) child characteristics such as age, sex, results from the baseline emergent literacy assessments, and language spoken at home at the time of baseline data collection; (2) household characteristics, such as mother’s education level and household income and assets; and (3) school characteristics at the time of baseline data collection, such as participation in other programs such as preschool participation, number of physical and health hazards observed in the school, and school infrastructure resources. Appendix C, Table C.1 presents the complete list of control variables included in the analyses of children’s reading skills measures. We imputed missing values in the control variables using a regression-adjusted average of the nonmissing values for the variable. We describe the process to impute missing values in control variables in more detail in Appendix C. The approach for the reduction and analysis of qualitative data sought to meet the goals of the implementation research questions efficiently, and in keeping with the resources available for the qualitative component of this evaluation. Mathematica staff reviewed all transcripts, field notes, and interpretative summaries submitted by DMC’s field staff team for quality assurance. Then, we conducted a thematic analysis of the transcripts and other sources of qualitative data in three steps. First, we parsed meaningful segments of transcripts (or field notes) linked to the key implementation research questions. Second, we extracted memos of key findings and illustrative quotes, and organized them in synthesis tables. Finally, we triangulated findings from these tables with the field notes and interpretative summaries submitted by DMC’s field team. 10 With the intent-to-treat approach, all the evaluation participants in the schools that were randomly assigned to the three treatment groups (Groups A, B, and C) are included in the impact analysis and are analyzed in the groups to which they were randomized. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 17 F. Characteristics of the sample at endline This section provides a snapshot of the characteristics of the schools, teachers, and students in the evaluation sample during the third year of program implementation, when most of the students in the sample attended the 3rd grade. 1. Schools The schools in the evaluation are located in the Department of El Quiché, located northwest of the country’s capital, Guatemala City. The Department of El Quiché was selected for the evaluation because its population is ethnically and linguistically diverse,11 which provided the opportunity to examine how the Leer Juntos, Aprender Juntos intervention works in a context with a heterogeneous population, particularly given their variation in languages. The evaluation schools were selected from five municipalities in the Department of El Quiché. In Phase I, schools were drawn from three of those five municipalities: San Antonio Ilotenango, Santa Cruz del Quiché, and Santo Tomás Chichicastenango. Phase II schools were drawn from the other two municipalities: San Andrés Sajcabajá and Zacualpa. Figure II.3 presents a map of Guatemala that highlights the location of the Department of El Quiché and the municipalities of the evaluation schools. Figure II.3. The Department of El Quiché in Guatemala 11 The language spoken by most people in the region is K’iche’, which is the Mayan language with the highest prevalence in Guatemala. Other Mayan languages spoken in this region are Ixil, Q’eqchi’, Poqomchi’, Sakapulteko, Mam, and Uspanteko. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 18 As shown in Table II.2, most schools in the sample are located in the municipality of Zacualpa, followed by the San Andrés Sajcabajá and Santo Tomás Chichicastenango municipalities. The schools in all three intervention groups average five primary classrooms and about 17 3rd grade students. The classrooms in each treatment group have, on average, an equal number of female and male students. About three out of every five of the schools in each group have classrooms with more than one grade (multigrade or single-teacher schools12), meaning that the children in 3rd grade in these schools are taught in classrooms together with children from other grades. The average school characteristics were similar for each treatment group (Table II.2). Fifty￾five of the 56 differences between treatment groups in the school infrastructure characteristics that we examined were not statistically significant (see Appendix D, Tables D.1 through D.3). Nevertheless, our approach for estimating program impacts controls for baseline characteristics of schools, such as available utilities and facilities (see Appendix C, Table C.1 for the complete list of variables for which we controlled in calculating program impacts. For details on the infrastructure characteristics of evaluation schools, refer to the Leer Juntos, Aprender Juntos baseline report (Lugo-Gil et al. 2016a)). Table II.2. Characteristics of evaluation schools in the final follow-up year Characteristic Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Percentage of evaluation schools in the following municipalities: San Antonio Ilotenango 14.0 12.0 16.1 2.0 (0.483) -4.1 (0.159) Santa Cruz del Quiché 18.0 18.0 16.0 0.0 (1.000) 2.0 (0.224) Santo Tomás Chichicastenango 18.0 20.0 18.0 -2.0 (0.393) 2.0 (0.392) San Andrés Sajcabajá 22.0 22.0 22.0 0.0 (1.000) 0.0 (1.000) Zacualpa 28.0 28.0 28.0 0.0 (1.000) 0.0 (1.000) Percentage of schools with multigrade classrooms 62.0 62.0 57.9 0.0 (1.000) 4.1 (0.529) Number of primary grade classrooms in use (average) 5.0 5.2 5.1 -0.2 (0.454) 0.1 (0.710) Number of students enrolled in observed 3rd grade classrooms (average) Total 17.1 17.5 16.9 -0.4 (0.739) 0.6 (0.603) Female 7.9 8.0 7.9 -0.1 (0.912) 0.1 (0.900) Number of schools 50 50 49 Source: Leer Juntos, Aprender Juntos School Infrastructure Observation Form and Classroom Observation Form— Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. 12 In schools with multigrade classrooms, several grades are taught in the same classroom. In single-teacher schools, one teacher provides instruction to all the primary grades offered in the school, usually in the same classroom. In schools with single-grade classrooms, only one grade is taught in each classroom. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 19 2. Teachers The teachers surveyed in the 3rd grade follow-up have similar background characteristics across the three treatment groups. In particular, we did not find any statistically significant differences between treatment groups in terms of teachers’ sex, age, and highest level of education (see Table II.3). We also did not find any statistically significant differences in the composition of the sample of teachers in the evaluation in terms of years of teaching experience and tenure in the early grades. As shown in Table II.4, there were no statistically significant differences across treatment groups in the teachers’ number of years of experience—overall, at the current school, or teaching 3rd grade. About 14 percent of the 3rd grade teachers in Groups A and B, and 8 percent in Group C, taught the same evaluation children as they moved through 1st, 2nd, and 3rd grades. Another 10 to 22 percent of the 3rd grade teachers in the evaluation have taught at the current school for less than two years. This is equivalent to four to six fewer teachers in Group C than in Groups A or B that have taught at their current school for less than two years. However, these differences between treatment groups are not statistically significant. Table II.3. Background characteristics of the teachers of the evaluation students in the final follow-up year Characteristic Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Female (percentage) 56.0 50.0 57.1 6.0 (0.580) -7.1 (0.517) Age (average) 35.8 32.9 34.2 2.9 (0.053) -1.3 (0.381) Highest level of education (percentage) Degree in urban pedagogya 20.0 12.0 7.8 8.0 (0.174) 4.2 (0.481) Degree in rural pedagogya 54.0 52.0 58.1 2.0 (0.841) -6.1 (0.543) Degree in intercultural bilingual educationb 8.0 8.0 10.0 0.0 (1.000) -2.0 (0.700) Professoriate 6.0 16.0 9.0 -10.0 (0.115) 7.0 (0.269) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. a Teachers with a degree in urban (rural) pedagogy have completed a secondary school-level pedagogy program and have practiced in schools in urban (rural) areas. b Teachers with a degree in intercultural bilingual education have the ability to understand, speak, and write in K’iche’ (or the indigenous language of the community where they teach). In addition, teachers with this degree have received training in pedagogical methods to teach bilingual and multicultural students. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 20 Table II.4. Experience and tenure characteristics of the teachers of the evaluation students in the final follow-up year Characteristic Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Years of teaching experience (average) 10.5 9.8 10.8 0.7 (0.483) -1.0 (0.352) Years of teaching at current school (average) 6.4 5.9 7.0 0.5 (0.613) -1.0 (0.300) Years teaching 3rd grade (average) 4.6 3.6 3.8 1.0 (0.115) -0.1 (0.825) Percentage of teachers who have taught evaluation children for three years (when they were in the 1st, 2nd, and 3rd grades) 14.0 14.0 8.0 0.0 (1.000) 6.0 (0.363) Percentage of the teachers in 3rd grade in the evaluation who have less than two years of experience at current school 22.0 18.0 9.8 4.0 (0.585) 8.2 (0.268) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. Finally, we found that although most teachers (82 percent) in each treatment group reported that they understand K’iche’, fewer than 50 percent of the teachers in the sample reported that they speak K’iche’ well or very well, and fewer than 20 percent of the teachers in each treatment group reported an advanced level of written K’iche’ (see Table II.5). Teachers’ reported language skills in K’iche’ were similar across treatment groups. Table II.5. Teacher-reported language skills (percentages of teachers) Skills Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Teacher’s mother tongue Spanish only 44.0 48.0 42.9 -4.0 (0.653) 5.1 (0.573) K’iche’ only 36.0 28.0 25.9 8.0 (0.342) 2.1 (0.802) Both Spanish and K’iche’ 20.0 24.0 31.2 -4.0 (0.618) -7.2 (0.375) Teacher understands K’iche’ 82.0 82.0 82.0 0.0 (1.000) 0.0 (1.000) Teacher’s proficiency level in spoken K’iche’. Very poor to poor 26.0 34.0 23.9 -8.0 (0.342) 10.1 (0.234) Fair 26.0 26.0 40.3 0.0 (1.000) -14.3 (0.090) Good to very good 48.0 40.0 35.8 8.0 (0.359) 4.2 (0.635) EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 21 Skills Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Teacher’s proficiency level in written K’iche’. Very poor 22.0 24.0 27.1 -2.0 (0.809) -3.1 (0.711) Poor 20.0 22.0 13.9 -2.0 (0.796) 8.1 (0.298) Fair 38.0 36.0 45.2 2.0 (0.833) -9.2 (0.338) Good to very good 20.0 18.0 13.9 2.0 (0.787) 4.1 (0.583) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. 3. Students As we show in Table II.6, about 52 to 57 percent of the students in each treatment group had progressed to the 3rd grade by the time the evaluation team administered the endline (final follow-up) reading skills assessment. The difference in the percentage of students who had progressed to the 3rd grade between Groups B and C is statistically significant at the 0.10 level (p-value = 0.094), suggesting that the in-school component of the program had the effect of increasing the grade promotion rate by almost 6 percent. When assessed for the final follow-up, about 43 to 48 percent of the students in each treatment group were retained in earlier grades; that is, they were not enrolled in the 3rd grade (Table II.6). The rates of grade repetition we found for the evaluation sample are higher than El Quiché’s grade repetition rate in elementary school of 15 percent in 2012 (Instituto Nacional de Estadística 2017), and higher than the prevalence of overage students in elementary school (students who are at least two years older than the official age of entry in a given grade) equal to 19 percent in Guatemala in 2015 (UNESCO 2017). We also found that at the time of the endline reading skills assessment, most children in the sample attended the same school they attended at the time of the baseline data collection. Therefore, most children in the treatment groups had been enrolled in a school that implemented the in-school component of the intervention for about three years. We did not find any statistically significant differences across treatment groups in the school that children in the evaluation sample attended at the time of the endline data collection. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 22 Table II.6. Grade and school at the time of the final follow-up data collection (percentage of students) Grade and school Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) In 3rd grade 54.8 57.3 51.4 -2.5 (0.473) 5.9 (0.094) Current school is the same as the school attended at the time of the evaluation’s baseline data collection 91.9 94.6 93.9 -2.6 (0.128) 0.7 (0.748) Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. G. Baseline equivalence of the analysis sample The random assignment process yielded similar treatment groups in most of the baseline characteristics examined. Because the evaluation is based on an experimental design (schools were randomly assigned to the treatment groups), we did not expect to find systematic differences across all the baseline characteristics of the treatment groups that we measured. However, differences in some characteristics could exist because it may not be possible to assign 150 schools to three treatment groups such that the groups are identical. Random assignment facilitates ensuring the three groups are, on average, similar and that any unusual characteristic has an equal chance of representation in any of the three groups. We examined differences across treatment groups in the characteristics of students in the analytic sample that are unlikely to be influenced by the program and therefore can indicate whether our evaluation design produced similar treatment groups. As we show in Table II.7, the treatment groups are similar in all student baseline characteristics, with two exceptions: children in schools in Group B had, on average, slightly higher scores in the baseline assessment of letter￾identification skills than children in schools in Group A, and children in Group C were, on average, two months older that children in Group B at the time of the baseline assessment. To account for the baseline characteristics of the evaluation participants, we included the variables presented in Table II.7, among other student- and school-level variables, as controls in the regression models we used to calculate program effects. Table II.7. Baseline characteristics of children in the evaluation sample Characteristic Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Average age at baseline (in months) 93.0 91.7 94.0 1.3 (0.137) -2.3* (0.010) Female (percentage) 49.7 50.2 52.4 -0.5 (0.747) -2.2 (0.189) Baseline letter identification score (0 to 24) 16.8 17.6 18.0 -0.8* (0.047) -0.4 (0.297) EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 23 Characteristic Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Baseline emergent reading score (0 to 9) 2.2 2.3 2.5 -0.0 (0.883) -0.2 (0.397) Baseline emergent writing score (0 to 22) 8.1 8.6 8.8 -0.5 (0.366) -0.2 (0.744) Baseline phonemic awareness score (0 to 10) 3.3 3.6 3.3 -0.2 (0.475) 0.2 (0.453) Baseline number of pseudo-words read in one minute (0 to 50) 3.1 3.8 3.4 -0.7 (0.133) 0.4 (0.373) Baseline passage listening comprehension score (0 to 1) 0.7 0.7 0.8 -0.0 (0.453) -0.1 (0.191) Child assessed in Spanish only at baseline 12.6 8.6 7.1 4.0. (0.197) 1.6 (0.575) Child assessed in K’iche’ only at baseline 65.1 68.0 66.9 -2.9 (0.581) 1.1 (0.822) Child attended preschool or kindergarten 56.1 58.7 55.1 -2.6 (0.604) 3.7 (0.461) Child’s mother completed at least one year of formal schooling 31.0 37.0 33.4 -6.0 (0.090) 3.6 (0.241) Child’s parents can read 35.4 39.0 36.2 -3.6 (0.314) 2.7 (0.362) Language spoken at home is K’iche’ only 76.2 80.7 78.1 -4.5 (0.135) 2.6 (0.346) Number of childrena 439 458 441 Source: Leer Juntos, Aprender Juntos Students’ Emergent Literacy Skills Assessment—Baseline 2013 and 2016, and Household Survey—Midline 2014 and 2015. Note: Regression-adjusted means. P-values in parentheses. a Number of children who completed an endline reading skills assessment and who have complete data on age and sex. The number of children in the analytic sample who completed a baseline assessment is 1,311 (428 in Group A, 448 in Group B, and 435 in Group C). * Difference in group means is statistically significant at the 0.05 level. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 25 III. WAS LEER JUNTOS, APRENDER JUNTOS IMPLEMENTED AS INTENDED? As we described in Chapter I, the Ministry of Education in Guatemala has been implementing the Leamos Juntos program since 2012. This initiative aims to improve the quality of education and student achievement in reading through activities similar to those carried out by Leer Juntos, Aprender Juntos, such as training teachers and distributing reading materials in Spanish and in mother tongues. Therefore, implementation of Leer Juntos, Aprender Juntos occurred in a setting where several practices and programs aiming to improve early-grade literacy coexisted across the treatment groups. This chapter presents findings on how Leer Juntos, Aprender Juntos services were delivered, how services were perceived by stakeholders, and what services were received in each evaluation group. It also presents findings on challenges to implementation. These implementation findings help to contextualize and explain impact findings. A. School participation in education interventions The practices and programs that coexisted with Leer Juntos, Aprender Juntos were present in equal measure in all three experimental groups. This finding is based on reports by principals (Table III.1) and teachers (Appendix E). Most schools (66 to 70 percent in each treatment group) reported participating in Leamos Juntos, the national reading program. The percentage of schools participating in the Intercultural Bilingual Education and Literacy Program ranged from 6 to 10 percent across treatment groups; participation in the Education Support and Technical Assistance Program in the Department of El Quiché ranged from 28 to 38 percent across treatment groups; and participation in other education programs provided by the Ministry of Education ranged from 12 to 20 percent across treatment groups. The differences in participation between treatment groups represented a small number of schools and were not statistically significant. School principals in both Groups A and B reported higher participation in the Leer Juntos, Aprender Juntos program, as expected (Table III.1). The reported participation was not close to 100 percent of the schools in Groups A and B, because at the time of the endline data collection in Phase II schools (in 2016), the activities of the Leer Juntos, Aprender Juntos program were no longer offered to any schools by Save the Children. Table III.1 also indicates that 9 percent of the principals in the prevailing practice group reported participating in Leer Juntos, Aprender Juntos, whereas none of the teachers in those same prevailing practice schools reported receiving any component of Leer Juntos, Aprender Juntos. We believe this reflects differences in respondents’ knowledge about the programs that are actually being implemented in their schools. School directors may be less familiar than teachers with the program activities being rolled out in classrooms. Although school leaders may be more aware of the general activities occurring in the school, they may be less informed about the specific teacher training activities or the identity of the implementer, program name, or funding source for those activities. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 26 Table III.1. School participation in education and social programs, reported by principals or other school administrators Program. Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Percentage of schools participating in the following education-focused programs Leer Juntos, Aprender Juntos 52.0 54.0 9.1 -2.0 (0.744) 44.9* (0.000) National Reading Program (Leamos Juntos) 70.0 66.0 70.0 4.0 (0.680) -4.0 (0.679) Intercultural Bilingual Education and Literacy Program 6.0 10.0 8.0 -4.0 (0.468) 2.0 (0.718) Other Ministry of Education programs 12.0 20.0 14.0 -8.0 (0.265) 6.0 (0.403) Education Support and Technical Assistance Program in the Department of El Quiché (Programa Verde y Azul) 38.0 28.0 31.0 10.0 (0.151) -3.0 (0.672) Percentage of schools participating in the following health-focused and other programs Health promotion programs 44.0 40.0 36.9 4.0 (0.678) 3.1 (0.749) National Children’s Nutrition Program 10.0 14.0 28.3 -4.0 (0.587) -14.3 (0.056) Other programs 18.0 18.0 22.1 0.0 (1.000) -4.1 (0.594) Number of schools 50 50 49 . . Source: Leer Juntos, Aprender Juntos School Infrastructure Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. B. Implementation challenges in Guatemala The implementation of Leer, Juntos, Aprender Juntos occurred in a climate of heightened attention to education quality, which has resulted in public and nongovernmental efforts aimed at improving the quality of education in the region. Multiple and diverse efforts from regional educational authorities and nongovernmental organizations sought to improve students’ achievement and grade progression, particularly in the early grades. During the period of implementation and evaluation of Leer Juntos, Aprender Juntos, the education authority for El Quiché, known as DIDEDUC, ran an initiative to improve 1st grade promotion and prevent grade repetition and dropout in the Department of El Quiché. The initiative included awareness-raising meetings with parents, teacher training and coaching, and active monitoring of school performance indicators. Even as Leer Juntos, Aprender Juntos began its activities, stakeholders at DIDEDUC were already observing a positive trend in their performance indicators; for instance, they reported reductions in school failure rates for 1st grade (dropout and grade repetition) from 36 percent in 2012 to 30 percent in 2014 (DIDEDUC 2014a). The education improvement efforts at the department level were implemented in parallel with the roll out of Leer Juntos, Aprender Juntos, providing additional community, school, and classroom level activities, as follows: EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 27 ● Departmental Reading Committee. This committee was organized to improve efficiency in the delivery of trainings and materials to promote reading and strengthen reading instruction. ● Educational Network.13 This network of organizations offers services to improve the quality of education in the Department of El Quiché. DIDEDUC began collaborating with the Educational Network since 2013, and by 2015, the network included more than 12 organizations providing a range of supports. For example, Verde y Azul provided pedagogical materials, school furniture, and teacher training. Proyecto de Desarrollo Santiago (PRODESSA) provided teacher training and coaching and pedagogical materials. School the World offered school libraries, and supported school infrastructure. Corazones y Manos offered teacher training and coaching focused on reading instruction. A full list of network members, services and target municipalities is available in Appendix E, Table E.1. ● Sponsorship strategy14. This strategy was initiated in July 2014 in collaboration with the Educational Network and DIDEDUC’s Technical-Pedagogical Coordinators15 to improve the quality of education in the Department of El Quiché. It aimed to monitor students at risk for academic failure, to design methods to support students at risk, to reduce 1st grade repetition by 5 percent, and to promote progression from 1st to 2nd grade. The program designated “sponsors,” who were regional staff from the Ministry of Education and who had been recognized for their leadership skills, knowledge of the region, and commitment with DIDEDUC’s strategy. Each “sponsor” was tasked with monitoring and supporting 1st grade teachers, conducting awareness-raising meetings with parents, visiting schools to provide assistance, information gathering, and tracking performance indicators at the school level (DIDEDUC 2014b). ● Plan Lector and provision of library books. This program aimed to provide reading books for primary grades across all municipalities in the Department of El Quiché. In addition, it instituted a compulsory 30-minute-per-day reading practice period for early primary grades. ● Other actions. Additional actions included meetings with parents to promote school enrollment, enforcement of student performance criteria for grade promotion, and mapping availability of educational programs in the Department of El Quiche for an equitable allocation of resources. C. Implementation of the in-school component Teachers described the Leer Juntos, Aprender Juntos methodology as practical and feasible to apply in the classroom and cited the complementary relationship between the program’s instructional approach and the Ministry of Education’s national curriculum. The programs’ training methodology was based on experiential learning and applied knowledge; teachers who participated in focus groups expressed that the concrete tools and strategies they learned in training were feasible to use in their classroom and led to improved student 13 This network of organizations is referred to as Red Educativa. 14 This strategy is known in the Department of El Quiché as Programa de Apadrinamiento. 15 Or coordinadores técnico pedagógicos. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 28 engagement. Teachers liked activities such as echo reading, timed reading syllabic segmentation, and alphabet soups, and used them readily in class. However, other types of activities required a steeper learning curve and teachers found them more challenging to use with their students, particularly in multigrade classrooms. During training sessions, specialists noticed that teachers needed applied examples to improve their understanding of reading instruction strategies and incorporate those examples into their training materials. Beginning teachers found Leer Juntos, Aprender Juntos teaching strategies and pedagogic activities particularly helpful; those who participated in focus groups reported the program helped them transition from theory to practice. New teachers felt that Save the Children’s training afforded them with strategies for emergent reading instruction such as alphabetic principle, identification of initial letter sounds, echo reading, and word generation. In focus groups, teachers discussed how the strategies and activities they learned in training prompted them to change the way they taught reading—moving away from traditional teacher￾centered practices toward more student-centered approaches. Take-up of reading-instruction strategies and activities was particularly challenging for teachers who lacked mother tongue skills, and for those who taught in multigrade classrooms. Teachers who did not have mother tongue skills faced numerous challenges with reading instruction in the early primary grades. They faced greater challenges implementing activities that required instruction in mother tongue. Variation in teachers’ mother tongue skills resulted in different adaptations of mother tongue instruction in the classroom, ranging from translation to full immersion. In focus groups, we learned that teachers from intervention schools who were not fully bilingual in K’iche’ were less able to use mother tongue for reading instruction. This is consistent with survey findings showing that less than half of the teachers in the evaluation sample reported having good to very good oral skills in K’iche’, and less than 20 percent of the teachers in each treatment group reported having advanced written skills in K’iche’ (see Table II.5). “Before, my way of working was traditional, and I was the center of the class. I did not take students into account. But now, through the techniques they have given us, children are very involved ... they participate more” –Phase I teacher, Chichicastenango Some teachers with limited K’iche’ skills made efforts to learn K’iche’ from their bilingual students in class but still felt their mother tongue competency was insufficient to incorporate mother tongue reading instruction as expected. Further, in communities where there is little or no access to pre-primary education, teachers felt that 1st grade students overwhelmingly lacked pre-literacy skills. This factor represented an additional challenge for reading instruction, particularly when 1st grade teachers felt they lacked adequate mother tongue skills to help students transition from the home to the school’s communicative demands. Teachers are aware of the needs and benefits of using K’iche’ in their “My appointment is bilingual, but I'm not bilingual. So yes, it was difficult for me ... I have learned simple [K’iche’] words with children ... when children do not understand me [in Spanish], there is a child that translates in class”— Phase II teacher, Zacualpa EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 29 classrooms, and some school directors made efforts to assign bilingual teachers to 1st, 2nd, and 3rd grade classrooms. However, as we learned from our qualitative data, not all school directors were willing or able to prioritize bilingual skills when placing teachers in early-grade classrooms that had a substantial number of students dominant in K’iche’. A number of teachers also reported experiencing challenges using reading instruction strategies and activities in multigrade classrooms, particularly when classes were large or when the grades where not consecutive. Other multigrade teachers, however, found opportunities for peer-learning, grouping students based on their reading skills, and reported that such reading activities were engaging and well received by students. Similarly, Leer Juntos, Aprender Juntos activities, such as reading buddies (Amigos de la Lectura) and the teacher learning circles, sparked collaboration among teachers of different grade levels. They conducted reading activities jointly, giving students new opportunities to practice and improve their reading skills through peer-learning with students from other classrooms. The individualized coaching sessions provided to teachers made it possible to support teachers’ needs in the classroom, enrich their teaching skills, and identify areas for improvement. During coaching visits, program specialists visited and observed teachers’ practices onsite, assessed the extent to which the program’s techniques and strategies were being implemented as intended, checked the quality of the literacy environment in the classrooms, demonstrated teaching strategies, and identified areas of improvement in teachers’ practice. Specialists used information obtained during coaching visits to inform priorities for future coaching sessions. Teachers who participated in focus groups stated that, in geographically isolated municipalities such as San Antonio Ilotenango and San Andrés de Sajcabajá, teachers received less frequent support from the DDE and had less experience with coaching sessions or technical support outside the Leer Juntos, Aprender Juntos program. Program specialists also noticed that teachers in intervention schools located in geographically isolated municipalities had more challenges implementing the program’s strategies and activities and were more likely to feel monitored or supervised during the coaching visits. Based on focus groups, teachers had mixed opinions about their experience with coaching visits depending on the knowledge, skills, and longevity of the coach. Some teachers were very pleased with the types of active and hands-on coaching and effective feedback they received, and others felt they received little feedback and less support than their peer teachers did. Teachers discussed the varying levels of experience and coaching skills among program specialists and turnover among specialists, which was disruptive to teachers; changing specialists over the course of the project created challenges, because effective coaching requires trust and relationship-building. Teachers who reported to be satisfied with this program component were coached by more skilled, knowledgeable, and long-lasting specialists. The local implementation team contributed innovative project materials for teachers, students, and volunteers. The local implementation team in the region created new reading materials for students incorporating Guatemalan indigenous languages and cultural themes. Specialists produced original materials in K’iche’ with useful culturally relevant examples. Also, the program provided books and stories in Spanish and K’iche’ and customized three types of support materials: (1) instruction guides for teachers on how to implement the methodology, (2) EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 30 guides for volunteers on how to support the activities in the community, and (3) portfolios for parents on how to help their children with reading at home. Stakeholders valued these materials, which reflected program investments based on local needs. Teachers in intervention schools who participated in focus groups shared that program materials were helpful for classroom instruction and highlighted their longtime struggle with access to sufficient pedagogical materials for reading instruction. In many cases, teachers reported that they did not receive teaching materials from the Ministry of Education for language and communication, and they felt that the program filled that need to some extent. Further, teachers in intervention schools learned how to make low-cost materials for reading instruction in creative ways and with resources readily available in their communities. High teacher turnover and rotation of teachers to upper grades made it difficult to deepen teachers’ content knowledge over time and build the skills of teacher cohorts from year to year. Annual contract renewals and teacher rotation to grade levels other than 1st, 2nd, or 3rd, even when they stayed in the same school, were barriers to strengthening the program’s reading instruction strategies in the classroom. The implementation team made efforts to engage educational authorities and school principals with the goals of the program and emphasized the importance of keeping the cohorts of trained teachers assigned to the early elementary grades (1 through 3), but grade-level assignments were beyond their control, and teacher mobility could not be prevented. From interviews and focus groups with stakeholders, we learned that some treatment classrooms had teachers who had not participated in the Leer Juntos, Aprender Juntos training workshops. At midline, teacher survey data showed that about 6 percent of the 2nd grade teachers interviewed in Group A and 8 percent in Group B did not report having received training from Leer Juntos, Aprender Juntos (Lugo-Gil et al. 2017b). At endline, only about half of the 3rd grade teachers surveyed in intervention schools reported having received training from the program that year (50 percent in Group A and 48 percent in Group B). In addition, the implementation team faced challenges with the length of the nine training modules, and the DIDEDUC’s mandate that teacher trainings should not interfere with the required 180 days of class per year. Program staff identified two areas of improvement for the implementation of Leer Juntos, Aprender Juntos: better use of formative assessment and more work on bilingual materials in the classroom. First, teachers’ understanding and regular use of formative assessments for reading instruction could improve substantially. The program’s formative assessment strategy and the use of assessment results to inform reading instruction was challenging for many teachers. Teachers typically relied on summative assessments and struggled to understand the proper use and benefits of formative assessments to measure students’ progress in the core areas of reading acquisition. Teachers in intervention schools felt formative assessments were too time-consuming, and they were unsure about how to use the results. Program staff elaborated additional guidelines to help teachers understand how to use formative assessments, but despite those efforts, program staff felt teachers’ formative "It has motivated us to make our own materials without the need to spend money, instead using what’s available in the environment of the child and around the community…The program has taught me to be more productive, and take advantage of all the things that are within reach.” — Phase I teacher, Chichicastenango EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 31 assessments did not quite work in this context as intended by the program. Second, program staff felt additional work was needed to help teachers understand how to create a print-rich bilingual environment in the classroom, in K’iche’ to the same degree as in Spanish. D. Implementation of the community action component Community reading activities usually took place on school premises, after regular school hours, and were led by program specialists with support from community volunteers. Reading activities usually took place in the hallways, classrooms, the schoolyard, or an open field near the school. Children usually sat on mats on the floor or on the ground. Activities were typically led by a program specialist (occasionally two or more, depending on the size of the group), with support from a community volunteer, and were conducted in both Spanish and K’iche’. Take-up of community activities was difficult at the outset and required additional engagement with leaders from the community development councils and with school principals. Through interviews with Save the Children’s implementation team we learned that, to increase recruitment of volunteers and participation in community activities, the local implementation team conducted communitywide informational meetings with support from the presidents of the community development councils, the school directors, and program specialists and supervisors. Endorsement of the program by community authorities facilitated parents’ buy￾in and participation in community activities. Recruitment of community volunteers was a major challenge to implementation. It required adjustments during the rollout of the community action component because such volunteering was not common practice in the region, and the original recruitment criteria for volunteers were not feasible to fulfill. The type of volunteering that the community action component required was not common in the region. Although parents often collaborated with meal preparation in schools, unpaid help outside regular school hours was uncommon. The program struggled with an insufficient number of volunteers to implement community activities, irregular attendance of volunteers, and high volunteer turnover. For instance, we learned through program staff interviews and quarterly reports that about 50 percent of volunteers were no longer participating after the first year of program implementation. Some of the barriers for volunteering were lack of monetary or in-kind incentives, household responsibilities, kin-care, need for paid labor, and lack of time. These barriers seemed to be more pronounced in some municipalities, such as Santo Tomás Chichicastenango, where other organizations offered incentives to volunteers in their programs. Volunteers who participated in focus groups shared that they were hesitant to join the team, because they initially felt they should receive something in exchange for their time and effort. Over time, volunteers learned that volunteering was intrinsically rewarding, and that it helped them develop valuable life skills such as leadership and public speaking. “At the beginning of the project, I did not want to accept this responsibility, I thought I should receive something in return…but now I have the satisfaction of being able to teach children”. — Volunteer, phase I, Santa Cruz EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 32 In response to the challenge of recruiting volunteers, Save the Children modified the target profile for volunteers. Initially, recruitment criteria required secondary education; but with shortage of volunteers, criteria were made more flexible, and anyone who knew how to read and write could qualify as a volunteer, regardless of his or her level of education. Implementers also targeted middle-grade and university students as volunteers. Upper middle school students volunteered during their class time, with permission from their teachers. Students could participate as long as they caught up with class notes or exams afterward. In addition, implementers sought ways to foster in volunteers a sense of belonging to the project team and communitywide recognition of their role. They gave each volunteer a welcome kit, a project shirt, a hat, and a sweater with the organization’s logo. Challenges with the recruitment and retention of volunteers resulted in the community action component’s implementation lagging the in-school component’s implementation. As discussed earlier, the recruitment and retention of community volunteers was challenging and hindered the implementation of community action activities as initially expected. Further, the elections and campaign activities in August and September of 2015 led to political unrest in the region and altered the implementation of the community action activities. Areas of improvement in the community action component included fostering habits of returning materials borrowed from the book banks, having a mechanism to check whether children had actually read and understood the books they borrowed, strengthening the skills of community volunteers to lead activities and ensuring that they implement activities consistently, and improving adults’ participation in community activities. Attendance at community activities fluctuated seasonally, and there were context and environmental barriers to children’s participation in after school reading activities. Student and volunteer attendance fell during school breaks, periods of seasonal agricultural activities, and local market days. Inclement weather, typically from May through July, also interfered with community activities and reduced children’s attendance. “I’m also a student... we leave school to teach the children ... The teacher supports us, and we have authorized those hours so we can leave the classes, and … after we finish, we return to the classes”. —Volunteer, phase I, San Antonio Community activities did not reach all parents and children for whom they were made available. Household survey findings showed that, at midline, 34 to 57 percent of the surveyed parents in Group A reported they had been invited to participate in reading activities in their community such as reading camps, story hour, reading buddies, or reading fairs, and about 26 to 45 percent of the parents in that group had actually participated in those community activities (Table III.2). Additionally, as shown in Table III.2, about 15 percent of the parents in Group A reported having been invited to borrow books, and a similar percentage had actually borrowed books. Most parents who participated in focus groups understood that the community activities sought to improve children’s reading skills, although a few were unaware that these offerings were available in their community. Word of mouth from parents whose children were attending community activities and recognized them as fun and enjoyable learning opportunities prompted “I would like to ask… when did the program start? Because I have not heard about [it]. No one has told us, not even at the school.” — Parent, Santa Cruz EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 33 other children to attend. However, information about the community activities and availability of book banks in the community failed to reach some parents of children in Group A schools. Table III.2. Parents’ reports on children’s exposure to and participation in reading activities in the community Blank cell Leer Juntos (A) Leer Juntos– schoolonly (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Invited to participate in the following activities (percentages): Reading camps 57.4 1.4 0.1 56.0* (0.000) 1.3 (0.526) Story hour 53.2 1.2 0.0 52.0* (0.000) 1.2 (0.381) Reading buddies (tutoring and reading games with older children) 51.5 0.7 0.0 50.8* (0.000) 0.7 (0.553) Reading fairs 34.0 1.6 0.0 32.4* (0.000) 1.6 (0.336) Borrowing books from a book bank or a library 15.7 1.5 2.5 14.2* (0.000) −1.0 (0.531) Participated in the following activities (percentages): Reading camps 44.8 0.9 0.0 43.9* (0.000) 0.9 (0.526) Story hour 42.5 0.8 0.0 41.7* (0.000) 0.8 (0.381) Reading buddies (tutoring and reading games with older children) 39.4 0.3 0.0 39.1* (0.000) 0.3 (0.553) Reading fairs 25.5 1.2 0.0 24.3* (0.000) 1.2 (0.336) Borrowing books from a book bank or a library 15.1 1.7 1.5 13.4* (0.000) 0.2 (0.914) Number of childrena 452 468 470 . . Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Regression-adjusted means. P-values in parentheses. a Number of children for whom a household survey was completed. *Difference in group means is statistically significant at the 0.05 level. The most prevalent barriers to children’s participation were long distances from home to the location of the community activities, and risks in children’s travel to reading activities after school. Although many families live close to schools and find the location of the community activities accessible, others live farther away and felt roads were unsafe for children to walk on their own later in the afternoons. Practically all of the families in the sample (at least 98 percent of the families in each treatment group) reported that they walk their children to school, which takes an average of 16 to 17 minutes. Recognizing the importance of parents’ involvement in children’s reading, implementers began conducting home visits in addition to the parent workshops and reading activities for students during the third year of the program. The goal of home visits was to help parents apply what they learned in the workshops and encourage them to send children to the reading activities in the community. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 35 IV. WHAT ARE THE IMPACTS ON INSTRUCTIONAL PRACTICES AND LITERACY ENVIRONMENT? As indicated in the program’s logic framework presented in Chapter II (Figure II.2), we expected that participation in the in-school component of the program would result in trained teachers who use the strategies learned in the training effectively, track children’s reading achievement with formative assessments, and create a print-rich classroom environment where reading materials are available in mother tongue and in Spanish. We also expected that the community action component of the program would provide children with more opportunities to practice reading outside of school hours and help parents support their children’s reading. The teacher survey and classroom observation data provide evidence that the children in the schools implementing the in-school component of the intervention (Group B) experienced improved classroom environments when compared with the children in the schools implementing the prevailing practice (Group C). This is consistent with reports from intervention teachers in focus groups (see Chapter III): that teachers from the in-school component of the intervention (Group B) felt motivated to create new pedagogical materials and more effectively use the materials already available in their classrooms. Regarding children’s activities outside of school hours, our findings from the analysis of household survey data are consistent too with the findings from the analysis of qualitative information collected. In particular, there was no pattern of impacts of the community action component on children’s reading activities at home. This finding reflects the challenges experienced in recruiting and retaining volunteers for leading the community action component activities and the barriers faced to participation. A. Classroom instruction 1. Teacher participation in professional-development activities The Leer Juntos, Aprender Juntos intervention significantly increased the percentage of teachers receiving coaching. Teachers in the treatment groups (Groups A and B) reported that they actively participated in professional development activities focused on reading instruction. More teachers in the Leer Juntos, Aprender Juntos intervention groups reported receiving training and coaching than did teachers in the prevailing practice group (Group C) during the calendar year when the teacher survey was administered (2015 for Phase I teachers and 2016 for Phase II teachers). We expected this finding, given that one of the components of the intervention focused on providing teachers with training and coaching. The differences in training rates (62 percent in Group B versus 29 percent in Group C) and coaching rates (50 versus 20 percent, respectively) between Groups B and C were statistically significant (see Figure IV.1). In terms of the sources for training and coaching, essentially all Group C teachers reported that the Ministry of Education provided their training and coaching. Teachers in Groups A and B reported that their training and coaching were provided by a combination of Save the Children (for Leer Juntos, Aprender Juntos) and the Ministry of Education (see Appendix E, Tables E.2 through E.4 and Figure E.1). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 36 Figure IV.1. Teacher-reported participation in reading instruction training and coaching in the endline year Source: Leer Juntos, Aprender Juntos Teacher Survey—Endline 2015 and 2016. * Difference in group means is statistically significant at the 0.05 level. 2. Teacher-reported and observed practices We measured classroom practices using both teacher reports and live classroom observations. In the endline teacher survey, the evaluation team asked teachers about the methods, strategies, materials, and allocation of instruction time they use to teach reading, and about the needs and challenges they face in teaching their students to read. The evaluation team also conducted structured observations of 3rd grade classrooms during a typical language or reading class to measure the proportion of observed instruction time that teachers allocated to literacy instruction activities. Observers noted features of the print environment and recorded teachers’ and students’ use of mother tongue and Spanish language in the classroom. Observers also completed a checklist on teachers’ use of instructional practices focused on teaching the five foundational reading skills targeted in the program’s training (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension) and of general practices such as providing feedback to students. We found some statistically significant program effects on the instructional practices that teachers reported, but there was not a meaningful pattern of effects. We found statistically significant differences between Groups A and B in the percentage of teachers who reported using a “guided reading” approach, and the percentage of teachers who reported using a “constructivism” approach, to teach reading. We also found three statistically significant differences between Groups A and B, and four statistically significant differences between Groups B and C, in the percentage of teachers who reported using strategies to teach reading fluency, new words, and reading comprehension. As we show in Table IV.1, 46 percent of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 37 teachers in Group B, compared with 14 percent of teachers in Group A and 30 percent of teachers in Group C, reported using a “guided reading” approach to teach reading skills. These differences between treatment groups are statistically significant (at the 0.05 level for the difference between Groups A and B, and at the 0.10 level for the difference between Groups B and C). In the teacher survey that we administered in the final follow-up of the evaluation, we asked teachers open-ended questions about the methods or approaches they used more frequently to teach reading to their students in 3rd grade. That is, teachers were free to describe their approaches to teach reading in their own words. The survey did not offer “guided reading” as a response; rather, in constructing variables we used “guided reading” to categorize any reported approach that helps children read through text. Therefore, differences in the reports from teachers in Groups A and B about using “guided reading” may not mean that the teachers in these two groups were using completely different approaches to teach reading. This finding indicates, however, that more teachers in Group B than in Group A referred to helping children read text when they teach reading. Because teachers in Groups A and B received the same training and coaching from the intervention, we would expect them to report using similar methods and/or approaches to teach reading. More teachers in Group A (40 percent) than in Groups B (22 percent) reported using a constructivism approach to teaching reading. As shown in (Table IV.1), constructivism is a widely used pedagogical method, based on principles proposed by Jean Piaget and Lev Vygotsky, that focuses on giving students tools to build their own learning process. The Leer Juntos, Aprender Juntos program did not emphasize any particular pedagogical approach. We speculate that teachers in schools in Group A who had recently received information on constructivism may have viewed the method as similar to their teaching approach. We also found statistically significant differences between treatment groups in strategies that teachers use to teach fluency, new words, and reading comprehension. As we show in Table V.1, with respect to fluency strategies, more teachers in Group B than Groups A and C reported that they instruct their students to read with a chronometer. Also, more teachers in Group B than in Group C reported using the strategies of reading aloud in groups of pairs, guided oral reading, and reading in chain. Reading with a chronometer and reading aloud are instruction strategies emphasized in the teacher training provided by Leer Juntos, Aprender Juntos. Therefore, it is not surprising that more teachers in Group B (teachers who received the intervention’s training) reported using those strategies than teachers in Group C (teachers who did not receive the training). More teachers in Group A than in Group B used synonyms and antonyms to teach new words, and more teachers in Group B than in Group A asked questions about reading to their students to teach reading comprehension (Table IV.1). These differences were statistically significant. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 38 Table IV.1. Teacher-reported methods and strategies to teach reading Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Percentage of teachers who reported using the following methods or approaches to teach reading: Syllabic 12.0 12.0 4.0 0.0 (1.000) 8.0 (0.176) Phonological/global 2.0 4.0 6.0 -2.0 (0.620) -2.0 (0.620) Guided reading 14.0 46.0 30.0 -32.0* (0.000) 16.0 (0.066) Constructivism 40.0 22.0 20.0 18.0* (0.042) 2.0 (0.820) Strategies for reading comprehension 14.0 4.0 8.0 10.0 (0.066) -4.0 (0.459) Percentage of teachers who reported using the following strategies to teach: Reading fluency Reading with chronometer 22.0 38.0 14.0 -16.0* (0.045) 24.0* (0.003) Reading aloud in groups or pairs 40.0 36.0 16.0 4.0 (0.656) 20.0* (0.028) Guided oral reading 28.0 44.0 20.0 -16.0 (0.069) 24.0* (0.007) Reading in chain 12.0 20.0 2.0 -8.0 (0.194) 18.0* (0.004) Silent reading 8.0 10.0 6.0 -2.0 (0.715) 4.0 (0.465) Strategies for reading comprehension 24.0 8.0 24.0 16.0 (0.059) -16.0 (0.059) New words Discovering the name of things using the dictionary 60.0 70.0 70.0 -10.0 (0.242) 0.0 (1.000) Synonyms and antonyms 12.0 2.0 6.0 10.0* (0.042) -4.0 (0.411) Building sentences and phrases with word lists 12.0 6.0 8.0 6.0 (0.293) -2.0 (0.726) Reading comprehension Asking questions about the reading 30.0 62.0 52.0 -32.0* (0.002) 10.0 (0.334) Identifying the sequence of events 4.0 2.0 0.0 2.0 (0.481) 2.0 (0.481) Predictions about history or pre-reading reflections 0.0 4.0 2.0 -4.0 (0.160) 2.0 (0.481) Abstracts, paraphrasing, or storytelling 26.0 22.0 18.0 4.0 (0.640) 4.0 (0.640) Identifying characters, problems, and solutions 10.0 2.0 12.0 8.0 (0.157) -10.0 (0.078) Number of teachers 50 50 49 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 39 We did not find any statistically significant program effects on the instructional practices observed in the classroom. The findings from teacher reports presented in Table IV.1 contrast with the findings from classroom observations. Whereas the teacher reports indicated differences between the treatment groups, the classroom observations did not yield statistically significant differences between treatment groups on how teachers spent the time during the language or reading lesson that we observed. Across all treatment groups, teachers spent most of the observed literacy instruction time on reading texts and working with the alphabet and written words to cultivate phonemic awareness and alphabet knowledge. As Figure IV.2 shows, teachers in all treatment groups spent 33 to 37 percent of the observed time on activities that involved reading text, such as reading aloud or silently, in groups or individually. The observed teachers also spent about one-quarter of the observed time working with the alphabet and written words to cultivate phonemic awareness and alphabet knowledge. Figure IV.2. Observed allocation of instruction time in a typical language or reading class Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: None of the differences between treatment groups presented in this figure is statistically significant at the 0.05 level. Teachers in the three treatment groups were also similar in terms of their use of instructional practices focused on teaching five foundational literacy skills, as we show in Table IV.2. In the evaluation’s classroom observations, observers recorded whether teachers used instructional practices that teach the five foundational literacy skills that were the focus of the teacher training provided by Leer Juntos, Aprender Juntos: (1) alphabet/letter knowledge, (2) vocabulary, (3) phonemics and phonological knowledge, (4) fluency, and (5) reading comprehension. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 40 Table IV.2. Observed instructional practices to teach literacy skills Practice Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Percentage of classrooms where teachers used the following practices to teach: Vocabulary Introduced or rehearsed vocabulary words and explained word meaning or elaborated on new concepts 82.0 84.0 81.0 -2.0 (0.786) 3.0 (0.683) Fluency 32.0 28.0 18.8 4.0 (0.656) 9.2 (0.308) Letter knowledge Encouraged students to recognize letters, identify the names of letters, or distinguish upper and lower case 20.0 20.0 20.0 0.0 (1.000) 0.0 (1.000) Phonemics and phonological awareness Provided instruction on phonemic awareness 2.0 6.0 10.1 -4.0 (0.356) -4.1 (0.345) Provided instruction on the correct pronunciation of words in K’iche’ or Spanish 4.0 2.0 2.0 2.0 (0.547) 0.0 (0.995) Prompted students to segment words into specific units (syllables or phonemes) 20.0 12.0 11.9 8.0 (0.267) 0.1 (0.991) Provided instruction on grammar, mechanics, or spelling 24.0 26.0 28.1 -2.0 (0.829) -2.1 (0.825) Provided instruction on word writing 86.0 86.0 96.2 0.0 (1.000) -10.2 (0.075) Book reading and reading comprehension Pre-reading or book-preview 0.0 4.0 8.1 -4.0 (0.317) -4.1 (0.306) Encouraged predictions 0.0 2.0 4.1 -2.0 (0.483) -2.1 (0.473) Explored children’s interest in the story and facilitated connections with their lives 4.0 0.0 2.0 4.0 (0.163) -2.0 (0.487) Explained the uses of different types of texts and written materials 2.0 2.0 0.0 0.0 (1.000) 2.0 (0.387) Encouraged students to use context clues for reading comprehension 4.0 2.0 2.0 2.0 (0.547) 0.0 (0.995) Discussed text structure 0.0 4.0 0.0 -4.0 (0.085) 4.0 (0.084) Discussed the characters in the text, who they are, their motivation and/or goals 4.0 2.0 6.1 2.0 (0.622) -4.1 (0.321) Asked questions about events in history, encourages predictions or inferences 38.0 26.0 30.0 12.0 (0.214) -4.0 (0.683) Helped students understand the text by making predictions, applying prior knowledge 42.0 26.0 30.9 16.0 (0.070) -4.9 (0.576) Used other tools in addition to books for the practice of reading skills 8.0 10.0 3.9 -2.0 (0.715) 6.1 (0.270) Reinforced the pedagogical objective with playful or motivational activities 18.0 18.0 9.8 0.0 (1.000) 8.2 (0.261) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 41 Most teachers in all treatment groups were observed providing instruction on vocabulary and word writing. In contrast, the percentage of teachers observed providing instruction on letter knowledge, phonemic awareness, pronunciation, and segment of words was 20 percent or less in all treatment groups. We found no statistically significant difference between treatment groups in the teachers’ use of instructional practices focused on teaching the five foundational skills (Table IV.2). The finding of no statistically significant differences between treatment groups in the observed teaching practices may reflect the challenges that the program implementation team experienced in delivering the program. As we discus in Chapter III, the implementation team had no control over decisions to move teachers from the evaluation’s target grades (grades 1 to 3) to non-target grade levels (grades 4 to 6). This factor made it challenging for teachers in the intervention schools (Groups A and B) to implement the instruction strategies they learned as part of the intervention. Also, the delivery of training had to be adapted so that it did not interfere with the teachers’ work schedules (see Chapter III). Because the training was compressed, and teachers did not have as much time as originally envisioned to practice new techniques, they may have been unable to fully incorporate those new strategies into their instructional practices. We present additional findings on teacher-reported and observed practices in Appendix E, Tables E.5 through E.10. B. Reading environment and behaviors 1. Environment in the classroom Classroom observers recorded the presence of a reading area in the classrooms, the location and accessibility of books, and the availability of physical print resources for students, noting their language and physical condition. While the treatment groups were similar across several dimensions, schools receiving only the in-school component (group B) had materials to support the teaching of reading. As we show in Table IV.3, more classrooms in Group B than in Group A had a complete alphabet in K’iche’ visible to students. Also, more classrooms in Group B than in Group C had a complete alphabet in K’iche’ that was visible to students. Finally, the percentage of classrooms that displayed familiar words written in Spanish and in K’iche’ was statistically significantly higher in classrooms in Group B than in classrooms in Group C. These findings (see also Table IV.3) are consistent with teacher reports during the focus groups that the Leer Juntos, Aprender Juntos training provided them with tools that were useful to creating a print-rich environment in their classrooms. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 42 Table IV.3. Materials to facilitate early reading instruction that are visible to students in the classroom (percentage of classrooms) Materials Leer Juntos (A) Leer Juntos– school only (B) Prevailing Practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Reading area equipped with reading materials for students 0.0 2.0 0.0 -2.0 (0.226) 2.0 (0.225) Classroom with books located in areas that are accessible to students 90.0 86.0 94.1 4.0 (0.483) -8.1 (0.159) Classroom with books placed in several locations 28.0 26.0 30.1 2.0 (0.806) -4.1 (0.620) Materials to facilitate early reading instruction Complete alphabet in Spanish visible to students 46.0 50.0 47.0 -4.0 (0.703) 3.0 (0.775) Complete alphabet in K’iche’ visible to students 2.0 10.0 0.0 -8.0* (0.044) 10.0* (0.012) Familiar words written in Spanish visible to students 92.0 92.0 77.7 0.0 (1.000) 14.3* (0.030) Familiar words written in K’iche’ visible to students 44.0 54.0 18.4 -10.0 (0.287) 35.6* (0.000) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. Students in group C had more access to textbooks but it is not possible to infer from this finding that they had more opportunities to practice reading. Textbooks were observed in most of the classrooms in each treatment group, and narrative books were observed in slightly less than half of the classrooms in each treatment group (Table IV.4). There were no statistically significant differences between treatment groups in the types of books observed in the classrooms, with one exception: more classrooms in Group C (96 percent) than in Group B (80 percent) were observed having textbooks. Therefore, students in classrooms in schools in Group C had more access to textbooks on topics such as science and social studies. However, from that finding it is not possible to infer that students in classrooms in Group C had more opportunities to practice reading than students in classrooms in schools in Group B. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 43 Table IV.4. Characteristics of the books observed in the classrooms (percentage of classrooms) Difficulty and genre Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Difficulty level of books Books with figures or the alphabet, without a story 8.0 2.0 3.0 6.0 (0.129) -1.0 (0.809) Picture books 6.0 8.0 10.1 -2.0 (0.670) -2.1 (0.662) Picture books with some text 24.0 28.0 40.3 -4.0 (0.649) -12.3 (0.167) Simple story books (obras sencillas) 30.0 20.0 16.8 10.0 (0.222) 3.2 (0.700) Textbooks 90.0 80.0 96.2 10.0 (0.123) -16.2* (0.014) Other books 16.0 6.0 5.9 10.0 (0.091) 0.1 (0.986) Book genre Reference books 12.0 8.0 11.0 4.0 (0.500) -3.0 (0.613) Story books 48.0 44.0 47.0 4.0 (0.674) -3.0 (0.753) Workbooks 12.0 4.0 3.9 8.0 (0.105) 0.1 (0.987) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. Observers also recorded the number of books and other instructional texts in Spanish and Quechua that they observed in the classroom, but we did not find any statistically significant difference between treatment groups in the availability of those materials in the classroom. We present these results in Appendix E, Table E.11. 2. Environment in the home As we mention in Chapter II, we conducted a household survey to obtain information on the characteristics of the literacy environment in the homes of the children participating in the evaluation. The evaluation team collected information on the availability of books for children in the household, the frequency with which the children use those books, the family members who read with the child, the existence of dedicated time for reading at home for children, and the activities—related to reading and not—that children do when they are at home. a. Availability of reading materials at home We found no statistically significant differences between treatment groups in the availability and use of books for children at home. As we show in Table IV.5, books for children were available in most households in the evaluation (82 to 86 percent in each treatment EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 44 group). Also, households in each treatment group had, on average, two to three books for children. Books were available only in Spanish in 54 to 59 percent of the households in each treatment group, and about one-quarter of the households in each treatment group reported the availability of children’s books written in Spanish and in K’iche’. Those differences between treatment groups were not statistically significant (Table IV.5). Furthermore, we found that in each treatment group, 36 to 40 percent of the children use the children’s books one or two times per week, and about 30 percent of the children in each treatment group use the children’s books every day. Across all the comparisons presented in Table IV.5, we did not find any statistically significant difference. Table IV.5. Availability of books for children at home Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Percentage of households in which books for children are available 85.8 84.6 82.2 1.2 (0.689) 2.4 (0.389) Number of books for children in the household (average) 2.6 2.7 2.4 −0.1 (0.645) 0.3 (0.185) Percentage of households with books for children in the following languages: Spanish only 57.1 59.3 54.0 −2.3 (0.529) 5.3 (0.137) K’iche’ only 2.1 1.5 1.3 0.5 (0.468) 0.2 (0.819) Spanish and K’iche’ 25.1 22.9 24.8 2.2 (0.459) −1.9 (0.542) Percentage of children using the books for children in the household with the following frequency: Never 14.8 16.7 18.6 −1.9 (0.520) −1.9 (0.496) One or two times per week 40.0 36.0 36.6 4.1 (0.191) −0.6 (0.857) Three or four times per week 16.7 17.5 17.3 −0.8 (0.754) 0.3 (0.908) Every day 28.5 29.8 27.6 −1.3 (0.642) 2.2 (0.454) Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. b. Reading at home Family/household members participate in reading activities with the child at home. As we show in Figure IV.3, about one-third of the children in each treatment group are read to by the household survey respondent. This finding is consistent with the household survey EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 45 respondents’ reports about their literacy skills: about one-third of survey respondents (35 percent in Group A, 39 percent in Group B, and 36 percent in Group C) reported they are able to read (Table II.7), which means that about two-thirds of household survey respondents in each treatment group are not able to read. Therefore, we asked whether they look at pictures in books with the evaluation children. We also asked whether other family members read with the evaluation child/children living in the household and found one statistically significant difference between treatment groups: siblings read with a larger percentage of children in Group A than in Group B (Figure IV.3). This result suggests that the activities of the community action component of the intervention—the reading buddies activity, for example—may have encouraged older siblings to read to their brothers and/or sisters in the evaluation. Figure IV.3. Percentages of evaluation children whose parents or other household members read with them Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. Leer Juntos, Aprender Juntos aimed to increase the time scheduled for reading in the home, but effects were not apparent at the time we conducted the household survey. For most children in each treatment group (87 percent in Group A, 89 percent in Group B, and 86 percent in Group C), parents or caregivers scheduled a time for their children to dedicate to reading activities at home (Table IV.6). On average, the dedicated time for reading at home is three hours per week in each treatment group. We found no statistically significant differences between treatment groups in the scheduled time for reading at home. At the time of the household survey, children in Group A had been exposed to the full intervention for about 10 to 16 months (for students in Phase II and Phase I schools, respectively). However, even if the time children spend on reading activities at home increased in the period after the household survey, EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 46 children’s reading outcomes did not improve. (See Chapter V for the impact findings on children’s reading outcomes.) Table IV.6. Dedicated time for reading at home Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school compone nt (B–C) Percentage of children who have dedicated time for reading at home 86.9 88.7 86.0 1.8 (0.429) 2.7 (0.268) Hours per week scheduled for reading at home (average) 2.9 3.0 2.7 −0.2 (0.503) 0.3 (0.143) Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. It is possible that we did not find impacts of the community activities on the time scheduled for reading in the home, because most parents in the evaluation already set up time for their children to do reading-related activities. Therefore, there was little room for the intervention to create an effect in this respect. Children in all three treatment groups engaged in reading activities outside of school hours. At home or in the community, most children in the evaluation (at least 75 percent in each treatment group) participated in reading activities such as reading books and doing homework, and 39 to 44 percent of the children in each treatment group heard or told stories at home (Figure IV.4). When asked on the household survey about the amount of time children spent on different reading activities in the preceding week , parents in the three treatment groups reported that children did homework on four days of the week preceding the household survey and heard or told stories on one day of that week (results not shown). Children in Group A played reading games with other children in the communities one day of that week, whereas children in Groups B and C did not spend time on that activity in that week. This finding is consistent with children in the communities in Group A having had access to the community action component activities of the intervention, whereas children in communities in Group B and C did not. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 47 Figure IV.4. Reading-related activities at home (percentage of children) Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. * Difference in group means is statistically significant at the 0.05 level. c. Activities at home other than reading Children in the group receiving the community action component of the intervention spent less time at home helping with household chores. As we show in Table IV.7, in the week before administration of the household survey, fewer children in Group A (78 percent) than in Group B (85 percent) helped with housework, and children in Group A did that activity on fewer days (fewer than four days, on average) than children in Group B (more than four days, on average). This statistically significant difference is small; however, it suggests that the community component intervention activities to which the children in Group A had access may have led them to replace time on household chores with time spent participating in the reading activities offered by the program in their community. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 48 Table IV.7. Children’s activities at home that are not related to reading Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Percentage of children who in the week before the administration of the household survey: Played inside or outside the house 96.8 98.0 98.0 −1.2 (0.178) 0.1 (0.923) Listened to the radio or watched television 71.4 74.4 76.0 −3.0 (0.380) −1.6 (0.594) Helped with housework 78.4 85.4 88.6 −7.0* (0.000) −3.1 (0.056) Helped with vegetable garden/corn field and/or with livestock 73.8 71.4 72.7 2.4 (0.461) −1.3 (0.689) Cared for younger siblings 71.3 73.0 71.8 −1.7 (0.536) 1.2 (0.640) Worked for pay or wage 3.8 5.3 4.2 −1.5 (0.236) 1.2 (0.302) Number of days (average) in the week before the administration of the household survey in which children: Played inside or outside the house 6.3 6.4 6.3 −0.2 (0.228) 0.1 (0.514) Listened to the radio or watched television 4.0 4.3 4.5 −0.2 (0.228) −0.2 (0.335) Helped with housework 3.8 4.3 4.4 −0.5* (0.002) −0.1 (0.711) Helped with vegetable garden and/or with livestock 3.3 3.2 3.2 0.1 (0.463) 0.0 (0.948) Cared for younger siblings 3.8 4.0 3.9 −0.1 (0.469) 0.1 (0.713) Worked for pay or wage 0.1 0.1 0.1 −0.0 (0.468) −0.0 (0.910) Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Household Survey—Midline 2014 and 2015. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 49 V. WHAT WERE THE IMPACTS ON CHILDREN’S EARLY-GRADE READING SKILLS? In this chapter, we present our findings on the relative effects of the community action and in-school components of the Leer Juntos, Aprender Juntos program on the reading skills of children in Guatemala. We did not find evidence of impacts of any of the components of the program on early-grade reading skills. That is, our findings indicate that the children in the schools that did not receive any of the components of the intervention (Group C, the prevailing practice group) learned as much as the children in the schools that implemented the in-school component only (Group B, implementing the teacher training and coaching component), and who in turn learned as much as the children in the schools that implemented the full intervention (Group A, implementing the community action and the teacher training and coaching components).16 We detail our findings on program impacts below. A. How we measured early-grade reading skills We assessed the reading skills in Spanish of the 3rd grade children in the evaluation with three tasks: (1) pseudo-word reading (decoding), (2) reading fluency, and (3) reading comprehension. These three tasks were adapted from REAULA’s materials for curriculum-based reading assessments (USAID REAULA 2013) following the guidelines of the Early Grade Reading Assessment (EGRA) Toolkit (RTI International 2015). We identified reading passages for the fluency and reading comprehension tasks from materials provided by Proyecto Alianzas in Guatemala (USAID and Juárez and Asociados 2013) and from grade-level reading materials from the Ministry of Education in Guatemala (DIGEDUCA 2014a, 2014b). All children were assessed in Spanish, as most of the teachers in the evaluation used Spanish as the language of reading instruction. We measured decoding and fluency skills, because they are precursors to reading comprehension—that is, children must be able to translate a printed word into sound (decoding skills) and to read connected text with speed, accuracy and expression (fluency skills) before they can achieve reading comprehension (Kim et al. 2016; National Reading Panel 2000). In the pseudo-word decoding task, children read aloud as many pseudo-words as they can within 60 seconds, with a maximum of 50 pseudo-words. Pseudo-words are vowel-consonant combinations that follow Spanish phonological and spelling rules but are not actual meaningful words (for example, lete and golpa). To administer this task, examiners were instructed to stop (discontinue) the task and move onto the next task (fluency) if the child answered the first five items in the task incorrectly or could not complete the first five items in 60 seconds. We used the number of pseudo-words read or decoded correctly to examine program effects. In the reading fluency task, children read aloud as many words as they can within 60 seconds from a short story with a total of 112 words (21 short sentences). This task has a time limit so that we can obtain a measure of children’s ability to read text quickly and accurately. We 16 Although not the main focus of our evaluation, we did examine the impacts of the full intervention. That is, we examined the contrast between Group A (receiving both the in-school and the community action components) and Group C (implementing the prevailing practice in Guatemala). From that analysis, we found similar results to those found when we compared Group A with Group B and Group B with Group C. See Appendix, Table H.4 for more details on the impacts of the full intervention on children’s reading skills. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 50 used the number of words read correctly in 60 seconds to examine the effects of the program. Examiners were instructed to stop the task if the child could not read correctly any of the 20 words included in the first two lines of text within 60 seconds. In addition, examiners did not administer the subsequent reading comprehension task to the children whose fluency task was stopped, to avoid causing them excessive stress. From the decoding and reading fluency tasks, we also constructed “accuracy” scores. We calculated these scores by dividing the number of correctly read words by the total number of words that the child read (correctly or incorrectly). We also used the decoding and fluency accuracy scores to examine program effects. Higher decoding and fluency accuracy scores suggest that children are more deliberate in translating print into sound (decoding or reading pseudo-words) and in reading words with fluency with a time limit, respectively. The reading comprehension task in the evaluation’s 3rd grade assessment consists of three passages that children are asked to read quietly to themselves. Children received up to four minutes to finish reading a passage. Lookbacks were not allowed. After a child finished reading each passage, the examiner asked him or her to answer five questions about that passage. We constructed two variables from this task to examine program impacts on reading comprehension achievement: (1) the total number of correct answers in the task (up to 15), and (2) an indicator for whether the child correctly answered at least one of the questions in the task. The second construct is a dichotomous measure indicating whether the child demonstrates emergent reading comprehension skills, that is, it indicates that the child was able to complete both the decoding and fluency tasks and at least one of the questions in the reading comprehension task We present standard deviations of all reading measures in Appendix F, Table F.1. B. Impacts on reading skills We expected to find that each of the components of the program had positive impacts on the reading skills of the children in the evaluation as they completed the 3rd grade. However, we did not find evidence of statistically significant impacts of either component on children’s reading skills. This finding does not mean that children in the schools that implemented the Leer Juntos, Aprender Juntos program did not learn. Rather, what the finding means is that the children in the schools that implemented the full program (Group A) and the children in the schools that implemented only the in-school component (Group B) achieved similar levels of reading comprehension skills as the children in the schools that did not implement the intervention (the prevailing practice group). 1. Impacts on decoding and fluency The impacts of Leer Juntos, Aprender Juntos on decoding and reading fluency skills measured in terms of correctly read words per minute were not large enough to be statistically significant. As mentioned earlier, we assessed the decoding and reading fluency skills of the children in the evaluation as they completed the 3rd grade to examine whether the intervention had an impact on foundational reading skills. Decoding and reading fluency skills are essential elements in achieving reading comprehension (Fuchs et al. 2001). Decoding skills— the ability to translate a printed word into sound—encompass alphabet knowledge (ability to recognize and link written letters to their sound) and phonological awareness (ability to focus on, EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 51 manipulate, break apart, and put together sounds orally). Reading fluency is the ability to read with speed, accuracy, and expression. For each of the treatment groups, Figure V.1 presents the average number of pseudo-words read correctly in the decoding task, and Figure V.2 presents the average number of words read correctly in the fluency task. On average, children in the schools in the three treatment groups read correctly in one minute a similar number of pseudo-words in the decoding task (22 in Groups A and C and 23 in Group B) and in the fluency task (42 in Groups A and C and 44 in Group B). We did not find any statistically significant differences between treatment groups in the sum of correct answers in the decoding and reading fluency tasks that we administered to children in the evaluation sample in the final (endline) follow-up (Figures V.1 and V.2). Figure V.1. Similar number of pseudo-words read correctly in the decoding task (of 50) for each treatment group Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: None of the differences between treatment groups presented in this figure is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 52 Figure V.2. Similar number of words read correctly in the fluency task (of 112) for each treatment group Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: None of the differences between treatment groups presented in this figure is statistically significant at the 0.05 level. We did not find positive statistically significant impacts of either of the program’s components on the accuracy scores, but children in the schools implementing the in-school component obtained, on average, higher fluency accuracy scores than the children in schools implementing the full intervention. Figure V.3 presents children’s decoding and fluency accuracy scores on a scale of 0 to 100. As we show in that figure, we found no impacts on the accuracy scores. However, Group B children obtained higher fluency scores compared with Group A children, this difference is statistically significant. At baseline (see Table II.7), we found that on average, children in schools in Group B had higher scores on the early literacy skills assessments we administered than the children in schools in Group A (but only the difference in the letter identification score was statistically significant). Our analysis of program impacts controlled for those differences at baseline between treatment groups, therefore, the fact that we still see those differences between Groups A and B in the final follow-up seems to indicate that the intervention did not help in closing that gap between the students in those groups. Our findings are consistent with findings from research conducted by Save the Children. The finding presented in this report on fluency accuracy scores, as well as our findings regarding the differences in number of words read correctly in the decoding and fluency tasks between Groups A and B are consistent with a descriptive evaluation conducted by Save the Children after the second year of program implementation in Guatemala (Quixtan and Pava 2014). In Save the Children evaluation, the authors collected and examined data on the vocabulary, fluency, reading accuracy, and reading comprehension skills of 391 children in Groups A and B as they attended 1st and 2nd grades. The findings suggest a pattern of children EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 53 in Group B schools obtaining, on average, higher scores than children in Group A schools on the vocabulary, fluency, reading accuracy, and reading comprehension tests administered in 1st and 2nd grade. Although not all of the differences in means tested in Save the Children’s evaluation were statistically significant, the overall tendency was for children in Group B to show more advanced reading skills than children in Group A, on average. Figure V.3. No positive program impacts on decoding and fluency accuracy scores Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. * Difference in group means is statistically significant at the 0.05 level. 2. Impacts on reading comprehension The Leer Juntos, Aprender Juntos program targeted foundational reading skills with the objective of helping children in a multilingual context achieve reading comprehension in Spanish by 3rd grade. Children need to understand what they read so they can learn. Children who do not learn to read with comprehension in the earlier grades face greater difficulties in learning in later grades, as they have limited ability to understand information presented in print form, do not develop proper writing skills, and cannot study materials in other subject areas by themselves (Gove and Cvelich 2011). We found no statistically significant impacts of any of the components of the intervention on children’s performance on the reading comprehension task we administered in the final follow-up. As shown in Figure V.4, about two-thirds of the children in each treatment group (A and B) achieved basic reading comprehension skills—that is, children who were able to complete both the decoding and fluency tasks and answer at least one question correctly in the reading comprehension task. We found no statistically significant differences between treatment groups in the percentage of children with basic reading comprehension skills EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 54 in the 3rd grade. In addition, children in all treatment groups had similar number of correct answers in the reading comprehension task of the evaluation’s assessment in the 3rd grade, about three questions (Figure V.5). Table G.1 in Appendix G presents all the findings shown here in Figures V.1 to V.5 in terms of standard deviation units (known as effect sizes). Figure V.4. A Similar percentage of children achieved a basic level of reading comprehension skills in each treatment group Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: None of the differences between treatment groups presented in this figure is statistically significant at the 0.05 level. Figure V.5. Similar number of questions answered correctly in the reading comprehension task (of 15) for each treatment group Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: None of the differences between treatment groups presented in this figure is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 55 3. Sensitivity analysis The findings are not sensitive to changes in assumptions for the impact estimation model. We conducted robustness checks on two important assumptions we made in the estimation of program impacts. First, we reweighted the data to account for children’s nonresponse to the 3rd grade assessment. Second, we estimated program impacts based on a regression model that included control variables without imputations, which reduced the size of the analytic sample by about 43 percent. The results from these alternative specifications are the same as those presented in this chapter. In particular, based on the two alternative specifications, we found no evidence of statistically significant impacts of either of the program components. We present the findings from these sensitivity analyses in more detail in Appendix H, Tables H.1 and H.2. We also examined program impacts on literacy outcomes for subgroups of students defined by phase (cohort). There were no positive, statistically significant impacts of any of the components of the program in either Phase I or Phase II (see Appendix H, Table H.3). As mentioned earlier, we did find that in Phase II, children in Group B had better decoding and fluency accuracy scores than their counterparts in Group A, with the differences being statistically significant. Program staff did note that , in their experience conducting teacher training and coaching visits in the classrooms, the teachers in schools in Group B demonstrated excellent command of the reading teaching strategies that were taught in the program’s training (Save the Children 2013, 2014, 2015). Those abilities of teachers in Group B could have translated into better outcomes for their students. Finally, we examined the contrast between Group A and Group C to assess the effect of the full intervention (the community action component and the teacher training and coaching component) on children’s reading outcomes (see Appendix H, Table H.4). We learned that the findings for the full intervention are consistent with the findings for each of the components of the program. That is, we did not find any positive, statistically significant effect on children’s reading skills of the full intervention, as compared with a control condition (the prevailing practice group). C. Impacts by gender Similar to our findings in the analysis based on the full sample, we did not find evidence of positive impacts of the program components on the reading skills of boys or girls in the evaluation. We did find, as we show in Table V.1, that girls in Group B read more words correctly in the fluency task and had slightly higher fluency accuracy scores, on average, than girls in Group A. These differences are statistically significant. We did not find any statistically significant difference between treatment groups on the reading skills of boys in the evaluation. We present the effects of each of the components of the program for girls and boys in terms of standard deviation units in Appendix I, Tables I.1 and I.2. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 56 Table V.1. Impacts on the literacy outcomes of girls and boys in the evaluation Literacy outcome Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Girls Decoding Number of pseudo-words read correctly in one minute (of 50) 20.8 23.0 21.7 -2.2 (0.076) 1.3 (0.276) Accuracy score (0 to 100) 66.2 71.4 72.9 -5.2 (0.106) -1.5 (0.627) Fluency Number of words read correctly in one minute (of 112) 40.5 45.4 43.3 -4.9* (0.036) 2.1 (0.365) Accuracy score (0 to 100) 72.0 79.8 77.9 -7.8* (0.021) 1.9 (0.537) Reading comprehension Number of questions answered correctly (of 15) 2.8 2.9 2.7 -0.1 (0.957) 0.2 (0.456) Percentage of girls who achieved basic reading comprehension skills 60.7 63.2 62.1 -2.5 (0.528) 1.1 (0.777) Boys Decoding Number of pseudo-words read correctly in one minute (of 50) 22.5 22.5 21.6 -0.0 (0.985) 0.9 (0.518) Accuracy score (0 to 100) 71.3 73.5 73.3 -2.2 (0.435) 0.2 (0.927) Fluency Number of words read correctly in one minute (of 112) 42.9 42.0 40.7 0.9 (0.736) 1.3 (0.578) Accuracy score (0 to 100) 77.0 79.3 79.9 -2.3 (0.414) -0.6 (0.834) Reading comprehension Number of questions answered correctly (of 15) 3.3 2.9 3.0 0.4 (0.108) -0.1 (0.925) Percentage of boys who achieved basic reading comprehension skills 65.7 65.8 68.5 -0.1 (0.983) -2.7 (0.536) Number of girls 218 230 231 Number of boys 221 228 210 Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 57 Our examination of focus groups and interview data collected in the qualitative evaluation (Chapter III) and of documents by the program implementation team (Save the Children 2013, 2014, 2015) did not show that any particular aspect of the program or its implementation might have favored girls. We also looked at whether the seating arrangements in the 3rd grade classrooms could be favoring girls in Group B, but we did not find any statistically significant differences in classroom seating arrangements for girls (or boys) between Groups B and A.17 We also examined, for the subgroups of boys and girls, differences between treatment groups (Groups A, B, and C) in baseline characteristics such as age, preschool attendance, grade retention, and socioeconomic characteristics (parents’ education, household income, and household assets); baseline scores in the emergent literacy skills assessments conducted at baseline; and characteristics of the literacy environment at home (number of books available in the home, reading activities with adults and siblings, and reading activities in the community).19 We found no statistically significant differences in baseline characteristics between the girls in Group B and the girls in Group A that could explain why the girls in Group B had slightly better fluency outcomes than the girls in Group A. At baseline, there were no statistically significant differences between the two sets of girls in terms of their age, preschool attendance, grade retention, parents’ education, and household income and assets. Likewise, we found no statistically significant differences between the two groups of girls in their baseline scores in emergent literacy skills or in the characteristics of the literacy environment in their homes. One possible explanation was that Group B just had more skilled teachers. Program staff reported that teachers in Group B schools showed excellent command of the teaching strategies that were part of the program’s training (Save the Children 2013, 2014, 2015). However, there were no observable differences between teachers in the two groups at baseline. Another possible explanation, mentioned earlier in this chapter, is that program resources or efforts were more focused for schools in Group B than for schools in Group A. Group B schools did not have to divide their attention between the in-school and community components, so it is possible that teachers in Group B had more time to practice the skills they learned through the program’s training than teachers in Group A. This could explain girls in Group B had higher scores in the fluency task than girls in Group A. 17 We do not present these findings in this report, but they are available from the authors upon request. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 59 VI. CONCLUSIONS, LIMITATIONS, LESSONS LEARNED, AND RECOMMENDATIONS This evaluation sought to answer three questions. In this chapter, we present conclusions corresponding to each of those questions and discuss lessons learned and recommendations. A. Conclusions 1. Were the program components implemented as intended? Leer Juntos, Aprender Juntos was implemented mostly as intended, although there were some challenges. Teachers in the schools assigned to implement the in-school component did receive training and individualized coaching sessions. Beginning teachers found that the Leer Juntos, Aprender Juntos teaching strategies and pedagogic activities were particularly helpful, and teachers in general reported receiving innovative project materials for use in the classroom in Spanish and mother tongue (K’iche’). However, take-up of reading instruction strategies and activities was particularly challenging for teachers who lacked mother tongue skills, and for those who taught in multi-grade classrooms. Additionally, some teachers felt they did not receive as much feedback and support in the coaching sessions as they were expecting, and there was high teacher turnover and rotation of teachers to upper grades. The community action component of Leer Juntos, Aprender Juntos had some of the intermediate impacts predicted by the program logic model, but it did not have impacts on students’ reading skills. The community action component did not increase the amount of time children spent reading by themselves or with parents, but it did increase the likelihood that a sibling would read to them. However, this intermediate impact did not translate into favorable impacts on reading skills: students in schools implementing the full Leer Juntos, Aprender Juntos intervention had similar reading skills to students in schools that implemented only the in￾school component. 2. What is the impact of the teacher training and support component of Leer Juntos, Aprender Juntos on early-grade reading and other intermediate outcomes relative to prevailing practice? The in-school component improved teachers’ participation in training and coaching and some aspects of the classroom literacy environment.. More teachers in the schools implementing the in-school component reported receiving training and coaching on reading instruction than did teachers in the prevailing practice group. In interviews, teachers said that the program’s training helped them implement instruction practices focused on developing students’ foundational reading skills, such as understanding the relationship between letters and sounds, identifying letters, and learning new words. Some teachers in the treatment groups also indicated they were pleased with the types of active and hands-on coaching and effective feedback they received, but others felt they received little feedback and less support than their peers did. The knowledge, skills, and experience of the program specialists who delivered the coaching seemed to influence teachers’ level of satisfaction with the in-school component of the program. The in-school component also improved some aspects of classroom literacy environment. Specifically, more classrooms in Group B schools than in Group C schools EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 60 displayed a complete K’iche’ alphabet and familiar words in Spanish and K’iche’. At the same time, there were no impacts on the availability of reading areas or books in the classrooms, and a negative impact on the availability of textbooks (80 percent in Leer Juntos, Aprender Juntos school-only classrooms and 96 percent of control classrooms). The favorable intermediate effects of the in-school component did not translate into impacts on observed literacy instruction or on children’s reading skills. We did not find evidence of impacts of the in-school component on how teachers spent the time during the language or reading lesson that we observed or on their use of instructional practices focused on teaching five foundational literacy skills (alphabet/letter knowledge, vocabulary, phonemics and phonological knowledge, fluency, and reading comprehension). In addition, we found no statistically significant positive impacts of the in-school component on children’s reading skills three years after the program started. This finding does not mean that the children did not learn. Rather, it shows that, on average, children whose schools were offered the in-school component of Leer Juntos, Aprender Juntos made progress in reading that was similar to that of children in the control group. The reason for the finding of null impacts of the two program components on children’s reading outcomes is unclear. One possible explanation for the lack of impacts on reading is a weaker than expected contrast between Groups B and C. While the treatment differences in receipt of any training were statistically significant, they were not as large as they could have been, influenced partly by the fact that many teachers trained by Leer Juntos, Aprender Juntos were no longer teaching students in the evaluation sample at endline. Also, the Leer Juntos, Aprender Juntos program was implemented in a context where several other initiatives—at the national and regional levels in the Department of El Quiché—had been launched to improve early-grade reading. These other initiatives, as discussed in Chapter I, had elements that were similar to the in-school component of the Leer Juntos, Aprender Juntos program and were also targeting the improvement of children’s reading skills. Therefore, it is possible that the Leer Juntos, Aprender Juntos approach was not able to provide a meaningful contrast to these collective efforts in the region and the country to train teachers in the early grades and to improve children’s early-literacy and reading comprehension outcomes. Another possible explanation is that teachers lacked the mother tongue (K’iche’) language skills necessary to implement the program as intended. As we describe in Chapter I, one of the goals of the teacher training and coaching component of Leer Juntos, Aprender Juntos is to equip teachers to provide reading instruction focusing on five foundational reading skills (alphabet knowledge, phonological awareness, vocabulary, fluency, and comprehension) in the students’ mother tongue and/or in Spanish, according to the linguistic background of students. This component of the program also aims to enable teachers to increase students’ access to and use of mother tongue reading materials. Therefore, if teachers do not have the mother tongue language skills to provide instruction on the five foundational reading skills and to promote students’ access to and use of mother tongue reading materials in classrooms where most students better communicate in their mother tongue than in Spanish, then the teachers cannot fully implement the program as intended. In interviews and focus groups, program staff and teachers indicated that most teachers in the early grades in the evaluation’s schools were not skilled enough in the K’iche’ language to be able to fully incorporate the teaching strategies they learned through the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 61 program’s training. If teachers could not fully apply the training and coaching provided by the program according to the linguistic background of students, then it is harder for those teaching practices to translate into effects on children’s reading outcomes. Another factor that could weaken the strength of any program involving teacher training is teacher mobility. Normal movements of teachers out of the school or out of the grades targeted by the intervention could mean that the percentage of teachers with current training from the program could go down over time if significant resources are not invested in annual training updates. This effect in Guatemala was documented in Chapter III. Also, the evaluation does not capture the potentially positive impact of teachers trained by the program who left the evaluation sample to teach in other grades or schools. Another difference is the prevailing pedagogical practices. The Leer Juntos, Aprender Juntos program provides training in best practices to teach foundational reading skills in multilingual contexts, monitor students’ mastery of those skills, and increase students’ access to and use of reading materials in the mother tongue. In Guatemala, the Ministry of Education’s approach to early-grade reading instruction also emphasizes instruction on foundational reading skills leading to reading fluency and reading comprehension. According to the standards of the national curriculum, students are expected to learn to decode familiar and unfamiliar words in 1st grade, develop reading fluency in 2nd grade, and read with comprehension and use reading as a tool to acquire new knowledge in 3rd grade. In Guatemala, there may not be a meaningful contrast between the prevalent approach to reading instruction and the Leer Juntos, Aprender Juntos program’s approach. 3. What is the impact of the community action component of Leer Juntos, Aprender Juntos on early-grade reading and other intermediate outcomes relative to an intervention that does not have the community action component? The community action component did not have impacts on the classroom environment, but did have some of the intermediate impacts predicted by the program logic model on the literacy environment at home. This component did not result in more books in the classroom or at home than the version of Leer Juntos, Aprender Juntos without the community action component. Classrooms did not differ substantially in the availability of reading materials (picture books, story books, textbooks, workbooks)—the percentages of classrooms with various types of books were comparable enough that any differences were not statistically significant. Similarly, the availability of reading materials in the homes of students in schools that received the community component were not significantly different from those of their counterparts who did not get the community action component. The community action component did not increase the amount of time children spent reading by themselves or with parents but did increase the likelihood that a sibling would read to them. Children in all of the intervention groups had about three hours per week scheduled for reading at home. About 30 percent of parents of children whose schools were assigned to the community action component had reported reading with their children and 65 percent reported looking at books with the child (we had found at baseline that slightly more than 60 percent of the parents in the sample were not able to read). Differences in these percentages between children in community action schools and other children were not statistically significant. Only EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 62 the differences in percentages who said they read with a sibling (64 percent from the community action group versus 55 percent from the other groups) were statistically significant. Despite having some intermediate effects, the community action component did not have impacts on student’s reading skills. After nearly three years of intervention, children in the schools that implemented the community action component did acquire reading skills: they progressed from being able to read 3 pseudo-words correctly (out of 50) in the decoding task at baseline to 22 words at endline or final follow-up. In fluency, they progressed from being able to read 2 words correctly (out of 9 words) at baseline to 42 words (out of 112 words in a passage) at endline. And at baseline, children were not testable in reading comprehension yet, but by the endline, 63 percent of the children in the schools that implemented the community action component were able to read a short passage and correctly answer at least one reading comprehension question. However, those gains were not significantly different from those achieved by students in schools that were not implementing the community component. We identified three possible explanations for the lack of impacts. First, barriers in the implementation of the community action component due to reliance on volunteers could have affected the quality of the community action activities. Second, barriers to children’s and parents’ participation in community activities could have negatively affected children’s and parents’ attendance resulting in fewer children and parents participating in community activities, or participating less frequently than expected. In fact, about 60 percent of the children in the schools assigned to implement the community action component did not participate in the reading camps, story hour, or reading buddies activities that the program offered. Program implementers reported that it was difficult to recruit and retain volunteers to lead the activities of this component. In focus groups, some parents reported barriers to getting their children to the activities, such as inclement weather and transportation costs. And finally, it is possible there was dilution of resources for the schools that implemented the community action component, because program staff had to focus on implementing not only the community action component, but also the in-school component. In contrast, program staff in the schools that did not implement the community action component could focus all resources into the implementation of only one program component (the in-school component). B. Limitations This evaluation of Leer Juntos, Aprender Juntos has some limitations to bear in mind. The first refer to the evaluation design, specifically the timing of the baseline data collection and the use of a three-armed randomized design, and the last is about generalizability of the findings. The baseline of the evaluation was not a “true” baseline. The evaluation team administered the baseline assessment of students’ early literacy skills about three months after random assignment, which was about two months after the rollout of the program’s teacher training activities. It was not feasible to measure baseline student outcomes before random assignment because several evaluation activities needed to be completed before data collectors could begin baseline data collection; these included identifying a local partner that could assist in recruiting and training field workers, implementing the data collection plan, and supervising data collection activities. As a result of the late baseline, the children’s skills that we captured in the baseline assessment (when students in the evaluation were in first grade) reflect the ability of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 63 students after up to three months of potential exposure to different conditions caused by assignment of schools to different intervention groups. Despite this limitation, it is still appropriate to consider the evaluation’s baseline assessment as the reference point for the evaluation. Under most circumstances, a late baseline will still be useful for estimating the impact of that intervention as long as the impact on skill development is slow in the early period of program implementation (Schochet 2010). That is the case in the current evaluation because the teachers in the groups receiving the program (Groups A and B) had only been practicing their newly acquired skills for at most two months when the baseline data collection took place, and literacy acquisition in children is a process that happens gradually over many months or even years. The evaluation design assumes that the impacts of in-school and community action components of the intervention are additive. Specifically, the evaluation design assumes that the effect of the community action component on its own relative to prevailing practice is equal to the sum of the impact of the combined in-school and community action components relative to the in-school component on its own. This assumption could be violated if implementing the in￾school component makes it easier or harder to simultaneously do community activities. We did not find clear evidence of such a violation, but it should be taken into account by any policymakers who wish to apply the lessons of this evaluation to a future implementation of community action on its own. Findings from this evaluation may not necessarily apply to other regions. It is always tempting to generalize from the experience of one or two evaluations to other contexts, but caution is warranted. The evaluation included schools from a few municipalities in El Quiche, which is just one of 22 departments in Guatemala. While we also conducted the evaluation in the Apurímac department of Peru (Lugo-Gil et al. 2021a; Lugo-Gil et al. 2021), these communities were selected with the following criteria in mind: the locations had to be within reasonable driving distance from each other to facilitate intervention and evaluation activities. They had to contain a high percentage of families that spoke one language besides Spanish, in this case K’iche’ (in Guatemala) or Quechua (in Peru). Results could differ if an approach like Leer Juntos, Aprender Juntos was implemented in other communities with different characteristics and challenges than those communities included in this evaluation. C. Lessons learned There were several lessons learned from the evaluation. These lessons refer to the need for a good fit between the sociocultural context, the needs of families and teachers, and the program’s requirements for successful implementation. Existing programs may be more effective or more similar to a new program than expected. In Guatemala, there are several efforts/programs at the regional and national levels with components and goals that are very similar to the components and goals of Leer Juntos, Aprender Juntos—for example, Leamos Juntos and the USAID REAULA initiative at the national level, and DIDEDUC’s initiatives at the regional level in the Department of El Quiché. Those efforts offer training to teachers in reading instruction, have components at the community level, and seek to improve children’s early literacy skills and ultimately reading comprehension EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 64 skills. If a new program’s approach is not different enough from that of programs already in place, it becomes difficult to distinguish the effects of the new program from the effects of the existing programs, and thus become difficult to justify investment in such a program. A reading program for children in multilingual communities probably requires multilingual teachers to be successful. Teachers’ proficiency in students’ mother tongue is key for early-grade reading instruction: in contexts where students’ dominant language does not match the primary language of instruction at school, early-grade teachers can help bridge that gap if they use mother tongue for literacy instruction. In this evaluation, most 3rd grade teachers were observed in the final follow-up using Spanish during reading instruction, while in about a third of the classrooms in the treatment groups, most students were observed using K’iche’ to communicate with their teachers. Also, program specialists from Save the Children expressed in interviews with the research team that teachers were not proficient enough in K’iche’ to adequately apply in that language the reading instruction strategies that were taught in the program. At the same time, only 32 percent of the students in the sample demonstrated oral (receptive and expressive) proficiency in Spanish at the beginning of 1st grade (baseline). Therefore, most students in this evaluation were not fully proficient in the language teachers most commonly used for early-grade reading instruction (Spanish), perhaps preventing them from taking advantage of the teaching strategies teachers learned in the Leer Juntos, Aprender Juntos training and coaching. Relying on volunteer work has substantial risks for fidelity of implementation. In the Department of El Quiché in Guatemala, where the program and evaluation took place, doing volunteer work is not common practice, and individuals want to be paid for their time working for the program. Therefore, it was difficult to recruit and retain volunteers, and to find volunteers that had the skills to lead the program activities consistently. That factor may have reduced the effectiveness of the community action component. D. Recommendations Implementers and funders should consider the local context and assess the programmatic needs in that context before investing in designing, implementing, and evaluating new programs. In the evaluation’s region in Guatemala, it was difficult to find, recruit and retain volunteers that had skills to lead the program activities consistently. Therefore, the feasibility of using volunteers should be tested in the local context before starting the implementation and impact evaluation of a program that relies on volunteer work. It is also important to understand the prevailing practices to establish whether a new program offers a clear contrast with what has already been done locally. A clear contrast is critical to any evaluation that seeks to answer questions about a program’s effectiveness, which in turn justifies whether the program is worthy of further investment. In the case studied here, we did not find enough evidence to suggest that, as implemented with the evaluation’s population, Leer Juntos, Aprender Juntos yielded results commensurate with its level of investment. To improve future implementation in multilingual contexts of Leer Juntos, Aprender Juntos and other early grade reading interventions, program implementers, regional and national education authorities, and donors should find teachers who are proficient in students’ mother tongue. Teachers’ proficiency in their students’ mother tongue is key to EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 65 meeting the learning needs of children whose dominant language upon school entry is different from the school’s main language of instruction. For example, to support the acquisition of children’s foundational literacy skills in this context, and be able to apply fully the Leer Juntos, Aprender Juntos’ teaching strategies, teachers need to be proficient in students’ mother tongue (K’iche’ in the evaluation’s region in Guatemala). When teachers are not able to use reading instruction strategies in the language that is most accessible to students, it is more difficult for the gains in teachers’ instructional skills resulting from teacher training and coaching to translate into improvements in children’s reading outcomes. Ensuring that early-grade teachers are proficient in their students’ mother tongue could be achieved through pre-service training, in￾service training, or alternative methods of recruitment and screening of new teachers. And if pre￾or in-service training is not possible, program implementers and donors should consider providing, as part of the program, additional training to teachers on their students’ mother tongue and on the use of instruction strategies in that language prior to implementing a program that relies on teachers to have those skills. This page has been left blank for double-sided copying. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 67 REFERENCES Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy, vol. 2, 2010, pp. 1–30. Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. “Remedying Education: Evidence from Two Randomized Experiments in India.” Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1235–1264. Bruns, B., and J. Luque. “Great Teachers: How to Raise Student Learning in Latin America and the Caribbean.” Washington, DC: World Bank, 2014. Chesterfield, Ray, and Adriana Abreu-Combs. “Centers for Excellence in Teacher Training (CETT): Two-Year Impact Study Report (2008–2010).” Washington, DC: U.S. Agency for International Development, January 2011. Comings, John. “Evidence-Based Reading Projects.” Version I. Washington, DC: U.S. Agency for International Development, December 2012. Cotto, E., and G. Quiñonez. “El Tesoro de la Lectura.” Material de apoyo para desarrollar la lectura inicial. Nivel de Educación Primaria, primer ciclo. Guatemala: Dirección General de Evaluación e Investigación Educativa. Ministerio de Educación, 2014. Crouch, Luis, Medina Korda, and David Mumo. “Improvements in Reading Skills in Kenya: An Experiment in the Malindi District.” Washington, DC: U.S. Agency for International Development, Bureau for Economic Growth, Agriculture and Trade, March 2009. Dirección Departamental de Educación (DIDEDUC). “Presentación del Programa Estrella: Éxito Escolar en Primaria Año 2014.” Ciudad de Guatemala: Ministerio de Educación, 2014a. DIDEDUC. “Presentación Departamental de la Agenda Estratégica: Éxito Escolar en Primaria 2014.” Ciudad de Guatemala: Ministerio de Educación, 2014b. Dirección General de Evaluación e Investigación Educativa (DIGEDUCA). “El Tesoro de la Lectura: Textos para Desarrollar la Lectura Comprensiva.” Material de Apoyo para el Docente, Comunicación y Lenguaje, Nivel de Educación Primaria, segundo ciclo. Ciudad de Guatemala: Ministerio de Educación, 2014a. DIGEDUCA. “El Tesoro de la Lectura: Textos para Desarrollar la Lectura Inicial.” Material de Apoyo para el Docente, Comunicación y Lenguaje, Nivel de Educación Primaria, primer ciclo. Ciudad de Guatemala: Ministerio de Educación, 2014b. Friedlander, Elliott, and Claude Goldenberg (eds.). Literacy Boost in Rwanda: Impact Evaluation of a 2-year Randomized Control Trial. Stanford, CA: Stanford University, 2016. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 68 Fuchs, L., D. Fuchs, M.K. Hosp, and J. Jenkins. “Oral Reading Fluency as an Indicator of Reading Competence: A Theoretical, Empirical, and Historical Analysis.” Scientific Studies of Reading, vol. 5, no. 3, 2001, pp. 239–256. Glazerman, Steven, Larissa Campuzano, Nancy Murray, and Camila Fernandez. “Final Leer Juntos, Aprender Juntos Evaluation Plan.” Washington, DC: Mathematica Policy Research, March 22, 2013. Gove, A., and P. Cvelich. “Early Reading: Igniting Education for All: A Report by the Early Grade Learning Community of Practice, Revised Edition.” Research Triangle Park, NC: Research Triangle Institute, 2011. Hernandez-Zavala, Martha, Harry Anthony Patrinos, Chris Sakellariou, and Joseph Shapiro. “Quality of Schooling and Quality of Schools for Indigenous Students in Guatemala, Mexico, and Peru.” World Bank Policy Research Working Paper 3982. Washington, DC: World Bank, August 2006. Instituto Nacional de Estadística (INE). “Estadísticas por Tema e Indicadores en 2012.” Guatemala City: INE, 2017. Available at https://www.ine.gob.gt/index.php/estadisticas/tema-indicadores. Accessed October 6, 2017. Kim, Y.-S. G., H.N. Boyle, S.S. Zuikowski, and P. Nakamura. “Landscape Report on Early Grade Literacy.” Washington, DC: U.S. Agency for International Development, 2016. Lugo-Gil, Julieta, Kara Contreary, Nancy Murray, Steven Glazerman, Larissa Campuzano, Camila Fernandez, and Kate Place. “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Guatemala: Final baseline report.” Washington, DC: U.S. Agency for International Development, 2016a. Available at https://pdf.usaid.gov/pdf_docs/PA00MMTD.pdf. Lugo-Gil, Julieta, Kate Place, Nancy Murray, Steven Glazerman, Larissa Campuzano, Camila Fernandez. “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Final baseline report.” Washington, DC: U.S. Agency for International Development, 2016b. Available at https://pdf.usaid.gov/pdf_docs/PA00MMTF.pdf. Lugo-Gil, Julieta, Nancy Murray, Camila Fernandez, Steven Glazerman, Ivonne Padilla, and Larissa Campuzano, “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Midline report.” Washington, DC: U.S. Agency for International Development, 2017a. Available at https://pdf.usaid.gov/pdf_docs/PA00X21P.pdf. Lugo-Gil, Julieta, Nancy Murray, Camila Fernandez, Steven Glazerman, and Larissa Campuzano, “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Guatemala: Midline report.” Washington, DC: U.S. Agency for International Development, 2017b. Available at https://pdf.usaid.gov/pdf_docs/PA00X21Q.pdf. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 69 Lugo-Gil, Julieta, Nancy Murray, Camila Fernandez, Steven Glazerman, Ivonne Padilla, and Larissa Campuzano, “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru: Final report.” Washington, DC: U.S. Agency for International Development, 2021a. Lugo-Gil, Julieta, Nancy Murray, and Steven Glazerman, “Evaluation of Leer Juntos, Aprender Juntos Early Grade Intervention in Peru and Guatemala: Summary report.” Washington, DC: U.S. Agency for International Development, 2021. Ministerio de Educación. “Programa Nacional de Lectura Leamos Juntos.” Guatemala City: Gobierno de Guatemala, 2012. Available at https://www.mineduc.gob.gt/leamos_juntos/ documents/programa_leamos_juntos.pdf. Accessed February 12, 2018. National Reading Panel. “Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction.” National Institute of Child Health and Human Development, NIH Publication No. 00-4769. Washington, DC: U.S. Government Printing Office, 2000. Nieto, Ana Maria. “Modifications of Literacy Boost’s General Parent Workshop Design and Parents’ Appraisals of the Workshops in Four Communities in Guatemala: Implications for the Revision of Literacy Boost’s Parent Workshops.” Cambridge, MA: Harvard Graduate School of Education, November 13, 2015. Patrinos, Harry Anthony, and Eduardo Velez. “Costs and Benefits of Bilingual Education in Guatemala: A Partial Analysis.” International Journal of Education Development, vol. 29, 2009, pp. 594–598. Piper, Benjamin, Stephanie S. Zuikowski, and Salome Ong’ele. “Implementing Mother Tongue Instruction in the Real World: Results from a Medium Scale Randomized Controlled Trial in Kenya.” Comparative Education Review, vol. 60, no. 4, November 2016, pp. 776–807. Quixtan, Ana, and Clara Pava. “Guatemala: Análisis de la Evaluación de Línea Media Interna de Leer Juntos, Aprender Juntos.” Guatemala: Save the Children, 2014. RTI International. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: U.S. Agency for International Development, 2015. Rubio, F.E. “Educación Bilingüe en Guatemala: Situación y Desafíos.” Document prepared for the workshop Balance y Perspectivas de la Educación Intercultural Bilingüe, Banco Mundial-PROEIB Andes, June 2004. Save the Children. “Literacy Boost Toolkit: Teacher Training.” Washington, DC: Save the Children, 2012a. Save the Children. “Literacy Boost Toolkit: Community Action.” Washington, DC: Save the Children, 2012b. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH 70 Save the Children. “Leer Juntos, Aprender Juntos Performance Management Plan” Washington, DC: Save the Children, 2013. Save the Children. “Leer Juntos, Aprender Juntos Quarterly Report, September 2012–December 2012 (Quarter 1).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2012. Save the Children. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2013–December 2013 (Quarters 2–5).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2013. Save the Children. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2014–December 2014 (Quarters 6–9).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2014. Save the Children. “Leer Juntos, Aprender Juntos Quarterly Reports, January 2015–December 2015 (Quarters 10–13).” Guatemala City, Guatemala and Lima, Peru: Save the Children, 2015. Schochet, Peter. “The Late Pretest Problem in Randomized Control Trials of Education Interventions.” Journal of Educational and Behavioral Statistics, vol. 35, no. 4, 2010, pp. 379–406. United Nations Educational, Scientific, and Cultural Organization (UNESCO). “Accountability in Education: Meeting Our Commitments.” Global Education Monitoring Report 2017– 2018, Paris, France: UNESCO, 2017. U.S. Agency for International Development (USAID). “Early Grade Reading and Teacher Training in the LAC Region: Influencing Factors, Lessons and Approaches.” Washington, DC: USAID, June 2011. USAID Reforma Educativa en el Aula (REAULA). Programa de Evaluación Formativa en Fluidez y Comprensión Lectora: Evaluación basada en Currículo, Manual de Aplicación. Guatemala, 2013. USAID y Juárez & Asociados. “Evaluación basada en Currículo (EBC). Manual de aplicación: Programa de evaluación formativa de fluidez y comprensión lectora para grados iniciales.” Material elaborado bajo la Orden de Trabajo No. EDH-I-00-05-00033 de la Agencia de los Estados Unidos para el Desarrollo Internacional, Misión Guatemala (USAID/Guatemala), con Juárez y Asociados: Proyecto USAID/Reforma Educativa en el Aula y en apoyo al Convenio de Donación de Objetivo Estratégico No. 520-0436.7, Guatemala, 2013. APPENDIX A: TIMELINE OF EVALUATION ACTIVITIES IN GUATEMALA, 2013–2016 This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH A.3 Timeline of evaluation activities in Guatemala Planning for the implementation of Leer Juntos, Aprender Juntos program activities in Groups A and B in Guatemala began in fall 2012 with the development of a work plan and project strategy, hiring of program implementation staff, a training workshop for key implementation staff from Save the Children, and meetings of key stakeholders—for example, USAID, local education authorities, and relevant local government entities (Save the Children 2012, 2013). Preparation activities continued in winter and early spring of 2013 and included recruiting and orienting technical and project administration staff, developing an implementation monitoring and evaluation system, mapping elements of the teacher training with the national curriculum, translating project materials into K’iche’, making reading materials for local book banks, and recruiting schools to participate in the evaluation. Figure A.1 presents a timeline of Leer Juntos, Aprender Juntos activities in Guatemala from 2013 to 2016. Figure A.1. Timeline of Leer Juntos, Aprender Juntos activities, 2013–2016 Random assignment for the Phase I cohort was conducted in May 2013. The Leer Juntos, Aprender Juntos intervention was rolled out in Guatemala in June 2013 with the first teacher training sessions for Phase I teachers in Groups A and B. The teacher training continued in 2013 with sessions conducted in July, August, and October. Data collection for the baseline of the Phase I cohort occurred in August 2013, approximately two months after teacher training began. Although it would have been ideal to collect the baseline data prior to the rollout of teacher EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH A.4 training in Phase I schools, it was not possible to procure a local data collection firm in the time between reaching an agreement with stakeholders on the evaluation design (May 2013) and the rollout of the teacher training (June 2013). We submitted and agreed upon the evaluation design with stakeholders in May 2013, and only then we were able to share the evaluation design in the terms of reference in the request for proposals from prospective data collectors. We also had to allow time for preparation and review of proposals and, once we had a data collection partner (DMC Consultores), for recruitment and training of interviewers. For Phase II schools, we were able to accomplish baseline data collection closer to the rollout of teacher training. Recruitment and training of the community volunteers began in August 2013 in Group A communities, as did the community-based intervention activities, such as reading banks, story hours, reading buddies, and reading camps (Figure II.1). For the Phase II cohort, random assignment ended in December 2013. The first teacher training sessions for Phase II teachers in the treatment groups occurred in January 2014, and teacher training continued with sessions held in March, May, July, September, and November. Data collection for the baseline of the Phase II cohort was conducted from March through May 2014. The first training of community volunteers for Phase II took place in May 2014, and in July, the community action components began for Phase II communities in Group A (Figure II.1). Data collection for the midline of the Phase I cohort was conducted in July and September through November 2014. The midline data collection for the Phase II cohort was conducted in June and July 2015 (Figure II.1). The teacher training and support (coaching) activities, as well as training of teacher leaders and principals, continued throughout 2014 and 2015 for the Phase I and II cohorts in the treatment groups (A and B). Recruitment and training of community volunteers and community reading activities also continued in 2014 and 2015 for the Phase I and II communities in Group A. The implementation of the teacher training and support and community components of the intervention (Groups A and B) ended in March 2016 for the Phase I and II cohorts (Figure II.1). The final data collection (endline) took place in August and September 2015 for the Phase I cohort and in August and September 2016 for the Phase II cohort. APPENDIX B: QUALITATIVE DATA COLLECTION This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH B.1 Qualitative data collection Table B.1 provides detailed information on the respondent types, methods, topics of inquiry, and location for all qualitative data collection activities. Table B.1. Respondents, methods, topics of inquiry, and locations for the qualitative data collection Respondents a Method Topics of inquiry Location Departmental Education Directorate (DDE) 2 interviews (1 female; 1 male) ● Challenges in reading instruction ● Knowledge about Leer Juntos, Aprender Juntos and other education interventions in the region Santa Cruz del Quiché Save the Children coordinators 3 interviews (2 female; 1 male) ● Program leadership and management ● Literacy Boost adaptation to Guatemala ● Program implementation strategy ● Facilitators and barriers to implementation Guatemala City Save the Children specialists One focus group (2 female; 4 male) ● Program leadership and management ● Literacy Boost adaptation to Guatemala ● Program implementation strategy ● Facilitators and barriers to implementation Santa Cruz del Quiché Community volunteers 5 focus groups (25 female; 25 male) ● Roles and activities of volunteers ● Motivation for and barriers to volunteering ● Perception of program participation and take-up Santa Cruz del Quiché, Chichicastenango, San Antonio Ilotenango, San Andrés Sajcabajá, Zacualpa Program teachers (in schools in Groups A and B) 5 focus groups (25 female; 12 male) ● Program knowledge and expectations ● Program participation and take-up ● Barriers to implementation of program activities ● Changes in reading-instruction practices ● Perception of program activities ● Perception of program outcomes Santa Cruz del Quiché, Chichicastenango, San Antonio Ilotenango, San Andrés Sajcabajá, Zacualpa Prevailing practice teachers (in schools in Group C) 5 focus groups (19 female; 11 male) ● Reading-instruction practices and challenges ● In-service training ● Participation in reading instruction programs Santa Cruz del Quiché, Chichicastenango, San Antonio Ilotenango, San Andrés Sajcabajá, Zacualpa Parents of children in schools implementing the program (Groups A and B) 5 focus groups (30 female; 6 male) ● Program knowledge and expectations ● Program participation and take-up ● Barriers to participation ● Perceived benefits of program services Santa Cruz del Quiché, Chichicastenango, San Antonio Ilotenango, San Andrés Sajcabajá, Zacualpa Parents of children in control schools 2 focus groups (14 female; 3 male) ● Program knowledge and expectations ● Program participation and take-up ● Barriers to participation ● Perceived benefits of program services San Andrés Sajcabajá, Zacualpa EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH B.2 Respondents a Method Topics of inquiry Location Observations of community activities 18 observations ● Context of community activities (location, leaders, volunteers, and participants) ● Types of activities ● Participant engagement Santa Cruz del Quiché, Chichicastenango, San Antonio Ilotenango, San Andrés Sajcabajá, Zacualpa Note: Parent focus groups took place in November 2014; volunteer focus groups and observations of community action activities in July and November 2014; teacher focus groups in March 2015; and observations of parent workshops in May 2015. APPENDIX C: CONTROL VARIABLES IN CHILD-LEVEL REGRESSION ANALYSES This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH C.1 Control variables in child-level regression analyses Table C.1 lists the control variables included in the regression models we used to estimate program impacts on reading skills of 3rd graders in the evaluation. The table also indicates whether the variables are dichotomous or continuous, as well as their sources. All the variables in this list are unlikely to be affected by the intervention, and they were measured at baseline or at the midline data collections. Table C.1. Control variables included in the analyses of the reading skills of 3rd graders in the evaluation Level Variables Type Source Student ● Child age at final follow-up assessment (in months) ● Child sex Continuous Classroom rosters Student ● Baseline letter identification score ● Baseline emergent reading score ● Baseline emergent writing score ● Baseline phonemic awareness score ● Baseline number of pseudo-words read in one minute ● Baseline passage comprehension score Continuous Leer Juntos, Aprender Juntos (LJAJ) Emergent Literacy Skills Assessment, Baseline 2013 and 2014 Student ● Child was assessed in Spanish only at baseline ● Child assessed in both Spanish and in mother tongue at baseline Binary LJAJ Emergent Literacy Skills Assessment, Baseline 2013 and 2014 Student ● Child attended preschool or kindergarten ● Child’s mother completed at least one year of formal schooling ● Child’s parents are able to read ● Language spoken at home is mother tongue only Binary LJAJ Household Survey, Midline 2014 and 2015 Student ● Number of people living in the household ● Number of rooms in the household ● Family monthly income in dollars Continuous LJAJ Household Survey, Midline 2014 and 2015 Student ● Family owns land ● Household has floor built with finished materials ● Household has roof built with finished materials ● Household has electricity ● Household has phone service ● Household has a radio ● Household has a television ● Household has a refrigerator ● Household has a computer ● Household has a bicycle ● Household has a motorcycle ● Household has a car or truck Binary LJAJ Household Survey, Midline 2014 and 2015 School ● At baseline, school participated in the National Reading Program (Leamos Juntos) ● At baseline, school participated in the Intercultural Bilingual Education and Literacy Program ● At baseline, school participated in other Ministry of Education programs ● At baseline, school participated in the Education Support and Technical Assistance Program in the Department of El Quiché (Programa Verde y Azul) Binary LJAJ School Infrastructure Observation, Baseline 2013 and 2014 EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH C.2 Level Variables Type Source ● At baseline, school participated in health promotion programs ● At baseline, school participated in the National Children’s Nutrition Program ● At baseline, school participated in other programs ● School has multigrade classrooms ● School has library or resource room ● School has computers for students ● School has computers for teachers ● School has Internet connectivity ● School has phone service ● School has piped water supply ● School has unpiped water supply ● School has potable drinking water ● School has electricity ● School is a permanent building ● School has painted interior walls ● School has painted exterior walls ● School classrooms have ceilings ● School has perimeter fence ● School has working restroom facilities for children ● School has latrines or urinals ● School has functional toilets ● School has separate facilities for boys and girls ● School has handwashing facilities ● School has working waste disposal ● School has plumbing waste disposal ● School has septic or similar disposal ● School has other waste disposal ● School has kitchen ● School has vegetable garden ● School has teachers’ lounge ● School has outdoor recreational space ● School has gymnasium or sports facilities ● School has infirmary School ● Number of physical hazards observed in school ● Number of health hazards observed in school Continuous LJAJ School Infrastructure Observation, Baseline 2013 and 2014 For details on the construction of variables from baseline and midline data sources, refer to the Leer Juntos, Aprender Juntos baseline (Lugo-Gil et al. 2016a) and midline (Lugo-Gil et al. 2017b) reports, respectively. All the variables in Table C.1 except age at baseline and sex had missing values. Five-hundred seventy-nine observations were missing data in control variables, which reduces the size of the analytic sample from 1,338 children to 759 children. To retain in the analysis all the observations that have reading outcome data (1,338 children), we imputed the missing values in control variables. For imputations, we conducted regressions using each of the variables in the list (nonmissing data) as the dependent variable. In the regressions for the baseline literacy skills and child socioeconomic characteristics variables, we used baseline age and baseline age squared as the independent variables. In the regressions for the rest of the EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH C.3 variables (household assets and school characteristics and resources), we used as independent variables the subgroup of other variables from that list that contained the least amount of missing data. This page has been left blank for double-sided printing. APPENDIX D: SCHOOL INFRASTRUCTURE CHARACTERISTICS This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH D.1 School infrastructure characteristics The utilities and facilities in evaluation schools are basic (Tables D.1), but most schools have running water, electricity, and painted interior walls. For example, although most of the schools are in a permanent building and have a piped water supply, only about half of the schools in each treatment group have a potable drinking water supply. Table D.1. Available utilities in evaluation schools (percentages of schools) Utilities Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Permanent building 98.0 100.0 100.0 -2.0 (0.226) -0.0 (0.990) Painted interior walls 100.0 98.0 95.9 2.0 (0.483) 2.1 (0.473) Painted exterior walls 100.0 98.0 95.9 2.0 (0.483) 2.1 (0.473) Perimeter fence 42.0 40.0 40.0 2.0 (0.834) 0.0 (0.998) Finished (cement, wood, etc.) classroom floors Electricity 68.0 84.0 78.0 -16.0 (0.052) 6.0 (0.468) Piped water supply 96.0 96.0 91.9 0.0 (1.000) 4.1 (0.355) Potable drinking water supply 46.0 52.0 49.0 -6.0 (0.485) 3.0 (0.729) Unpiped water supply 2.0 8.0 12.1 -6.0 (0.246) -4.1 (0.426) Plumbing waste disposal 0.0 2.0 0.0 -2.0 (0.226) 2.0 (0.225) Septic (or similar) disposal 70.0 84.0 80.1 -14.0 (0.096) 3.9 (0.640) Number of schools 50 50 49 Source: Leer Juntos, Aprender Juntos School Infrastructure Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH D.2 Almost all of the schools in each treatment group have working restroom facilities for children (Table D.2). However, few schools have amenities such as sports facilities, a music room, an infirmary, or a teachers’ lounge. Importantly, we found two differences between treatment groups in the schools’ available utilities and facilities: more schools in Group B than in Group C have functional toilets and handwashing facilities. We also found one difference between Groups A and B: fewer schools in Group A than in Group B had an outdoor recreational space. These differences were only marginally significant. Table D.2. Other facilities available in schools Facilities Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Working restroom facilities for children 96.0 94.0 94.0 2.0 (0.590) 0.0 (0.996) Separate restroom facilities for boys and girls 50.0 56.0 47.9 -6.0 (0.535) 8.1 (0.406) Latrines or urinals 68.0 66.0 73.1 2.0 (0.822) -7.1 (0.427) Functional toilets 76.0 84.0 63.7 -8.0 (0.289) 20.3* (0.008) Handwashing facilities 90.0 92.0 79.8 -2.0 (0.751) 12.2 (0.056) Kitchen 86.0 86.0 81.9 0.0 (1.000) 4.1 (0.554) Vegetable garden 10.0 16.0 21.2 -6.0 (0.400) -5.2 (0.472) Teachers' lounge 6.0 6.0 1.9 0.0 (1.000) 4.1 (0.318) Outdoor recreational space 48.0 64.0 63.1 -16.0 (0.054) 0.9 (0.918) Gymnasium or sports facilities 14.0 14.0 9.9 0.0 (1.000) 4.1 (0.387) Infirmary 0.0 0.0 2.0 0.0 (1.000) -2.0 (0.220) Computers for students 4.0 4.0 8.1 0.0 (1.000) -4.1 (0.355) Computers for teachers 0.0 2.0 2.0 -2.0 (0.393) -0.0 (0.993) Internet connectivity 0.0 0.0 2.0 0.0 (1.000) -2.0 (0.220) Library or resource room 10.0 12.0 12.0 -2.0 (0.763) -0.0 (0.998) Music room 0.0 2.0 0.0 -2.0 (0.226) 2.0 (0.225) Number of schools 50 50 49 Source: Leer Juntos, Aprender Juntos School Infrastructure Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH D.3 As we present in Table D.3, on average, there were no statistically significant differences between treatment groups in the number of hazards—related to either physical factors (such as broken glass or furniture, dangerous materials, and unsafe building structures) or health factors (such as stagnant water reservoirs)—observed in the schools. Table D.3. Physical and health hazards observed in schools Hazards Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) (A–B) (B–C) Number of physical hazards observed (average) 3.8 3.3 3.7 0.5 (0.091) -0.4 (0.163) Number of health hazards observed (average) 2.0 1.9 2.1 0.0 (0.937) -0.2 (0.534) Number of schools 50 50 49 Source: Leer Juntos, Aprender Juntos School Infrastructure Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. This page has been left blank for double-sided printing. APPENDIX E: CLASSROOM AND HOUSEHOLD ENVIRONMENTS This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.1 Classroom and household environments In this section, we present additional information on school, classrooms, and households in Guatemala that supports the findings we present in Chapter IV. 1. Schools’ participation in programs different to Leer Juntos, Aprender Juntos We measured school participation in programs other than Leer Juntos, Aprender Juntos to explore the setting in which these schools operate and gather information on what the prevailing practice offers. Schools in the evaluation participate in a number of education and nutrition programs. In particular, we gathered information about school participation in the following programs: ● Programa de Educación Bilingüe Intercultural o Alfabetización Bilingüe, an initiative of the Dirección General de Educación Bilingüe Intercultural (DIGEBI), a division of the Ministry of Education in Guatemala that is designed to promote and support bilingual education in schools at all levels. Through this initiative, the Ministry of Education provides schools with textbooks that are appropriate for teaching bilingual students, offers professional development to teachers on intercultural and bilingual instruction, provides technical assistance to school administrators, and monitors the quality of bilingual instruction in schools. ● The following Ministry of Education programs: - Escuelas del Futuro, a school intervention designed to improve schools’ infrastructure, provide schools and teachers with access to computer equipment, and promote involvement and support on educational activities from parents and the community - Programa de Apoyo a la Calidad Educativa o Proceso de Formación de Funcionarios de la Educación (PROFFE), which provides professional-development support to teachers, offers technical assistance to educators and school administrators, and receives funding from the Deutsche Gesellschaft für Internationales Zusammenarbeit (GIZ) in Germany - Abriendo Futuro, an initiative to provide financial support to teachers to acquire computers and obtain professional development in using the computers to improve instruction. This project receives support from Microsoft and the Fundación Sergio Paiz Andrade (FUNSEPA) in Guatemala ● Fortalecimiento Educativo Institucional para la Mejora de la Asistencia Técnica Administrativa en el Departamento de Quiché, which is implemented by the nonprofit organization Asociación Verde y Azul, with funds from the international development non￾governmental organization (NGO) Intervida and with the collaboration of Guatemala’s Ministry of Education. This program provides culturally appropriate teaching materials, coaching, and technical assistance to educators and school administrators, and support for school infrastructure improvements. ● Other programs, including: - Participation in professional development provided by private organizations EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.2 - Implementation of the Nueva Escuela Unitaria Bilingüe (NEUBI) model, a teaching model tailored to the students’ cultural and language context that promotes interaction of the students with their teacher and classmates during instruction - Hambre 0, an initiative of the government of Guatemala to reduce malnutrition in children and eliminate extreme poverty in the country - Physical education initiatives from the Ministry of Education ● Programa Infantil de Nutrición, a program provided by the Ministry of Public Health in Guatemala that offers free breakfast, lunch, and/or snacks for children in participating schools ● Health promotion programs, including: - Escuelas Saludables, a program implemented by the Peace Corps and the Ministry of Education in Guatemala that offers hygiene education and provides assistance to schools to obtain the necessary infrastructure to create a healthy environment for students and teachers - A program supporting access to safe drinking water, improved sanitation, and hygiene education, provided by Water for People, an international, nonprofit, humanitarian organization that works in the Department of El Quiché 2. The Educational Network in the Department of El Quiché Table E.1 presents a list of the organizations, services, and target municipalities that are part of the Educational Network (Red Educativa) in the Department of El Quiché. Table E.1. Members, services, and municipalities in the Educational Network in the Department of El Quiché Organization Services Municipalities Verde y Azul ● Pedagogical materials ● School furniture ● Water filters ● Teacher training Santa Cruz, Chiché, Chichicastenango y San Pedro Jocopilas Leer Para Aprender ● Reading and writing instruction ● Education for out-of-school youth Joyabaj, San Pedro Jocopilas Proyecto de Desarrollo Santiago (PRODESSA) ● Teacher training and coaching ● Pedagogical materials Santa Cruz, Chiché, Chichicastenango y San Pedro Jocopilas, Nebaj, Uspantán y Zona Reina AKEBI ● Community school involvement ● School meals ● School vegetable gardens Santa Cruz, Chiché, Chichicastenango, San Pedro Jocopilas, Zona Reina, Uspantàn, Chicamàn (Belejù) School the World ● School libraries ● School infrastructure and construction Santa Cruz, Chichicastenango, Chiché y Chinique Corazones y Manos ● Teacher training and coaching for reading instruction Uspantán y Chicamán EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.3 Korea International Cooperation Agency (KOIKA) ● Teacher training and coaching ● School texts ● Monitoring Cotzal Programa Educación para la Vida y el Trabajo (EDUVIDA) ● Pedagogical materials for social sciences ● School-to-work training Uspantán, Nebaj, Chinique, Santa Cruz, Patzité, Pachalum, Chichicastenango, Chajul Programa Académico de Desarrollo Profesional (PADEP) ● Teacher training and coaching N/A Child Fund ● Teacher training on school readiness, reading instruction (“posicionamiento” method) ● Early grade reading Patzitè Lápices de Promesa ● School construction Cunén, Uspantán CEFA Onlus ● Scholarships for girls Sacapulas, San Pedro Jocopilas, Zacualpa, Chichè y Chichicastenango Water for People ● Teacher training on health and hygiene ● Construction of water and sanitation infrastructure ● Patent training on health and hygiene ● Management, operation and maintenance of water and sanitation infrastructure San Antonio Ilotenango, Santa Cruz del Quiché, San Andrés Sajcabajá, San Bartolomé Jocotenango, Santo Tomás Chiché y Santo Tomás Chichicastenango Source: DIDEDUC (2014). Mapeo de la red educativa del departamento de Quiché. Ministerio de Educación de Guatemala, Subdirección Técnica Pedagógica de Educación Intercultural Bilingüe. 3. Participation in professional development activities Results from teacher survey data (see Table E.2) show that, on average, more teachers in Groups A and B (schools implementing the training and coaching component of the intervention) participated in reading instruction professional development (PD) activities than in Group C (prevailing practice schools) during the year of administration of the teacher survey (2015 for Phase I teachers and 2016 for Phase II teachers). The difference in reported rates of participation in PD activities between teachers in Groups B and C was statistically significant (Table E.2). Additionally, a larger percentage of teachers in the two intervention groups than in the prevailing practice group reported receiving coaching, and the difference between Groups B and C was statistically significant (30 percent, shown in Table E.2). The reported rates of participation in PD in the treatment groups (Groups A and B) were not closer to 100 percent, because of the timing of the endline (final follow-up) data collection. By the time the teacher survey was administered in Phase II schools, Save the Children was no longer offering PD activities (the activities ended in March 2016). However, Save the Children staff provided training to staff at the Departmental Education Directorate (DDE) in the Department of El Quiché so that they could continue implementing Leer Juntos, Aprender Juntos activities such as the training and coaching for teachers. Therefore, teachers in the evaluation could have continued receiving training and coaching based on the Leer Juntos, Aprender Juntos model at the time we collected data for the final follow-up, but we do not know the intensity of that support or its fidelity to the program. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.4 In terms of the providers of the PD activities, most Group A and Group B teachers reported receiving training from Save the Children (through Leer Juntos, Aprender Juntos), whereas most Group C teachers received their training from the Ministry of Education. Considering the entire period of program implementation, the participation of intervention teachers (Groups A and B) in PD activities was considerable. For instance, in the midline data collection (July, September– November 2014 in Phase I and June–July 2015 in Phase II), more than 90 percent of the intervention teachers (96 percent in Group A and 92 percent in Group B) reported they participated in PD activities in the 12 months before the midline survey administration (Lugo-Gil et al. 2017b). Also, during the year before the endline survey administration (2014 for Phase I and 2015 for Phase II teachers), the percentage of teachers in the intervention groups who reported participating in PD activities was 78 percent in Group A and 72 in Group B, and the difference in the participation rate of Group B teachers with teachers in Group C was 37 percentage points, which is statistically significant. We present the findings on teachers’ reports of participation in PD activities in the year before the endline teacher survey in Table E.3. Table E.2. Teacher-reported participation in reading instruction professional development (PD) activities in the endline year (percentages of teachers) Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Teachers who participated in reading instruction professional development activitiesa 70.0 62.0 29.3 8.0 (0.329) 32.7* (0.000) Teachers who reported they received reading instruction professional development from: Ministry of Education or other institution 22.0 16.0 28.2 6.0 (0.414) -12.2 (0.101) Save the Children through Leer Juntos, Aprender Juntos 50.0 48.0 1.0 2.0 (0.754) 47.0* (0.000) Teachers who reported they received coaching 48.0 50.0 20.4 -2.0 (0.831) 29.6* (0.002) Teachers who reported they received in-class coaching from: Ministry of Education or other institution 10.0 14.0 20.0 -4.0 (0.575) -6.0 (0.401) Save the Children through Leer Juntos, Aprender Juntos 44.0 42.0 12.4 2.0 (0.817) 29.6* (0.001) Total number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. a The percentages shown refers to the participation during the year when the endline survey was administered (2015 for Phase I teachers and 2016 for Phase II teachers). * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.5 Table E.3. Teacher-reported participation in reading instruction professional development (PD) activities in the year before endline survey administration (percentages of teachers) Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Teachers who participated in reading instruction PD activities 78.0 72.0 35.2 6.0 (0.489) 36.8* (0.000) Teachers who reported they received coaching 66.0 56.0 24.3 10.0 (0.322) 31.7* (0.002) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.6 At the endline or final follow-up, teachers in the intervention groups (Groups A and B) also reported higher rates of participation in reading instruction PD activities focused on topics that are addressed in the Leer Juntos, Aprender Juntos training. As we show in Figure E.1, 56 percent or more of the teachers in Groups A and B reported receiving training on teaching letter knowledge, reading fluency, reading comprehension, vocabulary, and phonological awareness. In contrast, only 18 to 28 percent of the teachers in the prevailing practice group reported participating in PD focused on teaching specific literacy skills. The difference between Groups B and C is statistically significant for all types of PD focused on teaching specific literacy skills. Figure E.1. Rates of participation in professional development (PD) activities focused on specific literacy skills Source: Leer Juntos, Aprender Juntos Teacher Survey— Final Follow-ups 2015 and 2016. * Difference in group means is statistically significant at the 0.05 level. Teachers also reported participation in PD activities focused on other topics related to reading instruction in the endline or final follow-up year: as we show in Table E.4, approximately 60 percent of teachers in each of the intervention groups reported participation in PD focused on teaching reading in K’iche’ or teaching bilingual students. In contrast, only 19 percent of teachers in the prevailing practice group reported participating in training focused on reading in K’iche’ or teaching bilingual children. About 42 to 62 percent of teachers in each of the intervention groups reported participating in PD focused on making or adapting materials to teach reading, on creating a suitable classroom environment for learning to read, and on the use of students’ evaluations to track their progress in reading. Those differences between Groups B and C were statistically significant (see Table E.4). EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.7 Table E.4. Teacher-reported participation in reading instruction professional development (PD) activities focused on specific topics in the endline year Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in-school component (B–C) Percentage of teachers who participated in other PD focused on: Teaching reading in K’iche’ or teaching bilingual students 60.0 58.0 19.2 2.0 (0.811) 38.8* (0.000) Support for literacy activities at home 46.0 38.0 12.0 8.0 (0.327) 26.0* (0.002) Classroom management 46.0 36.0 14.0 10.0 (0.237) 22.0* (0.010) Making or adapting materials to teach reading 62.0 60.0 18.0 2.0 (0.804) 42.0* (0.000) Creating a suitable classroom environment for learning reading 50.0 42.0 8.0 8.0 (0.320) 34.0* (0.000) Use of regular evaluations 58.0 52.0 18.0 6.0 (0.494) 34.0* (0.000) Parent/community participation in reading activities 32.0 20.0 6.0 12.0 (0.111) 14.0 (0.064) Other curricular areas 38.0 30.0 20.0 8.0 (0.332) 10.0 (0.226) Management of multiple-grade classrooms 34.0 26.0 8.0 8.0 (0.307) 18.0* (0.023) Other related to literacy instruction 8.0 6.0 6.0 2.0 (0.669) 0.0 (1.000) Total number of teachers 50 50 49 . . Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. 4. Teacher-reported practices We asked teachers about literacy-related practices they use in their classrooms, such as the types of reading materials that students use and the frequency with which they assess their students’ literacy skills. Most of the teachers in each treatment group (56 percent in Group A, 54 percent in Group B, and 60 percent in Group C) reported that their students use textbooks every day (Table E.5), but we did not find statistically significant differences between groups. We found one statistically significant difference in the types of reading materials students use: more teachers in Group B (50 percent) than in Group A (30 percent) reported their students use simple reading books every day. In general, more teachers in the treatment groups (Groups A and B) than in in the prevailing practice group reported that their students use materials that provide reading opportunities, such as workbooks or exercise guides and cards with syllables or words. Although these differences were not statistically significant, these results are encouraging, because the use of reading books and materials strengthens the development of decoding, fluency, and reading comprehension skills for 3rd grade students. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.8 Table E.5. Students’ use of different types of text, as reported by teachers Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Percentage of teachers who reported their students use the following types of text every day: School textbooks 56.0 54.0 60.1 2.0 (0.847) -6.1 (0.559) Workbooks or exercise guides 32.0 40.0 29.9 -8.0 (0.414) 10.1 (0.305) Picture or artwork books 20.0 16.0 22.1 4.0 (0.622) -6.1 (0.458) Books to name objects or letters of the alphabet 18.0 28.0 32.2 -10.0 (0.250) -4.2 (0.632) Simple reading books 30.0 50.0 44.1 -20.0* (0.037) 5.9 (0.535) Cards with syllables or words 20.0 22.0 15.9 -2.0 (0.805) 6.1 (0.456) Books with chapters 8.0 14.0 10.0 -6.0 (0.333) 4.0 (0.520) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.9 Teachers also reported on the time their students spend on reading-related activities during a typical language class (Table E.6). In all treatment groups, teachers reported that their students spend the most time practicing reading fluency, building vocabulary and new words, and practicing reading comprehension. Based on these teacher reports, we found no statistically significant difference between treatment groups in the amount of time students spend on different activities during a typical language class (Table E.6). Table E.6. Number of minutes students spend on the following activities during a typical language and reading class, as reported by teachers Activity Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Learning to identify the letters of the alphabet 20.4 18.8 23.0 1.6 (0.628) -4.2 (0.213) Practicing phonemic awareness activities 16.6 20.2 20.6 -3.6 (0.201) -0.4 (0.875) Practicing reading fluency 23.8 27.8 27.2 -4.0 (0.154) 0.6 (0.839) Practicing reading comprehension 24.0 25.0 25.2 -1.0 (0.694) -0.2 (0.933) Building vocabulary and learning new words 26.6 25.2 25.6 1.4 (0.649) -0.4 (0.899) Narrating familiar stories without reference to text 22.2 20.6 24.8 1.6 (0.478) -4.2 (0.068) Copying letters or words 19.4 17.4 20.3 2.0 (0.359) -2.9 (0.182) Taking dictation 19.0 19.0 19.5 0.0 (1.000) -0.5 (0.836) Motivational activities 16.6 18.0 15.8 -1.4 (0.469) 2.2 (0.254) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. Interviewers also asked teachers about the needs and challenges they face in teaching reading (Table E.7). The most-mentioned barriers by teachers in intervention groups (Groups A and B) were the lack of training in intercultural bilingual education and the lack of teaching skills in K’iche’. There were no statistically significant differences between treatment groups in those reports. However, more teachers in Group C (84 percent) than in Group B (66 percent) cited the lack of training in methods to teach literacy as a barrier to teach reading. This difference was significant at the 0.10 level (p-value = 0.054). Table E.7 also shows the teachers’ reports on the resources that they perceive as lacking when they teach reading to their students and the issues they reported as challenges to teaching reading. In all treatment groups, teachers’ most￾mentioned barrier to teaching reading was students’ absenteeism. We found no statistically significant difference between treatment groups in the challenges that teachers reported in teaching reading. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.10 Table E.7. Teacher-reported needs and challenges (percentages of teachers) Needs and challenges Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Lack of supports and resources for teaching reading and writing Lack of books 66.0 64.0 77.2 2.0 (0.834) -13.2 (0.171) Lack of teaching resources 72.0 64.0 72.1 8.0 (0.334) -8.1 (0.333) Lack of sufficient instructional or class time 64.0 60.0 53.8 4.0 (0.699) 6.2 (0.554) Lack of training in methods to teach literacy 62.0 66.0 84.4 -4.0 (0.670) -18.4 (0.054) Lack of other types of training 64.0 58.0 71.2 6.0 (0.548) -13.2 (0.192) Lack of support from parents 66.0 58.0 65.1 8.0 (0.420) -7.1 (0.480) Lack of training in intercultural bilingual education 66.0 76.0 69.0 -10.0 (0.299) 7.0 (0.467) Lack of classroom equipment or material 70.0 68.0 72.1 2.0 (0.826) -4.1 (0.658) Lack of K’iche’-language teaching skills 72.0 64.0 60.9 8.0 (0.405) 3.1 (0.745) Other 4.0 14.0 8.0 -10.0 (0.067) 6.0 (0.272) Challenges to teaching reading Students’ absenteeism 88.0 82.0 88.1 6.0 (0.380) -6.1 (0.379) Students' lack of motivation 62.0 62.0 63.0 0.0 (1.000) -1.0 (0.914) Students’ lack of Spanish-language knowledge 46.0 44.0 51.1 2.0 (0.833) -7.1 (0.457) Lack of parents’ involvement in school activities 76.0 62.0 77.2 14.0 (0.105) -15.2 (0.082) Students’ malnutrition or poor health 70.0 58.0 58.9 12.0 (0.204) -0.9 (0.925) Students’ vision problems 32.0 22.0 21.9 10.0 (0.253) 0.1 (0.991) Number of teachers 50 50 49 Source: Leer Juntos, Aprender Juntos Teacher Survey—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.11 5. Observed practices To measure observed teaching practice, the evaluation team observed the instruction in the 3rd grade classrooms during a typical language or reading class. On average, teachers in the three treatment groups were observed for 41 minutes (there were no statistically significant differences between treatment groups in the average time of the classroom observations). Teachers’ use of general teaching practices such as providing verbal reinforcement and feedback to students was similar across treatment groups. Most of the teachers in each treatment group gave students opportunities to respond to the print/text materials and asked open-ended questions to stimulate oral language. Most of the teachers in all treatment groups (at least 82 percent) also provided verbal reinforcement and feedback to students. None of the differences between treatment groups was statistically significant (Table E.8). Table E.8. Observed general instructional practices Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Percentage of classrooms in which teachers: Stimulated oral language by Giving students opportunities to respond to the print/text materials 90.0 88.0 79.8 2.0 (0.774) 8.2 (0.245) Asking open-ended questions 70.0 60.0 74.2 10.0 (0.267) -14.2 (0.119) Provided verbal reinforcement to students 82.0 84.0 84.0 -2.0 (0.777) 0.0 (0.998) Provided appropriate feedback to students 86.0 88.0 92.1 -2.0 (0.763) -4.1 (0.539) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between treatment groups presented in this table is statistically significant at the 0.05 level. 6. Language use in the classroom Spanish was the spoken language that teachers used most often to communicate with their students, but we did not find statistically significant differences in teachers’ language use between treatment groups. As we show in Table E.9, teachers in at least 76 percent of the classrooms in each treatment group were observed using only Spanish or mostly Spanish to discipline and manage students’ behavior, and most of the observed teachers in all treatment groups used Spanish to provide literacy instruction. In contrast, teachers were observed using only K’iche’ or mostly K’iche’ to communicate with their students in only 2 to 4 percent of the classrooms in each treatment group. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.12 Table E.9. Observed teachers’ use of language in the classroom Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Classrooms in which teachers were observed using only Spanish or mostly Spanish to: Discipline and manage students’ behavior 76.0 80.0 77.0 -4.0 (0.628) 3.0 (0.717) Provide literacy instruction 92.0 90.0 90.0 2.0 (0.703) 0.0 (0.997) Classrooms in which teachers were observed using only K’iche’ or mostly K’iche’ to: Discipline and manage students’ behavior 0.0 4.0 0.0 -4.0 (0.085) 4.0 (0.084) Provide literacy instruction 2.0 4.0 2.0 -2.0 (0.547) 2.0 (0.546) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between evaluation means presented in this table is statistically significant at the 0.05 level. In most of the observed classrooms (at least 84 percent in each treatment group), half or more of the students used Spanish to communicate with their teacher. However, in about a third of the classrooms, half or more of the students communicated with their teacher in K’iche’. We did not find any statistically significant differences between treatment groups in the language use of students in the classroom (Table E.10). Table E.10. Observed students’ use of language in the classroom Blank cell Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Classrooms in which half or more of the students speak to the teacher in: Spanish 84.0 88.0 84.0 -4.0 (0.590) 4.0 (0.589) K’iche’ 32.0 24.0 26.0 8.0 (0.384) -2.0 (0.827) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between evaluation means presented in this table is statistically significant at the 0.05 level. 7. Classroom environment Table E.11 shows the books and materials in Spanish and K’iche’ observed in the 3rd grade classrooms in the evaluation. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH E.13 Table E.11. Books and materials observed in the classroom Books and materials Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Percentage of classrooms in which: More than 25 books in Spanish were observed 54.0 36.0 43.0 18.0 (0.060) -7.0 (0.467) More than 25 books in K’iche’ were observed 16.0 20.0 28.2 -4.0 (0.603) -8.2 (0.291) 10 or more of the following were observed: In Spanish Printed materials 18.0 16.0 9.9 2.0 (0.762) 6.1 (0.356) Handmade materials prepared by the teacher 36.0 32.0 28.9 4.0 (0.682) 3.1 (0.753) Handmade materials prepared by the students 32.0 38.0 26.8 -6.0 (0.505) 11.2 (0.220) In K’iche’ Printed materials 2.0 2.0 -0.0 0.0 (1.000) 2.0 (0.387) Handmade materials prepared by the teacher 8.0 10.0 3.9 -2.0 (0.715) 6.1 (0.270) Handmade materials prepared by the students 2.0 10.0 4.0 -8.0 (0.086) 6.0 (0.197) Number of classrooms 50 50 49 Source: Leer Juntos, Aprender Juntos Classroom Observation Form—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between evaluation means presented in this table is statistically significant at the 0.05 level. This page has been left blank for double-sided printing. APPENDIX F: STANDARD DEVIATIONS OF CHILDREN’S READING SKILLS MEASURES This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH F.1 Standard deviations of children’s reading skills measures Table F.1 presents the standard deviations of the reading skills measures (Chapter V) we used to examine the impacts of the components of Leer Juntos, Aprender Juntos. Table F.1. Standard deviations of children’s reading skills outcome measures Outcome Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Decoding Number of pseudo-words read correctly in one minute (of 50) 15.04 14.38 13.38 Accuracy score (0 to 1) 0.35 0.31 0.29 Fluency Number of words read correctly in one minute (of 112) 30.18 28.54 28.86 Accuracy score (0 to 1) 0.34 0.29 0.28 Reading comprehension Number of questions answered correctly (of 15) 3.24 3.06 3.11 Percentage of children who answered at least one question in the comprehension task correctly (achieved basic reading comprehension capabilities) 0.49 0.48 0.47 Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. This page has been left blank for double-sided printing. APPENDIX G: PROGRAM IMPACTS IN EFFECT SIZES This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH G.1 Program impacts in effect sizes Table G.1 presents the estimated impacts of each of the program components in terms of effect sizes (or standardized mean difference between Groups A and B, and between Groups B and C). Besides not being statistically significant, most of the effect sizes shown in Table G.1 are small (smaller than 0.1 standard deviations). We did find a statistically significant difference between Groups A and B in fluency accuracy scores, which translated into an effect size of 0.16 standard deviations in absolute value. A recent systematic review of evaluations studies on reading interventions in the developing world (Kim et al. 2016) found that the impacts in effect sizes of reading interventions in the studies they reviewed ranged from 0.14 to 0.73 for reading fluency outcome measures, and from 0.05 to 0.58 for reading comprehension outcome measures. Table G.1. Impacts on literacy outcomes in standard deviation units Literacy outcome Effect size of community action component Effect size of in-school component Decoding Number of pseudo-words read correctly in one minute -0.07 0.08 Accuracy score -0.11 -0.02 Fluency Number of words read correctly in one minute -0.07 0.06 Accuracy score -0.16* 0.02 Reading comprehension Number of questions answered correctly 0.06 0.02 Percentage of children who achieved basic reading comprehension skills -0.03 -0.02 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Effect sizes are calculated as the difference in adjusted means between two treatment groups (A and B, or B and C) divided by the pooled and weighted standard deviation of the two groups (Hedges’ g) for continuous outcome measures, and as the log odds ratio divided by 1.65 (Cox index) for dichotomous outcome measures * Effect is statistically significant at the 0.05 level. H.1 This page has been left blank for double-sided printing. APPENDIX H: SENSITIVITY ANALYSES This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH H.1 Sensitivity analyses As we mention in Chapter V, we conducted robustness checks on two important assumptions we made in the estimation of program impacts. First, we reweighted the data to account for differences in the number of children who completed the 3rd grade literacy skills assessment (“respondents”) and to avoid underweighting students whose classmates were absent. We reweighted the data of each child i in the 3rd grade in school s using the following formula: 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑤𝑤𝑁𝑁 𝑜𝑜𝑜𝑜 𝑠𝑠𝑡𝑡𝑁𝑁𝑠𝑠𝑤𝑤𝑠𝑠𝑡𝑡𝑠𝑠 𝑤𝑤𝑠𝑠 𝑡𝑡ℎ𝑤𝑤 𝑠𝑠𝑠𝑠𝑁𝑁𝑠𝑠𝑠𝑠𝑤𝑤 𝑤𝑤𝑠𝑠 𝑠𝑠𝑠𝑠ℎ𝑜𝑜𝑜𝑜𝑠𝑠 𝑠𝑠 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤ℎ𝑡𝑡𝑖𝑖𝑖𝑖 = 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑤𝑤𝑁𝑁 𝑜𝑜𝑜𝑜 𝑁𝑁𝑤𝑤𝑠𝑠𝑠𝑠𝑜𝑜𝑠𝑠𝑠𝑠𝑤𝑤𝑠𝑠𝑡𝑡𝑠𝑠 𝑡𝑡𝑜𝑜 𝑡𝑡ℎ𝑤𝑤 𝑠𝑠𝑠𝑠𝑠𝑠𝑤𝑤𝑠𝑠𝑠𝑠𝑁𝑁𝑤𝑤𝑠𝑠𝑡𝑡 𝑤𝑤𝑠𝑠 𝑠𝑠𝑠𝑠ℎ𝑜𝑜𝑜𝑜𝑠𝑠 𝑠𝑠 By reweighting the data, our results on regression-adjusted mean differences between treatment groups will correspond to the mean reading outcomes of the average respondent in the schools in the sample. The results do not vary when we estimate program effects based on the reweighted data. Table H.1 presents the regression-adjusted means of the literacy assessment skills we assessed in 3rd grade, by treatment group. As the table shows, the main result we present in Chapter V still holds when we account for students’ nonresponse: there were no positive, statistically significant impacts of either component of the program. Table H.1. Differences between treatment groups in children’s literacy outcomes, accounting for nonresponse to the 3rd grade assessment Outcome Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Decoding Number of pseudo-words read correctly in one minute (of 50) 21.7 22.6 21.6 -0.9 (0.339) 1.0 (0.265) Accuracy score (0 to 100) 69.0 72.2 72.7 -3.2 (0.155) -0.5 (0.838) Fluency Number of words read correctly in one minute (of 112) 41.8 43.5 41.9 -1.7 (0.396) 1.6 (0.346) Accuracy score (0 to 100) 74.8 79.5 78.5 -4.7* (0.036) 1.0 (0.660) Reading comprehension Number of questions answered correctly (of 15) 3.1 2.9 2.9 0.2 (0.261) 0.0 (0.781) Percentage of children who answered at least one question in the comprehension task correctly (achieved basic reading comprehension capabilities) 63.1 64.1 65.1 -1.0 (0.725) -1.0 (0.741) Number of children 439 458 441 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH H.2 For a second check on the robustness of our assumptions, we also calculated differences between treatment groups in mean literacy outcomes in the 3rd grade based on the sample of children that has complete data for all the control variables included in the regression model. The sample with complete data in literacy outcome variables and control variables includes 759 children. In comparison, the sample that has complete data on literacy outcome variables but has incomplete data in control variables (for which we imputed in the analyses presented in Chapter V) includes 1,338 children. Using complete data reduces the sample size by about 43 percent. As Table H.2 shows, the findings do not change when we use a sample with complete data. In particular, under this alternative assumption, we also found that there were no positive, statistically significant impacts of either component of the program. Table H.2. Differences between treatment groups in children’s literacy outcomes, based on sample with complete data Outcome Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community action (A–B) Impact of in￾school component (B–C) Decoding Number of pseudo-words read correctly in one minute (of 50) 21.7 22.6 22.6 -0.9 (0.465) 0.0 (0.976) Accuracy score (0 to 100) 69.6 72.5 73.8 -2.9 (0.525) -1.3 (0.606) Fluency Number of words read correctly in one minute (of 112) 43.0 43.3 43.9 -0.3 (0.910) -0.6 (0.813) Accuracy score (0 to 100) 74.5 78.4 79.6 -3.9 (0.116) -102 (0.665) Reading comprehension Number of questions answered correctly (of 15) 3.3 3.0 3.0 0.3 (0.213) 0.0 (0.930) Percentage of children who answered at least one question in the comprehension task correctly (achieved basic reading comprehension capabilities) 63.4 66.1 66.8 -2.7 (0.525) -0.7 (0.864) Number of children 250 244 265 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between evaluation means presented in this table is statistically significant at the 0.05 level. We also examined program impacts on literacy outcomes for subgroups of students defined by phase (cohort). Table H.3 presents the findings from that analysis. As we did with the main analysis approach, in the analysis by phase, we did not find positive, statistically significant impacts of any of the components of the program in either Phase I or Phase II. We did find that in Phase II, children in Group B had statistically significantly better decoding and fluency accuracy scores than their counterparts in Group A. We also did not find any statistically significant differences in the program impacts on children’s reading outcomes between Phase I and Phase II. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH H.3 Table H.3. Impacts on the literacy outcomes of children in Phase I and Phase II schools Outcome Leer Juntos (A) Leer Juntos– school only (B) Prevailing practice (C) Impact of community actiona (A–B) Impact of in￾school componentb (B–C) Children in Phase I schools Decoding Number of pseudo-words read correctly in one minute (of 50) 24.4 26.1 25.5 -1.7 (0.146) 0.6 (0.637) Accuracy score (0 to 100) 75.7 76.9 76.9 -1.2 (0.648) 0.0 (0.987) Fluency Number of words read correctly in one minute (of 112) 47.8 49.1 50.1 -1.3 (0.576) -1.0 (0.667) Accuracy score (0 to 100) 81.7 82.5 83.2 -0.8 (0.776) -0.7 (0.830) Reading comprehension Number of questions answered correctly (of 15) 3.8 3.8 3.7 0.0 (0.874) 0.1 (0.749) Percentage of girls who achieved basic reading comprehension skills 70.8 77.0 76.8 -6.2 (0.082) 0.2 (0.968) Children in Phase II schools Decoding Number of pseudo-words read correctly in one minute (of 50) 19.1 19.4 17.7 -0.3 (0.832) 1.7 (0.202) Accuracy score (0 to 100) 61.4 67.9 69.6 -6.5* (0.027) -1.7 (0.561) Fluency Number of words read correctly in one minute (of 112) 36.4 38.1 33.6 -1.8 (0.522) 4.5 (0.080) Accuracy score (0 to 100) 67.1 76.3 74.8 -9.2* (0.001) 1.5 (0.607) Reading comprehension Number of questions answered correctly (of 15) 2.4 2.0 1.9 0.4 (0.113) 0.1 (0.669) Percentage of boys who achieved basic reading comprehension skills 55.8 52.2 53.5 3.6 (0.340) -1.3 (0.755) Number of children in Phase I schools 228 225 214 Number of children in Phase II schools 211 233 227 Source: Leer Juntos, Aprender Juntos Students’ Reading Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. a The differences between Phase I and Phase II in impacts of the community of action component were not statistically significant for any reading outcomes. b The differences between Phase I and Phase II in impacts of the in-school component were not statistically significant for any reading outcomes. * Difference in group means is statistically significant at the 0.05 level. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH H.4 Finally, Table H.4 presents our findings on the effects of the full intervention (the community action component and the teacher training and coaching component) on children’s reading outcomes. We did not find any statistically significant differences (at the 0.05 level) between Groups A and C in any of the children’s reading skills measures that we examined. We did find, however, that children in schools in Group C had slightly higher decoding and fluency accuracy scores, on average, than children in schools in Group A, and those differences are statistically significant at the 0.10 level (p = 0.052 for the decoding accuracy score and p = 0.065 for the fluency accuracy score). Table H.4. Impacts of the full intervention Outcome Leer Juntos (A) Prevailing practice (C) Impact of full intervention (A–C) Decoding Number of pseudo-words read correctly in one minute (of 50) 21.6 21.7 0.0 (0.973) Accuracy score (0 to 100) 68.8 73.1 -4.3 (0.052) Fluency Number of words read correctly in one minute (of 112) 41.7 42.0 -0.3 (0.863) Accuracy score (0 to 100) 74.5 78.9 -4.4 (0.065) Reading comprehension Number of questions answered correctly (of 15) 3.1 2.8 0.3 (0.092) Percentage of children who answered at least one question in the comprehension task correctly (achieved basic reading comprehension capabilities) 63.1 65.3 -2.2 (0.480) Number of children 439 441 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Regression-adjusted means. P-values in parentheses. None of the differences between evaluation means presented in this table is statistically significant at the 0.05 level. APPENDIX I: IMPACTS OF THE PROGRAM COMPONENTS FOR BOYS AND GIRLS IN TERMS OF EFFECT SIZES This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH I.1 Impacts of the program components for boys and girls in terms of effect sizes As we describe in Chapter V, we did not find any positive, statistically significant differences between Groups A and B (effect of the community component), or between Groups B and C (effect of the in-school component), in the literacy outcomes of the evaluation’s girls and boys in the 3rd grade. Table I.1 presents the differences between Groups A and B in the means of the literacy outcomes of the evaluation’s boys and girls, expressed in standard deviation units (effect sizes). Table I.1. Community component effects on literacy outcomes for girls and boys, in standard deviation units Outcome Effect size for girls Effect size for boys Decoding Number of pseudo-words read correctly in one minute -0.14 0.00 Accuracy score (0 to 1) -0.15 -0.07 Fluency Number of words read correctly in one minute -0.16* 0.03 Accuracy score (0 to 1) -0.24* -0.07 Reading comprehension Number of questions answered correctly 0.00 0.12 Percentage of children who achieved basic reading comprehension skills -0.06 0.00 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Effect sizes are calculated as the difference in adjusted means between two treatment groups (in this table, Groups A and B) divided by the pooled and weighted standard deviation of the two groups (Hedges’ g) for continuous outcome measures, and as the log odds ratio divided by 1.65 (Cox index) for dichotomous outcome measures. * Effect is statistically significant at the 0.05 level. Table I.2 presents the differences between Groups B and C in the means of the literacy outcomes of the evaluation’s boys and girls, also in terms of effect sizes. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH I.2 Table I.2. In-school component effects on literacy outcomes for girls and boys, in standard deviation units Literacy outcome Effect size for girls Effect size for boys Decoding Number of pseudo-words read correctly in one minute 0.09 0.06 Accuracy score (0 to 1) -0.05 0.01 Fluency Number of words read correctly in one minute 0.07 0.05 Accuracy score (0 to 1) 0.07 -0.02 Reading comprehension Number of questions answered correctly 0.05 -0.01 Percentage of children who achieved basic reading comprehension skills 0.03 -0.08 Source: Leer Juntos, Aprender Juntos Students’ Literacy Skills Assessment—Final Follow-ups 2015 and 2016. Note: Effect sizes are calculated as the difference in adjusted means between two treatment groups (in this table, Groups A and B) divided by the pooled and weighted standard deviation of the two groups (Hedges’ g) for continuous outcome measures, and as the log odds ratio divided by 1.65 (Cox index) for dichotomous outcome measures. None of the effect sizes presented in this table is statistically significant at the 0.05 level. APPENDIX J: DATA COLLECTION INSTRUMENTS This page has been left blank for double-sided printing. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.1 J.1. Survey instruments The baseline, midline, and endline survey instruments are available online upon request at USAID’s Development Data Library website (https://data.usaid.gov/) EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.2 J.2. Qualitative data collection instruments This section includes the following qualitative data collection instruments 1. Save the Children staff interview instrument 2. Save the Children specialist interview instrument 3. Community volunteers interview instrument 4. Teachers in treatment Groups A and B focus group protocol 5. Teachers in Group C focus group protocol 6. Parents in treatment Groups A and B focus group protocol 7. Community action component activities observation instrument EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.3 1. LEER JUNTOS, APRENDER JUNTOS: SAVE THE CHILDREN STAFF A. ROLES AND RESPONSIBILITIES 1. What is your role and what are your responsibilities in the implementation of LJAJ? What were some of your primary activities in the LJAJ implementation? 2. How/when did you learn the program model and intervention goals? What type of support did you receive to understand the program goals? ● Probe: hires dedicated exclusively for LJAJ are more committed or steeped into the program? How much of the original intervention model gets lost in translation as it is adapted and implemented by different organizations? 3. How is the implementation team organized? ● Probe: communication line, decision-making, changes to the organization during the project 4. What challenges in the LJAJ implementation have you encountered in your role? How have you handled them? B. ADAPTATION OF THE LITERACY BOOST MODEL TO PERU 5. How was the Literacy Boost model adapted to Guatemala, particularly to the sociocultural and linguistic context of Quiché? ● Probe: who, when, how was it done; explore adaptations to the teacher training and the community action components 6. How is the K’iche’ language and culture incorporated into the program? 7. What challenges came up during the cultural adaptation? How were those challenges addressed? Has the model or the teaching strategies evolved over the course of the project? How? C. FIDELITY 8. How was the teacher training component organized and how did it take place? What have been the main challenges in training and coaching teachers? How were those addressed? ● Probe: explore specific challenges by activity type: book banks, book fairs, reading buddies, reading camps 9. How was the reading assessment component of the program implemented? What were the challenges in administering or using assessment data? How was the reading assessment data used? ● Probe: were teachers able to use assessment data to inform instruction? How did you provide assessment data to teachers, how they received it? Did they use it? 10. How has LJAJ been implemented in phase I and phase II schools? What are the main differences between phases? In what ways has implementation of LJAJ varied from school to school? 11. What aspects of the program did not take place according to plan or to SC guidelines? D. SC KEY STAFF’S PERCEPTIONS ON PROGRAM TAKE UP AND IMPACT 12. How have school principals, teachers, parents and children received the LJAJ activities? EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.4 13. What challenges have come up with parents’ and children’s participation in program activities? How were those challenges addressed? 14. In what ways do you think LJAJ can be improved to better serve teachers and students in that region? 15. How can program take up be improved? What changes would be needed? E. PROGRAM GOALS AND PERCEIVED IMPACTS 16. Is the program reaching the intended target population, and, if not, why not? 17. How do you track and measure the program’s goals? ● Probes: how are benchmarks set and tracked? How have you adjusted to meet program goals? 18. Do you think students in the program are actually reading better? How has the program benefited children and families? ● Do you think students in the program are actually spending more time reading at home and at school? D. WRAP UP 19. In your view, what aspects of the program have been the most successful ones, and which ones the least? 20. What recommendation would you make to future implementers if this programs were to be scaled up or implemented in other regions in Guatemala or other countries with indigenous populations? ¡Thanks for your collaboration! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.5 2. LEER JUNTOS, APRENDER JUNTOS: VOLUNTARIOS Presentación Gracias por aceptar reunirse con nosotros el día de hoy. Como ustedes saben, Mathematica Policy Research, una firma de investigación y evaluación independiente con sede en los EE.UU., está trabajando con Save the Children para llevar a cabo una evaluación del Programa “Leer Juntos, Aprender Juntos (LJAJ)” y para entender los programas educativos de la región de Quiché. El propósito de nuestra conversación de hoy es conocer más acerca de sus experiencias como voluntarios. Nuestra conversación durará unos 60 minutos aproximadamente. Nos gustaría grabar y tomar notas durante la discusión para que podamos recordar con precisión las experiencias que ustedes compartan con nosotros. Un grupo focal es una conversación grupal guiada, en este caso con voluntarios como usted, para aprender acerca de sus experiencias. El moderador del grupo hará preguntas acerca de la enseñanza de la lectura en primer y segundo grado. Por favor, tenga en cuenta que no hay respuestas correctas o incorrectas en la conversación. Las preguntas del moderador sirven para organizar la conversación, pero esperamos que los participantes quieran compartir sus opiniones y experiencias relacionadas al tema. La participación en la discusión de grupo es voluntaria y no es necesario responder a preguntas que no quiera contestar. Les pediremos a todos los participantes mantener en reserva lo que otras personas compartan durante el grupo y no compartir lo conversado fuera del grupo. Lo que ustedes compartan aquí se mantendrá en privado. Sus opiniones no se compartirán con personas fuera del equipo de investigación, y los participantes no se identificarán con información personal. En las transcripciones y notas de los grupos focales se omitirán los nombres de las personas que participan y su nombre no aparecerá en informes o presentaciones. Vamos a grabar la discusión en grupo para facilitar la toma de notas, sin embargo solamente el equipo de investigación tendrá acceso a la grabación. La grabación de audio se borrará después de que termine el estudio. Si tiene alguna pregunta o inquietud, por favor póngase en contacto con Guillermo Duarte al teléfono al 2339-3079, o por correo electrónico a gduartem@dmcconsultores.com. También puede comunicarse con Camila Fernandez en cfernandez@mathematica-mpr.com. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.6 A. ROL Y MOTIVACIÓN DE LOS VOLUNTARIOS Quisiera empezar preguntándoles sobre su rol como voluntarios en el programa LJAJ. ¿Que los llevó a convertirse en voluntarios de LJAJ? 1. ¿Cuáles han sido sus funciones y responsabilidades? 2. ¿En qué consisten las actividades en las que ustedes colaboran? 3. ¿Cuál es su opinión sobre las actividades de la comunidad? 4. ¿De qué manera ayudan estas actividades a los estudiantes? 5. ¿De qué manera ayudan estas actividades a los docentes? 6. ¿De qué formas han contribuido a las actividades de LJAJ? (Tiempo, materiales) 7. ¿Qué actividades que realizaban en su vida diaria han dejado de hacer por participar como voluntarios de LJAJ? 8. ¿Qué desafíos han tenido que afrontar como voluntarios del programa LJAJ? ¿Cómo los han afrontado? 9. ¿Qué les ha impedido participar en más actividades de su comunidad? 10. ¿Recomendarían el voluntariado a otras personas en su comunidad? ¿Por qué sí/Por qué no? 11. ¿Qué les ha gustado más de ser voluntarios para LJAJ? B. PERCEPCIONES DE LOS VOLUNTARIOS SOBRE EL PROGRAMA 12. ¿Cómo describirían la percepción de los miembros de la comunidad sobre las actividades comunitarias de LJAJ? 13. ¿Cómo se incorporan en las actividades comunitarias de LJAJ el lenguaje y la cultura quechua? 14. ¿Cuáles han sido los principales desafíos que han encontrado con respecto a la participación de padres y niños? 15. ¿Ha habido alguna dificultad específica en la implementación de las actividades comunitarias del programa LJAJ? 16. ¿Cómo sienten que ha sido la recepción del Programa por parte de la comunidad y las familias con las que participan? 17. ¿De qué manera creen que LJAJ puede mejorar para adaptarse a las necesidades de los miembros de la comunidad? ¿Qué aspectos del programa dirían que están funcionando bien y cuáles no? ¿Por qué? D. CIERRE Para finalizar, 18. ¿Tienen alguna idea o información adicional sobre el programa, las actividades del programa, o la implementación del programa que les gustaría compartir? ¡Gracias por su participación! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.7 3. LEER JUNTOS, APRENDER JUNTOS: FOCALES CON ESPECIALISTAS DEL PROGRAMA GRUPOS Presentación Gracias por aceptar reunirse con nosotros el día de hoy. Como ustedes saben Mathematica Policy Research, una firma de investigación y evaluación independiente con sede en los EE.UU., están trabajando con Save the Children para llevar a cabo una evaluación del Programa “Leer Juntos, Aprender Juntos (LJAJ)” y para entender los programas educativos de la región de Quiché. El propósito de esta conversación es conocer sus experiencias en la implementación de las actividades del Programa Leer Juntos, Aprender Juntos (LJAJ). Nuestra conversación durará alrededor de 90 minutos. Por favor, tenga en cuenta que no hay respuestas correctas o incorrectas en la conversación. La participación en la discusión de grupo es voluntaria y no es necesario responder a preguntas que no quiera contestar. Les pediremos a todos los participantes mantener en reserva lo que otras personas compartan durante el grupo y no compartir lo conversado fuera del grupo. Lo que ustedes compartan aquí se mantendrá en privado. Sus opiniones no se compartirán con personas fuera del equipo de investigación, y los participantes no se identificarán con información personal. En las transcripciones y notas de los grupos focales se omitirán los nombres de las personas que participan y su nombre no aparecerá en informes o presentaciones. Vamos a grabar la discusión en grupo para facilitar la toma de notas, sin embargo solamente el equipo de investigación tendrá acceso a la grabación. La grabación de audio se borrará después de que termine el estudio. Si tiene alguna pregunta o inquietud, por favor póngase en contacto con Guillermo Duarte al teléfono al 2339-3079, o por correo electrónico a gduartem@dmcconsultores.com. También puede comunicarse con Camila Fernandez en cfernandez@mathematica-mpr.com. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.8 A. ROL DE LOS ESPECIALISTAS Quisiera comenzar preguntándoles ¿cómo le enseñan ustedes a leer a los estudiantes de primero y segundo grado? 1. ¿Cuáles han sido sus responsabilidades? 2. ¿Cuáles han sido sus actividades principales en la implementación del programa? 3. ¿Qué tipo de preparación y soporte han recibido durante la implementación del programa? ¿De parte de quién? 4. ¿Cómo está organizado el equipo de implementación? ¿Con quienes se comunican para planear el trabajo y conversar sobre sus responsabilidades y actividades en el campo? 5. ¿En qué forma sienten ustedes que han contribuido con la implementación del Programa? (tiempo, habilidades, materiales) 6. ¿Qué desafíos ha enfrentado en su rol? ¿Cómo han manejado esos desafíos hasta el momento? B. FIDELIDAD DE LA IMPLEMENTACIÓN DEL PROGRAMA 7. ¿Cómo se ha implementado LJAJ en las escuelas de la fase I y la fase II? ¿Qué diferencias ha habido en la implementación del programa en las escuelas de la fase 1 las de la fase 2? 8. ¿Cómo se organizó el componente de formación a los docentes y cómo se realizó? ¿Cuáles han sido los principales desafíos? ¿Cómo fueron asumidos por los docentes? 9. ¿Cómo se organizó el componente de acompañamiento a los docentes y cómo se realizó? ¿Cuáles han sido los principales desafíos? ¿Cómo fueron asumidos por los docentes? 10. ¿Cómo se han organizado los talleres con padres y cómo se han realizado? ¿Cuáles han sido los principales desafíos? ¿Cómo fueron asumidos por los padres? 11. ¿Cómo se organizó el banco de libros y cómo ha sido usado en la comunidad? ● Explorar el contenido de la actividad y los roles de los diferentes actores. ¿Cuáles han sido los principales desafíos con respecto a esta actividad? ● ¿Cómo se respondió a dichos desafíos? 12. ¿Cómo se organizó el préstamo de libros y cómo se ha llevado a cabo en la comunidad? ● Explorar el contenido de la actividad y los roles de los diferentes actores. ¿Cuáles han sido los principales desafíos con respecto a esta actividad? ● ¿Cómo se respondió a dichos desafíos? 13. ¿Cómo se organizó la actividad amigos de la lectura y cómo se ha llevado a cabo en la comunidad? ¿Cuáles han sido los principales desafíos con respecto a esta actividad? ● Explorar el contenido de la actividad y los roles de los diferentes actores. ¿Cómo se respondió a dichos desafíos? 14. ¿Cómo se organizaron los campamentos de lectura y cómo se ha llevado a cabo en la comunidad? ¿Cuáles han sido los principales desafíos? ● Explorar el contenido de la actividad y los roles de los diferentes actores. ¿Cómo se respondió a dichos desafíos? 15. ¿Qué aspectos del programa no se han desarrollado de acuerdo al planeamiento o a las indicaciones de Kallpa? 16. ¿Qué tipo de adaptaciones han tenido que hacer para satisfacer las necesidades de la población objetivo? EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.9 17. ¿Qué tipo de diferencias se han producido en la implementación del programa en las escuelas según su tamaño, ubicación geográfica, tipo o cualquier otra característica relevante? C. PERCEPCIONES DE LOS ESPECIALISTAS 18. ¿Cómo han recibido los profesores, padres y estudiantes las actividades de LJAJ? 19. ¿Cómo describirían las características de los estudiantes/padres que participan de las actividades en la comunidad? 20. ¿Cómo se incorporan en las actividades de la comunidad el lenguaje y la cultura quechua? 21. ¿Cuáles han sido los principales desafíos que han encontrado con respecto a la participación de padres y niños? 22. ¿De qué manera creen que LJAJ puede mejorar para adaptarse a las necesidades de los miembros de la comunidad? D. CIERRE 23. ¿Tienen alguna idea o información adicional sobre el programa, las actividades del programa, o la implementación del programa que les gustaría compartir? 24. ¿Cuáles prácticas pedagógicas utiliza con mayor frecuencia, y por qué? ¿Podrían dar algunos ejemplos? 25. ¿Qué aspectos de la lectura y la escritura son más difíciles para los estudiantes? 26. ¿Qué aspectos de la lectura y la escritura aprenden con facilidad los estudiantes? ¡Gracias por su participación! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.10 4. LEER JUNTOS, APRENDER JUNTOS: FOCALES DE MAESTROS (GRUPO A & B) GRUPOS Presentación Gracias por aceptar reunirse con nosotros el día de hoy. Como ustedes saben, Mathematica Policy Research, una firma de investigación y evaluación independiente con sede en los EE.UU., está trabajando con Save the Children para llevar a cabo una evaluación del Programa “Leer Juntos, Aprender Juntos (LJAJ)” y para entender los programas educativos de la región de El Quiché. El propósito de nuestra conversación de hoy es conocer más acerca de sus experiencias enseñándole a leer a sus alumnos de primer y segundo grado. Nuestra conversación durará unos 60 minutos aproximadamente. Nos gustaría grabar y tomar notas durante la discusión para que podamos recordar con precisión las experiencias que ustedes compartan con nosotros. Un grupo focal es una conversación grupal guiada, en este caso con maestros como usted, para aprender acerca de sus experiencias. El moderador del grupo hará preguntas acerca de la enseñanza de la lectura en primer y segundo grado. Por favor, tenga en cuenta que no hay respuestas correctas o incorrectas en la conversación. Las preguntas del moderador sirven para organizar la conversación, pero esperamos que los participantes quieran compartir sus opiniones y experiencias relacionadas al tema. La participación en la discusión de grupo es voluntaria y no es necesario responder a preguntas que no quiera contestar. Les pediremos a todos los participantes mantener en reserva lo que otras personas compartan durante el grupo y no compartir lo conversado fuera del grupo. Lo que ustedes compartan aquí se mantendrá en privado. Sus opiniones no se compartirán con personas fuera del equipo de investigación, y los participantes no se identificarán con información personal. En las transcripciones y notas de los grupos focales se omitirán los nombres de las personas que participan y su nombre no aparecerá en informes o presentaciones. Vamos a grabar la discusión en grupo para facilitar la toma de notas, sin embargo solamente el equipo de investigación tendrá acceso a la grabación. La grabación de audio se borrará después de que termine el estudio. Si tiene alguna pregunta o inquietud, por favor póngase en contacto con Guillermo Duarte al teléfono al 2339-3079, o por correo electrónico a gduartem@dmcconsultores.com. También puede comunicarse con Camila Fernandez en cfernandez@mathematica-mpr.com. A. PEDAGOGÍA DE LA LECTURA A LOS ALUMNOS DE PRIMER Y SEGUNDO GRADO Quisiera comenzar preguntándoles ¿cómo le enseñan ustedes a leer a los estudiantes de primero y segundo? 1. ¿Cuáles son las prácticas pedagógicas para la enseñanza de la lectura que funcionan mejor con sus alumnos? ¿Qué métodos de instrucción se utilizan? ¿Podrían dar algunos ejemplos? 2. ¿Cuáles prácticas pedagógicas utiliza con mayor frecuencia, y por qué? ¿Podrían dar algunos ejemplos? 3. ¿Qué aspectos de la lectura y la escritura son más difíciles para los estudiantes? 4. ¿Qué aspectos de la lectura y la escritura aprenden con facilidad los estudiantes? EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.11 B. PARTICIPACIÓN EN EL PROGRAMA, APRENDIZAJE Y CAMBIOS EN LAS PRÁCTICAS Ahora hablemos del programa Leer Junto, Aprender Juntos (LJAJ). 1. ¿Qué conocen sobre el programa LJAJ? ¿Cuáles son los objetivos del programa? 2. ¿En qué tipo de formación o actividades de entrenamiento de LJAJ han participado? 3. ¿Qué cosas has aprendido en el entrenamiento LJAJ? ¿Podrían dar algunos ejemplos? 4. ¿Qué tipo de cosas aprendió a través del coaching entrenamiento (acompañamiento) LJAJ? ¿Podrían dar algunos ejemplos? 5. ¿De qué manera LJAJ ha cambiado su manera de enseñarles a sus alumnos a leer? ¿Podrían dar algunos ejemplos? 6. ¿De qué manera LJAJ ha cambiado sus rutinas en la clase o en la forma de interactuar con sus estudiantes? ¿Podrían dar algunos ejemplos? 7. ¿Cuáles son las diferencias y similitudes entre la pedagogía de la lectura del Currículo Nacional Base o del Programa Nacional de Lectura y la de LJAJ? 8. ¿Enfrentó algún desafío para estar disponible para participar en actividades de capacitación o entrenamiento de LJAJ? ¿Cuáles fueron esos desafíos? 9. ¿Qué aspectos de LJAJ les han gustado más? ¿Por qué? 10. ¿Qué aspectos de LJAJ les han gustado menos gustó menos? ¿Por qué? C. ASPECTOS SOCIOCULTURALES DEL PROGRAMA 11. ¿Qué aspectos del programa toman en cuenta la cultura, el idioma y las costumbres de su comunidad? 12. ¿Qué aspectos del programa son difíciles de integrar en la comunidad? D. CIERRE 13. ¿En qué medida creen que LJAJ le ha ayudado a mejorar su capacidad de enseñar a los niños a leer? 14. ¿Qué retos o dificultades tienen enfrentan aún al enseñarle a leer a los alumnos de primer y segundo grado? ¿Qué les podría ayudar a superar esos desafíos? 15. ¿Cuáles son los aspectos del programa que se pueden mejorar? 16. ¿Recomendaría el entrenamiento LJAJ a otros en la comunidad? ¿Por qué / por qué no? 17. ¿Tiene usted alguna idea o información adicional sobre el programa, las actividades del programa o de la implementación que le gustaría compartir? ¡Gracias por su participación! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.12 5. LEER JUNTOS, APRENDER JUNTOS: FOCALES DE MAESTROS (GRUPO C) GRUPOS Presentación Gracias por aceptar reunirse con nosotros el día de hoy. Como ustedes saben, Mathematica Policy Research, una firma de investigación y evaluación independiente con sede en los EE.UU., está trabajando con Save the Children para llevar a cabo una evaluación del Programa “Leer Juntos, Aprender Juntos (LJAJ)” y para entender los programas educativos de la región de El Quiché. El propósito de nuestra conversación de hoy es conocer más acerca de sus experiencias enseñándole a leer a sus alumnos de primer y segundo grado. Nuestra conversación durará unos 60 minutos aproximadamente. Nos gustaría grabar y tomar notas durante la discusión para que podamos recordar con precisión las experiencias que ustedes compartan con nosotros. Un grupo focal es una conversación grupal guiada, en este caso con maestros como usted, para aprender acerca de sus experiencias. El moderador del grupo hará preguntas acerca de la enseñanza de la lectura en primer y segundo grado. Por favor, tenga en cuenta que no hay respuestas correctas o incorrectas en la conversación. Las preguntas del moderador sirven para organizar la conversación, pero esperamos que los participantes quieran compartir sus opiniones y experiencias relacionadas al tema. La participación en la discusión de grupo es voluntaria y no es necesario responder a preguntas que no quiera contestar. Les pediremos a todos los participantes mantener en reserva lo que otras personas compartan durante el grupo y no compartir lo conversado fuera del grupo. Lo que ustedes compartan aquí se mantendrá en privado. Sus opiniones no se compartirán con personas fuera del equipo de investigación, y los participantes no se identificarán con información personal. En las transcripciones y notas de los grupos focales se omitirán los nombres de las personas que participan y su nombre no aparecerá en informes o presentaciones. Vamos a grabar la discusión en grupo para facilitar la toma de notas, sin embargo solamente el equipo de investigación tendrá acceso a la grabación. La grabación de audio se borrará después de que termine el estudio. Si tiene alguna pregunta o inquietud, por favor póngase en contacto con Guillermo Duarte al teléfono al 2339-3079, o por correo electrónico a gduartem@dmcconsultores.com. También puede comunicarse con Camila Fernandez en cfernandez@mathematica-mpr.com. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.13 A. PEDAGOGÍA DE LA LECTURA A LOS ALUMNOS DE PRIMER Y SEGUNDO GRADO Quisiera comenzar preguntándoles ¿cómo le enseñan ustedes a leer a los estudiantes de primero y segundo? ¿Qué métodos de instrucción se utilizan? 1. ¿Qué tipo de actividades de lectura y escritura hacen los niños en su salón de clases? ¿Puede darnos algunos ejemplos? 2. ¿Cuáles son las prácticas pedagógicas para la enseñanza de la lectura que funcionan mejor con sus alumnos? 3. ¿Cuáles prácticas pedagógicas utilizan con mayor frecuencia, y por qué? 4. ¿Qué aspectos de la lectura y la escritura son más difíciles para los estudiantes? ¿Puede darnos algunos ejemplos? 5. ¿Qué aspectos de la lectura y la escritura aprenden con facilidad los estudiantes? ¿Puede darnos algunos ejemplos? 6. ¿Qué retos o dificultades enfrentan al enseñarle a leer a los alumnos en primer y segundo grado? B. EXPERIENCIAS DE FORMACIÓN Y APOYO DOCENTE Ahora hablemos del programa de sus experiencias formándose como maestro, y del tipo de apoyo pedagógico que recibe en esta escuela. 7. ¿Han recibido alguna capacitación o entrenamiento enfocado específicamente en enseñar a leer a los niños en la escuela primaria? 8. ¿Cuál fue ese entrenamiento?¿Quién lo impartió? ¿Cuándo? ¿Puede darnos algunos ejemplos? 9. ¿Qué cosas aprendieron en el entrenamiento? ¿Puede darnos algunos ejemplos? 10. ¿En qué medida han podido aplicar lo que aprendieron en esas capacitaciones al enseñar la lectura? ¿Puede darnos algunos ejemplos? 11. ¿En qué medida lo que aprendieron ha mejorado su capacidad para enseñar a los niños a leer? ¿Puede darnos algunos ejemplos? D. CIERRE Para finalizar, 12. ¿Hay algo que crean que puede ayudarles a ustedes a mejorar la manera le enseñan a aprenden a leer a los alumnos? 13. ¿Qué tipo de apoyo piensan que podría ayudar a superar los desafíos que enfrentan al enseñarle a leer a los estudiantes de primer y segundo grado? ¡Gracias por su participación! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.14 6. LEER JUNTOS, APRENDER JUNTOS: PADRES DE FAMILIA (GRUPOS A&B) Presentación Muchas gracias por aceptar reunirse con nosotros en este día. Nosotros hacemos parte de un equipo que está realizando un estudio sobre programas educativos en la región de Quiché. En el estudio están colaborando varias organizaciones, entre ellas DMC, Mathematica Policy Research (centro de investigación basada en EEUU) y Save The Children. El propósito de la conversación de hoy es aprender más acerca de de sus hijos y miembros de la familia en el proceso de aprendizaje de la lectoescritura y conocer sobre su participación en actividades dentro y fuera de la escuela para ayudarle a los niños a aprender a leer. Nuestra conversación durara alrededor de 60 minutos. Un grupo focal es una conversación grupal guiada, en este caso con padres de familia como ustedes, para aprender acerca de sus experiencias. El moderador del grupo hará preguntas acerca de la lectura de los niños de primer y segundo grado. Por favor, tenga en cuenta que no hay respuestas correctas o incorrectas en la conversación. Las preguntas del moderador sirven para organizar la conversación, pero esperamos que los participantes quieran compartir sus opiniones y experiencias relacionadas al tema. La participación en la discusión de grupo es voluntaria y no es necesario responder a preguntas que no quieran contestar. Les pediremos a todos los participantes mantener en reserva lo que otras personas compartan durante el grupo y no compartir lo conversado fuera del grupo. Lo que ustedes compartan aquí se mantendrá en privado. Sus opiniones no se compartirán con personas fuera del equipo de investigación, y los participantes no se identificarán con información personal. En las transcripciones y notas de los grupos focales se omitirán los nombres de las personas que participan y su nombre no aparecerá en informes o presentaciones. Vamos a grabar la discusión en grupo para facilitar la toma de notas, sin embargo solamente el equipo de investigación tendrá acceso a la grabación. La grabación de audio se borrará después de que termine el estudio. Si tiene alguna pregunta o inquietud, por favor póngase en contacto con Guillermo Duarte al teléfono al 2339-3079, o por correo electrónico a gduartem@dmcconsultores.com. También puede comunicarse con Camila Fernandez en cfernandez@mathematica-mpr.com. EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.15 A. CONOCIMIENTOS Y EXPECTATIVAS DE LOS PADRES SOBRE EL PROGRAMA LJAJ Quisiera empezar preguntándoles: ¿Cómo han aprendido a leer sus hijos? 1. ¿Cómo participan ustedes en el aprendizaje de la lectura? ¿Cuánto tiempo le dedican a la práctica de la lectura en su hogar? 2. ¿Qué saben ustedes sobre el programa LJAJ? 3. ¿Cuáles son los objetivos de las actividades de LJAJ? ¿Para qué sirven? ¿A quiénes están dirigidas? 4. ¿Por qué son importantes las actividades de LJAJ para ustedes? 5. ¿Cómo participan ustedes en las actividades propuestas por LJAJ? 6. ¿Cuáles eran sus expectativas con respecto LJAJ? ¿Esas expectativas han cambiado? ¿De qué manera? ¿Por qué? B. PARTICIPACIÓN, APRENDIZAJE Y CAMBIOS EN LAS PRÁCTICAS 7. ¿En qué tipos de actividades de LJAJ han participados ustedes? ¿Con cuánta frecuencia? 8. ¿Qué han aprendido ustedes en esas actividades? 9. ¿En qué tipos de actividades de LJAJ han participados sus hijos? ¿Con cuánta frecuencia? 10. ¿Qué han aprendido sus hijos en esas actividades? 11. ¿Han afrontado algún desafío para poder participar en las actividades de LJAJ? ¿Cuáles fueron esos desafíos? (tiempo, relevancia percibida de la actividad, etc.) ¿Les gustaría participar más de las actividades de LJAJ? 12. ¿Cuáles son los aspectos de las actividades de LJAJ que más les gustaron? ¿Por qué? 13. ¿Cuáles son los aspectos de LJAJ que menos les gustaron? ¿Por qué? ¿De qué formas creen que LJAJ los ha beneficiado a ustedes y a sus familias?¿De qué forma LJAJ ha cambiado sus rutinas en casa o el modo en que interactúan con sus hijos? 14. ¿Las actividades de LJAJ con los padres les parecen apropiadas para esta región (Andahuaylas/Chincheros etc.) y para su cultura? ¿Por qué? 15. ¿Las actividades de LJAJ con los niños les parecen apropiadas para esta región (Andahuaylas/Chincheros etc.) y para su cultura? ¿Por qué? C. CIERRE Para finalizar, 16. ¿Creen que podrían haber algunos cambios para mejorar las actividades de LJAJ? ¿Cuáles serían sus sugerencias? 17. ¿Recomendaría participar de LJAJ a otras personas en la comunidad? ¿Por qué sí/Por qué no? 18. ¿Tienen alguna idea o información adicional sobre el programa, las actividades del programa, o la implementación del programa que les gustaría compartir? ¡Gracias por su participación! EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.16 7. LEER JUNTOS, APRENDER JUNTOS FORMATO DE OBSERVACIONES DE ACTIVIDES COMUNITARIAS ESTUDIO CUALITATIVO DE IMPLEMENTACIÓN Fecha de la observación: | | | | / 1 1 | | | / 2 0 | | 1 | 4 | Nombre del observador: Municipio: Tipo de Actividad: Descripción del lugar y del contexto donde se lleva a cabo la actividad: _________________________________________________________________________________ ____________ Número de Participantes: | | | , | | | Número de Lideres/Voluntarios en la actividad: | | | | Descripción de Participantes: EVALUATION OF LEER JUNTOS, APRENDER JUNTOS EARLY-GRADE READING INTERVENTION IN GUATEMALA: FINAL REPORT MATHEMATICA POLICY RESEARCH J.17 Descripción de Actividad de Lideres/Voluntarios: Descripción de la Actividad (lenguaje utilizado, objetivos de la actividad, materiales, instrucciones, etc.): _________________________________________________________________________________ ___________ Involucramiento de participantes’ en la Actividad (ej. qué tan involucrados estaban los participantes en la tarea, entendieron y siguieron las instrucciones dadas, parecían disfrutar la actividad): _________________________________________________________________________________ __________ Otras notas (ej. qué tan involucrados estuvieron los voluntarios y otros líderes en la tarea, dieron ellos instrucciones claras): _________________________________________________________________________________ U.S Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Telephone: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov