Latin America and the Caribbean (LAC) Reading Evaluation Contract PEFORMANCE EVALUATION OF THE COMMUNITY ACTION FOR READING AND SECURITY (CARS) ACTIVITY IN NICARAGUA: 2014 TO 2016 September 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Randall Blair, Virginia Poggio, Catalina Torrente, Emilie Bagby, and Nancy Murray of Mathematica Policy Research. This page has been left blank for double-sided copying. This page has been left blank for double-sided copying. ABSTRACT This report presents the results of an independent evaluation of the Community Action for Reading and Security (CARS) activity funded by USAID in Nicaragua’s South Caribbean Coast Autonomous Region (RACCS for its initials in Spanish). This reading activity is intended to strengthen community security, while improving overall educational outcomes. The evaluation’s main research questions were: (1) Is CARS implementation occurring as planned? and (2) Does CARS appear to be generating its desired effects? To answer these research questions, Mathematica conducted a mixed-methods performance evaluation of CARS implementation and potential effects from 2014 to late 2016. The evaluation used both quantitative data sources (available CARS monitoring and evaluation indicators, child assessment data, and data from structured interviews) and qualitative data sources (such as programmatic reports and stakeholder focus groups). Findings. The evaluation found that CARS is on track to meet major reading program milestones, including the number of after-school programs established and students served. However, the program has not met some major parent and community engagement objectives. Evidence suggests that CARS-trained teachers have incorporated CARS practices and materials into their daily activities with students. Furthermore, stakeholders provided several examples in which CARS generated positive effects on children’s learning and achievement. However, preliminary evidence still shows a large gap between current reading levels and program targets. Conclusions. CARS has been relatively well implemented and appears to have generated some promising effects on teacher behavior and student achievement. Given the multiple obstacles to improving student achievement in the RACCS, however, it is unlikely CARS will meet its goal with respect to the percentage of students who can read at grade level. The report includes recommended adjustments to strengthen programming. v This page has been left blank for double-sided copying. CONTENTS MATHEMATICA POLICY RESEARCH CONTENTS ACRONYMS ............................................................................................................................................... xiii EXECUTIVE SUMMARY: CARS PERFORMANCE EVALUATION ............................................................xv I. INTRODUCTION AND CARS DESIGN ........................................................................................... 1 A. Background................................................................................................................................ 1 B. Literature review ........................................................................................................................ 5 C. CARS activities and goals ......................................................................................................... 7 1. Formal and nonformal reading programs............................................................................ 8 2. Community engagement activities .................................................................................... 10 3. Local capacity development.............................................................................................. 11 4. Knowledge generation and management ......................................................................... 12 II. RESEARCH QUESTIONS AND METHODS ................................................................................. 13 A. Evaluation questions, study design, and data......................................................................... 13 B. Data collection ......................................................................................................................... 16 C. Data analysis ........................................................................................................................... 20 D. Limitations of this analysis....................................................................................................... 22 III. DESCRIPTION OF CARS STAKEHOLDERS ............................................................................... 23 A. What organizations are participating in CARS? ...................................................................... 23 B. What are the characteristics of educational communities participating in CARS?.................. 25 C. What are the characteristics of households and children participating in CARS? .................. 27 IV. IMPLEMENTATION FINDINGS..................................................................................................... 31 A. How is CARS being implemented? ......................................................................................... 31 1. Formal reading programs.................................................................................................. 32 2. EpCs ................................................................................................................................. 34 3. Community engagement efforts........................................................................................ 36 4. Local capacity development.............................................................................................. 38 5. Knowledge generation ...................................................................................................... 39 B. Is CARS being implemented as planned?............................................................................... 39 1. Formal reading programs and EpCs................................................................................. 40 2. CARS materials................................................................................................................. 42 3. Educator coaching and follow-up...................................................................................... 42 4. Parent schools .................................................................................................................. 44 vii CONTENTS MATHEMATICA POLICY RESEARCH 5. CAPs and community contributions .................................................................................. 45 C. What are the key facilitators and barriers of CARS implementation? ..................................... 46 V. EFFECTS OF CARS.......................................................................................................................51 A. Are teachers applying early-grade reading approaches—including using new materials and assessment tools? ............................................................................................ 51 B. Is CARS improving children’s outcomes? ............................................................................... 54 C. Is CARS reducing gender disparities? .................................................................................... 60 D. Is CARS improving parental and community support for early reading and security?............ 61 E. What type of outreach and awareness efforts are most successful in increasing engagement?........................................................................................................................... 63 F. Is CARS improving NGO capacity?......................................................................................... 65 G. Is CARS succeeding in increasing knowledge, skills, and resources? ................................... 69 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT ................................................................................................ 71 A. What are the major constraints to improving reading performance in CARS communities?........................................................................................................................... 71 1. Poor teacher capacity ....................................................................................................... 72 2. Lack of materials at school and at home .......................................................................... 73 3. Lack of parental engagement ........................................................................................... 73 4. Poor student attendance................................................................................................... 74 B. What are the major constraints to establishing a safe community environment?................... 74 C. What adjustments could improve CARS implementation?...................................................... 76 D. Are CARS activities sustainable?............................................................................................ 80 VII. REFERENCES............................................................................................................................... 83 APPENDIX A: CHARACTERISTICS AND OUTCOMES OF STUDENTS ASSIGNED TO ATTEND COHORT 1 EpCs .............................................................................................A.1 APPENDIX B: STAKEHOLDER PERCEPTIONS ON CARS COMPONENTS.......................................B.1 APPENDIX C: STAKEHOLDER SUGGESTIONS FOR MIDCOURSE CORRECTIONS TO CARS ...............................................................................................................................C.1 viii TABLES MATHEMATICA POLICY RESEARCH TABLES I.1 CARS implementation plans and performance targets.................................................................... 8 II.1 Data sources for research questions ............................................................................................. 14 II.2 CARS educational communities visited for the performance evaluation ....................................... 18 II.3 Sample sizes for primary data sources.......................................................................................... 19 III.1 NGOs implementing CARS............................................................................................................ 25 III.2 Characteristics of schools visited during data collection................................................................ 27 III.3 Characteristics of children assigned to attend EpCs (percentages unless otherwise indicated)........................................................................................................................................ 29 IV.1 CARS reading program implementation progress, as of late 2016 ............................................... 41 IV.2 Planned and actual materials distribution ...................................................................................... 42 IV.3 Planned and actual coaching visits by NGO staff.......................................................................... 44 IV.4 Planned and actual parent school attendance............................................................................... 45 IV.5 Planned and actual CAP completion ............................................................................................. 46 IV.6 CARS implementation facilitators and barriers according to stakeholders.................................... 47 V.1 Planned and actual use of reading assessments .......................................................................... 53 V.2 Stakeholders’ perspectives on effects of CARS EpCs (nonformal) and formal reading programs ........................................................................................................................................ 56 V.3 Planned and actual student achievement and retention, CARS formal and nonformal reading programs ........................................................................................................................... 59 V.4 Changes resulting from parent schools ......................................................................................... 62 V.5 Best practices in outreach events and communication efforts ...................................................... 64 V.6 Planned and actual average Organization Capacity Assessment total scores among CARS awardees............................................................................................................................. 65 V.7 Baseline and Midline OCA scores, CARS awardees..................................................................... 68 VI.1 Constraints to improving reading performance mentioned by interviewed stakeholders .............. 71 VI.2 CARS efforts to reduce constraints to early reading and security, and areas for improvement .................................................................................................................................. 77 VI.3 CARS sustainability assessment ................................................................................................... 80 ix This page has been left blank for double-sided copying. FIGURES MATHEMATICA POLICY RESEARCH FIGURES ES.1 Global CARS implementation timeline......................................................................................... xviii ES.2 Students served by CARS: 2014–2016 .........................................................................................xix ES.3 Percentage of primary school teachers who practice each activity at least once per day: CARS-trained versus nontrained teachers ................................................................................... xxii I.1 The RACCS, Nicaragua................................................................................................................... 2 I.2 CARS logic model ............................................................................................................................ 4 III.1 CARS organizational chart............................................................................................................. 24 IV.1 Global CARS implementation timeline........................................................................................... 32 IV.2 CARS training participants, January–September 2016 ................................................................. 34 IV.3 EpCs by cohort and municipality.................................................................................................... 35 IV.4 Timeline of EpC implementation .................................................................................................... 36 V.1 Percentage of primary school teachers who practice each activity at least once per day: CARS-trained versus nontrained teachers .................................................................................... 54 V.2 Percentage of educators who believe CARS formal and nonformal reading programs have had an effect on student outcomes....................................................................................... 55 V.3 Changes in Organizational Capacity Assessment scores from baseline to midline, CARS awardees........................................................................................................................................ 67 xi This page has been left blank for double-sided copying. ACRONYMS MATHEMATICA POLICY RESEARCH ACRONYMS AMC Acción Médica Cristiana APA Aprendo, Practico, Aplico BICU Bluefields Indian and Caribbean University CAP community action plan CARS Community Action for Reading and Security Activity CETT Centers for Excellence in Teacher Training EGRA early grade reading assessment EpC Espacio para Crecer FAS Fónico, Analítico, Sintético FHR Fundación Hermanamiento RAMA FQSF Fundación Yo Quiero Ser Feliz FZT Fundación Zamora Terán LAC Latin America and the Caribbean M&E monitoring and evaluation MINED Ministry of Education NGO nongovernmental organization OCA Organizational Capacity Assessment QL Quantum Learning RACCS Región Autónoma de la Costa Caribe Sur RACCN Región Autónoma de la Costa Caribe Norte SEAR Sistema Educativo Autonómico Regional URACCAN Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense USAID U.S. Agency for International Development xiii This page has been left blank for double-sided copying. EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH EXECUTIVE SUMMARY: CARS PERFORMANCE EVALUATION A. Introduction This report presents the results of a performance evaluation of the Community Action for Reading and Security (CARS) activity, which is funded by the U.S. Agency for International Development (USAID), in Nicaragua’s South Caribbean Coast Autonomous Region (known as RACCS for its initials in Spanish1 ). Implemented in 5 of the 12 RACCS municipalities, the four programmatic components of CARS—(1) formal and nonformal reading programs, (2) community engagement, (3) local capacity development, and (4) knowledge generation and management—are intended to strengthen community security, 2 while improving overall educational outcomes. The lead CARS contractor is DevTech Systems, a U.S.-based development firm with a local presence in the RACCS. DevTech works in conjunction with six local NGOs, community leaders, volunteers, and school staff to implement CARS in over 200 communities in the region. Together, DevTech and the implementing NGOs are known collectively as the CARS Team.3 The CARS Team began executing these activities in 2014, and the project is expected to continue in the RACCS through March 2019. CARS is also scheduled to expand to the Northern Caribbean Coast Autonomous Region (known as RACCN for its initials in Spanish) in 2017. This evaluation covers CARS implementation in the RACCS from 2014 to 2016. As such, advances in CARS implementation that occurred from 2017 onward are not reflected in this report. CARS formal and nonformal reading programs are designed to help preschoolers and elementary school students gain strong reading and socioemotional skills and improve their attachment to school, thus boosting their achievement and increasing attendance and enrollment rates. To complement the reading programs, community and parent engagement efforts—that is, parent schools and community action plans (CAPs)—are intended to generate increased parental engagement and community investment in early reading, as well as safer school environments. Through CAPs, parents could work among other things on infrastructure improvements, which constitute a very salient need in these educational communities. In addition, technical assistance to local nongovernmental organizations (NGOs) is designed to strengthen their capacity to administer CARS reading and community security programs as well as their ability to manage similar programs in future years. The CARS project also includes a knowledge generation component, under which it was envisioned that the CARS team would provide data and information support to policymakers and policy influencers in the region, including USAID staff. 1 In Spanish, the name is Región Autónoma de la Costa Caribe Sur. 2 In the context of CARS, community security is defined as a safe community environment, particularly an educational environment in which children can commute to school safely and feel at ease at school—free from physical or psychological abuse from teachers and other children, poor school conditions, or the threat of violence. 3 These NGOs are (1) Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense (URACCAN), (2) Fundación Zamora Terán (FZT), (3) Fundación Hermanamiento RAMA (FHR), (4) Acción Médica Cristiana (AMC), (5) Fundación Yo Quiero Ser Feliz (FQSF), and (6) Bluefields Indian and Caribbean University (BICU). xv EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH When CARS began, U.S. policy prohibited direct USAID involvement in Nicaraguan public schools. For this reason, CARS targeted privately managed and subsidized schools with in￾school (formal) reading programs, and designed before- and after-school, community-based (nonformal) programs to reach children who attend public schools, called Espacios para Crecer or EpCs.4 In the RACCS, private and subsidized schools charge modest fees and have somewhat better infrastructure than public schools. However, private and subsidized schools are not elite institutions, and the students who attend them similar educational attainment and risk factors as students who attend public schools. As such, CARS considers children served by both formal and nonformal reading programs to be at-risk of poor educational outcomes and school drop-out. CARS formal and nonformal reading programs both feature a mix of educational approaches, materials, and educator training and assistance. Among formal and nonformal reading programs, CARS initially planned to serve a total of 12,500 children from 2014 to September 2017.5 As a critical part of the CARS model, DevTech planned to train local NGOs to administer all of the CARS formal and nonformal reading programs, by using a train-the-trainer model. After the NGO staff received training on the reading programs, they would train the teachers and EpC directors (called facilitators) in the new teaching methods. With DevTech’s support, the NGO staff would also provide trained teachers and facilitators with follow-up coaching, including in-class observation and assistance. Drawing primarily from Aprendo, Practico, Aplico (APA); Quantum Learning (QL); and Fónico, Analítico, Sintético (FAS) methods, the CARS training and coaching would emphasize the use of proven, simple, and effective teaching and learning techniques to improve early grade reading among students. In particular, the CARS educational approach was designed to build the following five foundational reading skills: phonemic awareness, phonics, fluency, vocabulary and comprehension strategies. Also critical to the reading programs were learning materials—including, teacher guides, student workbooks, storybooks, games, and manipulatives—that would be contextualized as necessary to the local community and used extensively in participating schools and EpCs. With respect to community engagement activities, CARS envisioned establishing parent schools in all communities that participated in the formal and nonformal reading programs. Similar to the reading programs discussed above, local NGOs would be trained on community engagement activities, to galvanize community awareness of the importance of reading, and to identify and mitigate security risks for students. With support from DevTech, NGOs would also train EpC facilitators and teachers (under the train-the-trainer model) to moderate the parent schools. Parent schools would provide parents with a forum to discuss important topics in early reading, child development, and community security. In addition, CARS encouraged educators, parents, and community members to identify critical school and community needs through the CAP development process, and to secure USAID funds to make improvements. The CAP development process would also be valuable in itself, as it would empower participants through their mobilization, to solve community problems and help leverage local and private sector resources to do so. In sum, the development and implementation of CAPs would constitute an 4 This situation changed in late 2016, when CARS received approval to implement formal reading programs in public schools. In this report we will refer to the non-formal program as “EpC”. 5 These goals were revised in 2017, such that CARS was projected to serve 24,014 children by the end of 2019. However, this evaluation compares progress by late 2016 against initial CARS goals. xvi EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH opportunity for community members to engage in governance by focusing on education. Under the local capacity development component, DevTech provided NGOs with technical support to help administer reading and community engagement activities, as well as technical assistance and training centered on building the NGOs’ overarching financial and administrative capacity. Finally, as part of the knowledge generation and management component, CARS was tasked with contributing to the quality of education by promoting the use of evidence to drive decision making. The CARS team shared its experience and results with other USAID and USG partners, and provided data and information support to policymakers and policy influencers in the region, including USAID staff. CARS is implemented in the RACCS context, which is characterized by relatively high rates of crime and drug use, geographically dispersed and isolated communities, high rates of migration, and pockets of civil unrest linked to Nicaraguan government plans to build an interoceanic canal. A CARS community participation study found that the most prominent needs in four sampled municipalities in RACCS include education; economic opportunities; and health, water and sanitation (CARS 2014). Education outcomes are particularly poor in the RACCS compared to other regions in Nicaragua. According to the 2007 education census, 45 percent of school-age boys and 40 percent of school-age girls in the RACCS were out of school, while illiteracy rates for this population were only around 25 percent. A CARS baseline study in 2014 found that only 5 percent of first grade students, 40 percent of second grade students, and 48 percent of third grade students read with fluency (CARS 2016). CARS faces the challenge of improving education and security outcomes in a region in which most school-aged children have reading deficiencies and face nontrivial obstacles to attending school on a regular basis. The infeasibility of working directly with public schools in the region when CARS began program implementation further complicated the project’s ability to completely meet its goals. B. Methods Motivated by a desire to inform the remaining CARS activities as well as program expansion plans and USAID’s future projects in the region, USAID/Nicaragua representatives identified several research questions for this evaluation. These questions can be divided into five global sets of questions: (1) how is CARS being implemented? (2) is implementation occurring as planned? (3) what are the key barriers and facilitators to CARS implementation? (4) does CARS appear to be generating its desired effects? and (5) what adjustments could improve CARS implementation? To answer these core research questions, Mathematica conducted a performance evaluation of CARS implementation from 2014 to late 2016. A performance evaluation is designed to describe and assess the implementation of a program, including whether the program is meeting its performance targets, what aspects are working well, and what could be improved in the future. This is a mixed-methods evaluation: it uses a mix of quantitative data sources (such as available CARS monitoring and evaluation [M&E] indicators, child assessment data, and data from structured interviews) and qualitative data sources (such as programmatic reports and stakeholder focus groups) to answer the research questions. Using both qualitative and quantitative methods allows us to address each research question with all available data sources and to compare and contrast qualitative and quantitative findings. xvii EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH C. Findings 1. How is CARS being implemented? CARS is being implemented largely according to its initial design and timeline, with some delays. CARS implementation has followed its train-the-trainer design, in which NGO staff are initially trained by DevTech staff in active teaching methods (in a training called ENTRENA) and then supported by DevTech as they train and support preschool and primary school teachers and newly contracted EpC facilitators to employ these teaching methods in the classroom. EpC implementation kicked off with 10 pilot EpCs in 2014 and expanded dramatically in 2015 and 2016 to reach a total of 270 EpCs. Formal primary and preschool reading programs were first implemented in 2015. CARS led the first of several planned teacher certification workshops—the diploma program—in mid-2016 (Figure ES.1). CARS staff developed and distributed thousands of educational titles to the EpCs and participating schools from 2014 to 2016, but experienced some delays in contextualizing materials to the region and distributing them; these delays were linked to several factors, including lengthy approval processes for newly contextualized materials. The NGOs began implementing parent schools through the EpCs in 2015 and expanded to participating private schools shortly thereafter. Although the CAPs largely stalled in 2015, the NGOs and DevTech found ways of simplifying the development and approval process, which led to more and higher quality CAPs in 2016. Throughout implementation, DevTech staff worked closely with all six NGOs to strengthen their administrative, financial, organizational, management, and technical skills. Figure ES.1. Global CARS implementation timeline 2. Is CARS being implemented as planned? CARS is on track to meet major reading program milestones (Figure ES.2). By late 2016, the program had met its ambitious target of establishing and maintaining 270 EpCs. This is an impressive achievement that required extensive planning, coordination, and training across five municipalities from 2014 to 2016. Nearly 11,800 children were served by the CARS formal and nonformal reading programs by late 2016, but CARS had not yet met its original target of xviii EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH serving 12,500 children by 2017 (Figure ES.2). However, CARS was projected to surpass this target in early 2017. Figure ES.2. Students served by CARS: 2014–2016 Although CARS has not met community engagement goals of parent school participation and implementation of CAPs, it has surpassed its community contribution goal. Although parent schools have successfully engaged some parents on important topics, they have fallen short of participation targets throughout 2015 and 2016 due to several factors, including the NGOs’ prioritization of other CARS activities, NGOs’ difficulties with reaching remote communities in the rainy season, parents’ limited motivation or incentives to participate, and overly ambitious initial targets for very small communities. For example, parent school attendance in 2016 was 4,471 attendees, or 71 percent of the goal of 6,250 attendees, which assumed monthly attendance by parents of children enrolled in EpCs or participating schools. (However, it should be noted that parent school attendance increased steadily during 2016, as NGOs established a stronger presence in CARS communities.) In addition, the CAPs have fallen short of implementation targets for several reasons, including a lack of initial clarity with respect to CAP eligibility requirements, a long development and approval process, and deficiencies in the initial CAPs. By late 2016, only 5 CAPs had been approved by USAID and executed—far below the goal of 90 executed CAPs. However, CARS staff reported that the overall quality of CAPS improved significantly in late 2016, creating a strong pipeline of CAPS that could be executed in early 2017. Although parent school attendance and CAP execution targets were not met by late 2016, CARS surpassed its goal of $60,000 in community contributions by late 2016; these contributions included private and public donations of materials and space for CARS events. Other findings with respect to implementation targets include the following: • In 2015, CARS achieved 99 percent of its original goal of distributing 27,000 textbooks and learning materials to EpCs and participating schools in 2015, but fell short of a similar target xix EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH in 2016 due in part to delays in developing new materials and receiving ordered materials from suppliers. • Educator follow-up and coaching visits occurred less frequently than planned due to NGO capacity constraints and prioritization of other CARS activities. Although all facilitators and 76 percent of trained teachers reported at least one CARS visit in the past year, educators reported they were visited only between three and four times in the past year, on average—a number far below the goal of at least two visits per month. 3. What are the key facilitators and barriers of CARS implementation? Formal and nonformal reading programs have strong reading educational approaches, useful materials, and effective initial training (facilitators). In interviews and focus groups, stakeholders consistently identified three fundamental strengths of the CARS program: (1) CARS’ strong reading educational approach, based on APA, FAS, and QL, which educators saw as far superior to the existing reading curricula found in public and private schools; (2) the program’s wide range of learning materials and manipulatives, which educators reported using consistently to structure and complement in-class activities; and (3) CARS’ comprehensive initial training for reading programs, which educators reported was highly informative and relevant to their work. In this sense, CARS possesses the critical components of a successful reading program: strong educational approach, useful materials, and adequate teacher training. Material distribution delays, language mismatches, and a lack of consistent coaching and follow-up represent lost opportunities to instill teaching practices (barriers). In interviews and focus groups, stakeholders also identified several weaknesses of the CARS program. In particular, teachers and facilitators noted delays in receiving CARS materials at the beginning of the school year or EpC session, language mismatches between the materials provided and students’ mother tongue, and instances in which donated materials (such as instruments) required electrical access that their schools did not have. Educators also expressed a need for more frequent coaching and follow-up visits from CARS staff than they have received thus far, as well as a desire to see the new teaching techniques in action (over and above exposure to the techniques during training). These factors likely constrained teachers from fully implementing the educational approach and activities presented in the initial CARS training. xx EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH Other implementation facilitators and barriers of the CARS program include the following: • The EpCs and formal reading programs appear to have strong support among principals, parents, and community leaders. Several parents and community leaders noted that they valued the role that the EpCs play in supporting students who struggle academically or behaviorally (facilitator). • Driven by migration patterns, long commutes, and parents' decisions to keep their children home, low EpC attendance and retention pose a threat to the program’s ability to improve children’s learning outcomes—particularly for students who are most at risk of dropping out of school (barrier). • Linked in part to high staff turnover and lack of technical expertise in the region, capacity constraints hinder CARS’ ability to meet ambitious implementation targets, particularly with respect to follow-up visits, the CAPs, and parent schools. DevTech and NGO representatives claim that they do not have enough staff to complete all CARS tasks. However, USAID indicated that this fact was not communicated to the COR or the Contracting Officer. Personnel changes at DevTech and the NGOs have further exacerbated these capacity constraints and produced leadership gaps (barrier). 4. Does CARS appear to be generating its desired effects? There is evidence that primary school teachers changed their classroom practices as a result of CARS. A comparison of teacher-reported practices between CARS-trained teachers and teachers not trained by CARS suggests that CARS-trained teachers appropriated several core APA practices featured in formal reading programs In particular, CARS-trained teachers were over 25 percentage points more likely to use hugs and affection, try to resolve conflict through dialogue, have students use the dictionary in the classroom, and read stories out loud to the boys and girls (Figure ES.3). xxi EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH Figure ES.3. Percentage of primary school teachers who practice each activity at least once per day: CARS-trained versus nontrained teachers Source: In-person interviews with 38 trained preschool and 1st- through 3rd-grade teachers and 19 nontrained preschool and 1st- through 3rd-grade teachers in September/October 2016. The target for educators’ use of reading assessments was not met. In interviews, around two-thirds of interviewed facilitators, teachers, and principals who received CARS training said they used the results of reading assessments in the past year. This was below the 2016 goal of 85 percent across trained educators. Furthermore, educators often had difficulty articulating how they use reading assessments. Several interviewed directors and facilitators made general statements about how they use the results for “evaluation” but could not provide specific detail. Other facilitators and directors appeared to view assessments as more of a teaching or practice technique than an evaluation tool. CARS may have generated positive effects on children’s outcomes, but preliminary evidence still shows a large gap between current reading levels and program targets. In interviews, educators indicated that CARS had a positive effect on students’ reading, socialization, and attendance—and they gave concrete examples of these improvements. For xxii EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH example, teachers noted that students had registered tangible improvements in reading comprehension as a result of reading programs, which emphasize analysis of stories and reading passages. Some CARS student assessment data corroborate these reading improvements. In select cases, students’ reading fluency increased dramatically after one year of the EpCs and formal reading programs. Despite any potential improvements linked to CARS, however, a large gap remains between students’ current reading levels and the CARS targets. Among kids enrolled in the EpCs and participating primary schools, the percentage of kids that read at grade level in 2016 (33 percent and 34 percent for males and females, respectively) was far below CARS goals of 53 percent and 62 percent for males and females, respectively. Although students may have improved their reading performance under CARS, this improvement is not sufficient to qualify students as reading “at grade level,” given low baseline reading levels. Parent schools have successfully engaged some parents and community members, but poor attendance overall limits the schools' potential benefits. Stakeholders generally agreed that parent schools have helped parents participate in their child’s education and better understand their behavior. In structured interviews, over 90 percent of principals and EpC facilitators agreed that overall, CARS had succeeded in increasing parent participation in their children’s education. However, low parent school attendance across participating communities signals missed opportunities to engage parents on topics of reading and security. Stakeholders reported few direct effects of CARS on community security or parental and community engagement on security issues. In the context of CARS, community security is defined as a safe community environment, particularly an educational environment in which children can commute to school safely and feel at ease at school. There is little evidence that community engagement efforts had any tangible effects on community security or school and community conditions based on stakeholder reports. One NGO representative noted that as a result of CARS activities—including parent schools and CAPs—parents are more conscious of security concerns in the community, particularly within and around preschools and primary schools. However, parents and community leaders did not mention this greater awareness in focus groups. In addition, parents and community leaders noted no tangible improvements in school or community security as a result of CARS. In part, this may reflect low rates of CAP development and execution in sampled communities. According to the initial CARS design, CAPs could provide funding for school and community improvements oriented toward greater safety and security, such protective walls for schools, but fewer CAPS than expected were implemented. Other findings regarding CARS’ potential effects include the following: • Local NGOs have been strengthened as a result of CARS implementation—particularly in the case of two NGOs that have made large improvements in financial management and human resources. • DevTech has shared CARS data analyses with USAID and external stakeholders. USAID officials report that they use CARS-generated data for internal reporting, planning, and to monitor the evolution of education and security in the region. However, the effect of these knowledge generation activities on other decision making and early reading programming in the region is unclear. xxiii EXECUTIVE SUMMARY MATHEMATICA POLICY RESEARCH 5. What adjustments could improve CARS implementation? Introduce and enforce a minimum follow-up visit requirement for reading programs. In an effort to reinforce teacher adoption of CARS teaching techniques, CARS could prioritize the NGOs’ monthly field visits to schools and EpCs during the last few months of the project. This may include introducing and enforcing the requirement of at least one visit per month to each EpC and participating school. Offer enhanced modeling opportunities and direct feedback in training and coaching visits. Training and coaching could be modified to allow educators more opportunities to see techniques in action—preferably, seeing an experienced educator applying the teaching methods with actual students—and to give teachers real-time feedback on their application of CARS techniques with their students. During follow-up visits, CARS could also provide educators with hands-on assistance in applying, interpreting, and using student reading assessments to inform classroom activities. Redistribute materials to match students’ mother tongue and schools’ electricity access. CARS could immediately remedy mismatches between students’ mother tongue and the language of materials delivered, as well as instances in which schools without electricity are given electrical devices such as radios. In the future, CARS could avoid such mismatches by coordinating beforehand with teachers and facilitators to confirm students’ mother tongue and schools’ access to electricity. Introduce concerted efforts to counteract poor EpC retention rates. CARS could provide a snack during or prior to EpC sessions or engage community members to provide snacks, particularly in educational communities in which lunch is not provided in school. CARS could also afford each educational community some flexibility to schedule EpC hours to fit the school schedule, with the goal of a quick transition from school to the EpC. Incentivize parent school attendance and facilitate meaningful parent-child interactions in community events. NGOs implementing CARS could adopt emerging best practices with respect to community events and parent schools. This includes holding inclusive small-scale events and festivals at the community level, as well as 'reading club' sessions in which parents or siblings read to students. To boost parent school attendance, NGOs could also structure parent school sessions as conversations and discuss topics that most interest parents. xxiv I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH I. INTRODUCTION AND CARS DESIGN The U.S. Agency for International Development (USAID), Regional Sustainability Office of the Bureau for Latin America and the Caribbean contracted with Mathematica Policy Research to perform an independent impact evaluation and analysis of a series of promising reading interventions and programs designed to increase access to education throughout Latin America and the Caribbean (LAC). The LAC Reads evaluation contract envisioned funding approximately 8 to 10 evaluations from September 2012 to September 2017 that focused on effectiveness and costs. Impact evaluations of early reading and education programs are under way in Guatemala, Honduras, Nicaragua, and Peru. This report presents the results of a performance evaluation of the Community Action for Reading and Security (CARS) intervention program funded by USAID/Nicaragua, which was implemented in the South Caribbean Coast Autonomous Region (knows as the RACCS for its initials in Spanish6 ) in Nicaragua. This program includes several components, including after￾school programs for at-risk students, called Espacios para Crecer (EpCs); new preschool and primary school educational approaches and materials; teacher training and technical assistance; community engagement activities; local capacity development efforts; and knowledge generation and management. This evaluation draws on a variety of quantitative and qualitative data to show the progress and perceived impacts of CARS activities from 2014 to 2016. This performance evaluation complements Mathematica’s random assignment impact evaluation of the EpC component of CARS, which is already under way (Bagby et al. 2016). In this chapter, we provide background on the RACCS, present a brief literature review of related education and security programs worldwide, and summarize initial plans for the CARS intervention. A. Background The nations of Central America, though culturally, socially, and economically diverse, have all been affected by the same sociopolitical phenomena, including drug trafficking, poverty, violence, and conflict. The effects of these phenomena are likely different in countries that are not located in the Northern Triangle region, such as Costa Rica, Nicaragua, and Panama. Nicaragua is the poorest nation in Central America and the second poorest in all Latin America, with 48 percent of its population living on less than $1 a day (in U.S. dollars) and 76 percent on less than $2 a day. The RACCS is the second-poorest region in Nicaragua, with more than half of its population living in extreme poverty. Figure I.1 shows the location of the RACCS in Nicaragua. 6 In Spanish, the name is Región Autónoma de la Costa Caribe Sur. 1 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH Figure I.1. The RACCS, Nicaragua Though violence and insecurity are greater problems in other Central American countries, they are certainly a strong concern in Nicaragua. According to data reported by the World Bank, Nicaragua’s national homicide rate in 2012 was 11 homicides per 100,000 inhabitants.7 The national homicide rate increased 72 percent between 1998 and 2009 (DevTech 2013). However, there is geographic heterogeneity in these rates. Homicide rates in Nicaragua are particularly high in the RACCS, with rates as high as 39 per 100,000 inhabitants. In the RACCS overall, robbery, rape, assault, and homicide rates increased 150 percent, 145 percent, 53 percent, and 105 percent, respectively, between 1998 and 2011 (Ramirez 2013). More recently, civil unrest in the RACCS has increased as a result of government plans to build an interoceanic canal. According to the United Nations, the complex interconnections between violence and poverty erode social and human capital and harm the society-building processes necessary for human development. Large wealth and income inequalities, low law enforcement capacity, the presence of youth gangs involved in drug trafficking, and political corruption undermine the building blocks for economic growth and democracy in Nicaragua and other Central American nations. In addition to extreme poverty and security issues, Nicaragua has poor educational opportunities and attainment. USAID’s Country and Development Cooperation Strategy FY 2013–FY 2017 (USAID/Nicaragua 2013) mentions three key education challenges in Nicaragua: (1) low enrollment levels, (2) high student dropout rates, and (3) low reading levels. For example, 16 percent of primary students dropped out during the 2012 school year alone (EDUQUEMOS 2015). In 2013, 80 percent of 3rd-grade students in Nicaragua performed at levels I and II in Third Regional Comparative and Explanatory Study (TERCE) reading tests, meaning that they couldn’t understand unfamiliar texts, establish relationships, or interpret and infer meaning in simple reading passages (Flotts et al. 2015). Education outcomes are even worse in the RACCS. According to the 2007 education census, 45 percent of school-age boys and 40 percent of school-age girls in the RACCS were out of school (MINED 2008), while 7 See http://data.worldbank.org/indicator/VC.IHR.PSRC.P5. 2 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH illiteracy rates for this population were around 25 percent. A CARS baseline study in 2014 found that only 5 percent of first grade students, 40 percent of second grade students, and 48 percent of third grade students read with fluency (CARS 2016e). As such, CARS faces the challenge of improving education and security outcomes in a region in which most school-aged children have reading deficiencies and face nontrivial obstacles to attending school on a regular basis. The infeasibility of working directly with public schools in the region when CARS began working further complicated the project’s ability to completely fulfill its goals. Studies have shown a strong correlation between violence and a lack of educational opportunities in Nicaragua. Data for 2011 show that both victims and perpetrators of violent acts tend to be individuals with low levels of education. Among victims and perpetrators of rape, for example, 67.5 percent and 60.5 percent, respectively, had an education equivalent to primary school or less. For robbery, the respective percentages were 30.3 and 58.9; for injury, 49.6 and 51.9; and for homicide, 63.0 and 54.9 (Ramirez 2013). These data are consistent with the wider literature that shows that crime is often an occupational choice (Blattman and Ralston 2015). Theoretically, improving educational outcomes could lead to improved employment opportunities, which could then lead to a decrease in crime and violence. In the face of these challenges, USAID has prioritized education and security projects for its programming in Nicaragua, particularly in the RACCS. Implemented in 5 of the 12 RACCS municipalities, the CARS intervention is an investment to improve early grade reading skills and security through an integrated approach that includes the development of locally relevant reading materials, teacher training and technical assistance, community strengthening efforts, and information generation and application. Moreover, to sustain investments beyond the USAID funding timeline, CARS builds the capacity of local organizations to administer early reading and community development programs. Figure I.2 summarizes how CARS’ four programmatic components—(1) formal and nonformal reading programs, (2) community engagement, (3) local capacity development, and (4) knowledge generation and management—are intended to diminish community insecurity,8 while improving overall educational outcomes. The underlying hypothesis for the package of interventions contends that students who fail in school, particularly in early primary school, are more likely to drop out with few skills, which places them at increased risk of becoming involved in illicit activities within a few years. CARS’ combination of new reading materials, teacher training, educational approaches, and after-school programs is designed to help students gain strong reading and socioemotional skills and improve their attachment to school, thus enabling them to succeed in the early grades of school and beyond. To complement reading programs, community and parent engagement efforts—including parent schools and community action plans (or CAPs)—are designed to facilitate parental engagement and 8 In the context of CARS, community insecurity is defined as an unsafe community environment, particularly an educational environment in which children cannot commute to school safely or feel at ease at school due to physical or psychological abuse from teachers and other children, poor school conditions, or the threat of violence. “The goal is for boys and girls to be successful in reading and writing . . . learning to decode symbols, form syllables and words to understand what they read. . . . We wanted more than anything else to build an educational foundation that kids could use in the future.” — USAID representative 3 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH community investments in early reading and security. (Parent schools would provide parents with a forum to discuss important topics in early reading, child development, and community security, and CAPs would provide parents with an opportunity to develop and execute critical school and community improvement projects with USAID funds.) In addition, technical assistance to local nongovernmental organizations (NGOs) is intended to strengthen these organizations’ capacity to administer CARS reading and community engagement programs, as well as to manage similar programs in future years. In the longer term, this combination of interventions is expected to lead to safer communities and to reduce the economic and social vulnerability of children and youth as well as the communities in which they live. Figure I.2. CARS logic model COMPONENTS AND ACTIVITIES EXPECTED RESULTS 1. Formal and nonformal reading programs • 1st-grade readiness (preschool) curriculum and teacher training • 1st—3rd grade private school curriculum and teacher training • Community after-school program (EpC) • Generation of culturally￾appropriate materials • Weekend teacher training (diplomado program) 2. Community engagement • Community action plans • Parent schools 3. Local capacity development • Technical, programmatic, and financial assistance to local NGOs • Information and experience exchange 4. Knowledge generation and management • Generation and dissemination of evidence on education and security • Improved school engagement • Improved reading skills • Improved socioemotional skills • Increased achievement and reinsertion; reduced absenteeism, drop-out, and repetition • Increased community engagement and parental investment; safer school environment • Increased institutional capacity among NGOs and local stakeholders • More informed decision making among donors and policy makers Reduced economic and social vulnerability; safer communities One notable feature of CARS is that it aims to improve reading and educational outcomes while integrating ethnically, culturally, and linguistically diversified communities through joint planning and local development efforts. Equally important, CARS’ design and implementation seeks to respond to inequality in gender roles that affects family structures and community participation. Educational and recreational activities are executed with the intention of ensuring greater balance between genders and across participating indigenous groups in both community participation and access to resources. 4 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH DevTech Systems, a U.S.-based development firm, leads the CARS program. Working in conjunction with six local NGOs and community leaders, DevTech built a local presence in the region shortly after signing the CARS contract and began executing these activities in 2014. DevTech expects to continue its work through March 2019.9 Together, DevTech and the six implementing NGOs are known collectively as the CARS Team. USAID supervises the CARS contract and provides key guidance on major activities and deliverables. B. Literature review USAID has made reading central to its education strategy: Goal 1 is to improve reading skills for 100 million children in primary grades by 2015. USAID has invested heavily in early grade reading assessments (EGRAs) and related programs, and has a strong interest in identifying the most cost-effective ways for developing countries to achieve significant improvements in this foundational skill (USAID 2011b). CARS simultaneously targets four system components that have been identified as effective for reading instruction when implemented simultaneously: (1) teacher training and support, (2) student assessment and tracking, (3) community and parental support, and (4) texts and materials (Comings 2012). Below, we summarize the evidence on these components. Teacher training in reading techniques. One aim of CARS is to improve reading skills through training teachers in new techniques. Evidence suggests that training teachers in new pedagogical techniques does help to improve learning outcomes. A variety of programs worldwide have addressed deficiencies in teaching reading by providing teachers with professional development through in-service training, lesson plans, appropriate materials, and follow-up support. For example, EGRA reading interventions in Liberia and Kenya and the Systematic Method for Reading Success program in South Africa all have demonstrated improvements in reading (Crouch et al. 2009; Piper 2009). In the United States, a rigorous evaluation of the Early Reading First program (which created preschool centers of excellence and focused on teachers’ continuous, intensive, and classroom-focused professional development) showed positive impacts on children’s print and letter knowledge, as well as on classroom environment and teacher practices (Jackson et al. 2007). In Latin America and the Caribbean, the Centers for Excellence in Teacher Training (CETT) pedagogy and tools are becoming more widely used and show promise, but they have not been evaluated rigorously. An assessment of CETT showed success at improving teacher knowledge and practices as well as improving student achievement (Chesterfield and Abreu-Comps 2011). However, the evaluation used a pre-post design without a counterfactual. CETT also may not be cost-effective (USAID 2011). Part of the expense is due to the fact that training takes over two years and includes in-person coaching follow-up. A potential evaluation could explore the efficacy of shorter duration of training or of other less expensive strategies for teacher follow-up. In addition, there is a growing literature in developing countries that focuses on teacher incentives and their effects on student achievement (Glewwe et al. 2010). A potential 9 These NGOs are (1) Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense (URACCAN), (2) Fundación Zamora Terán (FZT), (3) Fundación Hermanamiento RAMA (FHR), (4) Acción Médica Cristiana (AMC), (5) Fundación Yo Quiero Ser Feliz (FQSF), and (6) Bluefields Indian and Caribbean University (BICU). 5 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH opportunity would be to study how incentives can be incorporated or complemented with teacher training programs. Child assessment. CARS plans to track the evolution of reading skills among children whose reading skills are systematically assessed. Although evidence shows that teachers need support in differentiated instruction and in tracking learning over time, there are limited results about what assessment tools are best for each context, how often they should be used, and how to help teachers translate results into better classroom practices (Korda and Piper 2011). In Liberia, education officials used a randomized trial to test two interventions: (1) a “full version” of the EGRA Plus program, which focused on teacher training in conducting and using reading assessments and providing communities with information on student performance, and (2) a “light version” of the EGRA Plus program which focused on the community information piece. Both versions showed impacts, with the full version showing much stronger effects on reading skills (Korda and Piper 2011). However, no cost data were gathered to see which version might be more cost-effective. Unfortunately, no studies of this kind have been completed in the LAC to date. Parental and community engagement. CARS dedicates a full component to engaging parents and the community in the education of children. Evidence suggests that some types of community engagement—in the form of campaigns, meetings and forums, referenda, and consultation and collaboration with authorities and businesses—may have positive effects on some education outcomes (The Dialogue 2016). Many initiatives across the LAC region have sought to include families, communities, and other stakeholders in promoting a culture of learning. The hope is that families and communities that are informed and engaged in education can promote accountability and learning by monitoring education performance, advocating for improved services, and encouraging learning in and out of school (Bruns et al. 2011). To date, evidence in Latin America has centered largely on school-based management reforms, which involve giving a school board or committee more autonomy or resources. Overall, the literature demonstrates that school-based management can have positive effects on attendance, repetition, and failure, but mixed results on test scores (Barrera-Osorio and Linden 2009; Jimenez and Sawada 1999; Gertler et al. 2010). Parental and community engagement initiatives can reinforce and improve school-based efforts to increase student test scores and decrease absenteeism (Barr et al. 2012; Pradhan et al. 2014). Globally, there are mixed findings concerning efforts to inform families and communities— through report cards or other mechanisms—about the state of learning in their area, and in this way to increase engagement in children’s schooling. Research shows that only providing information to communities appears to be ineffective, whereas providing a recommended strategy for addressing low learning levels is more promising (Banerjee et al. 2010; Nguyen and Lassibille 2008; Andrabi et al. 2009). Another strategy that has successfully been used is reading-focused community engagement. Recruiting and training community teachers and volunteers in how to teach low-performing students to read as well as effective remediation provided by locally trained community teachers have shown promise in India and in Kenya (Banerjee et al. 2010; Banerjee et al. 2007; Duflo et al. 2015). First-grade readiness (preschool). CARS has an activity that focuses on the transition to 1st grade through the use of preschools (understood as any schooling before 1st grade). Evidence 6 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH suggests that some preschool activities may be associated with improvement in some education outcomes later in school, especially in 1st grade, and that investments in preschool can generate larger rates of return than investments in later years (Carneiro at al. 2003). There is some evidence that high quality teacher-child relationships in kindergarten have a positive effect on math achievement in 1st grade, but no effect on reading (McCormick et al. 2013). Other studies find that children with more kindergarten made a better transition to 1st grade, other things being equal (Entwisle and Alexander 1998). According to a recent meta-analysis of rigorous studies on early childhood programs to improve language and literacy outcomes for preschoolers, the strongest evidence of positive literacy and language outcomes at the end of preschool and on follow-up measures was associated with balanced approaches—including, phonemic awareness, phonics, and other skills—and child-initiated activities (Chambers et al. 2015). Evidence from the United States suggests that children who participate in pre-K programs show better measures of literacy, language, and math over the pre-K year compared to children who did not participate in such programs, but these differences were no longer significant by the end of 1st grade. However, progression to 1st grade and attendance in 1st grade were still significantly higher among those who participated in the pre-K program (Lipsey et al. 2015). Learning and reading materials. As part of CARS, students and teachers use new reading materials that are culturally and linguistically appropriate. The general pattern of findings emerging from the LAC and elsewhere is that materials can matter—but not surprisingly, it is their content, appropriateness to the learner, and how teachers utilize them that matter most. C. CARS activities and goals When the CARS began, U.S. policy prohibited direct USAID support of Nicaraguan public schools. For this reason, CARS targeted privately managed and subsidized schools with in￾school (formal) reading programs, and designed before and after-school, community-based (nonformal) programs to reach children who attend public schools, called Espacios para Crecer or EpCs.10 In the RACCS, private and subsidized schools charge modest fees and have somewhat better infrastructure than public schools. However, private and subsidized schools are not elite institutions, and the students who attend these schools have similar educational attainment and risk factors as students who attend public schools. As such, CARS considers children served by both formal and nonformal reading programs to be at-risk of poor educational outcomes and school drop-out. Below we describe each of CARS’ four components in more depth and provide a summary of each component’s planned activities, target populations, and implementation goals by September 2017 (See Table I.1). This discussion reflects the basic design of CARS activities that was finalized in early 2015. 10 This situation changed in late 2016, when CARS received approval to implement formal reading programs in public schools. 7 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH Table I.1. CARS implementation plans and performance targets: 2014-2016 Component Subcomponent Activities Target population and targets 1. Formal and nonformal reading programs 1st grade readiness activities Formal reading program: Teacher workshops on preschool/transition to 1st grade 1,910 third-year preschool students in 70 preschools (ages 4–5) Quality reading instruction and materials Formal reading program: Active teaching-learning methods and coaching 2,440 students in grades 1–3 in 40 private schools in 5 municipalities Formal reading program: Diploma program for preservice, in-service, and first-year teachers (diplomado) Nonformal reading program: EpCs 8,150 students in grades 1–3 who are failing, have learning difficulties, or have dropped out; and 1,500 who have never attended school in EpCs (total of 270 educational communities) Formal and nonformal reading programs: Adaptation of existing and creation of new learning materials to local indigenous languages All 12,500 students participating in formal and nonformal reading programs 2. Community engagement Community engagement Development of community action plans (CAPs) to solicit resources through CARS and implement activities All educational communities in which reading programs are implemented Communications Parent schools to encourage promotion of education at home 3. Local capacity development Training and technical assistance Training of NGO personnel in the relevant CARS activities so that they can administer the programs in communities 6 NGOs Information and experience exchange Distribution of information to communities for educational purposes NGO workshops to share lessons learned Sub-grants Selection of local NGOs to implement activities under components 1 and 2 4. Knowledge generation and management Data for decision making Generation of reading and security data and distribution to policymakers to influence their decision making NGOs, donors, and policymakers in the RACCS USAID data support Generation of implementation and outcome data and sharing of data with USAID 1. Formal and nonformal reading programs By design, CARS featured a combination of formal reading programs (educational approaches and materials and teacher training and assistance provided to private preschools and primary schools) and nonformal reading programs (after-school programs for school-age children, called EpCs). Among formal and nonformal reading programs, CARS planned to serve a total of 12,500 children who (1) were in privately managed schools (from kindergarten to grade 8 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH 3), (2) were school-age but had never attended school, (3) had dropped out of school, or (4) were overage or had learning difficulties in grades 1 to 3 in public school. The overarching goal of these interventions was to improve children’s reading fluency and comprehension. As a critical part of the CARS model, DevTech planned to train and support local NGOs to administer all CARS formal and nonformal reading programs by using a train-the-trainer model. Below, we discuss each program’s original plan for activities and target populations in more depth. • First-grade readiness (preschool). The goal of this program was to strengthen community￾based private preschools in the five municipalities of the RACCS through a series of workshops with preschool teachers. Led by local NGOs, these workshops would introduce a preschool educational approach and learning materials designed to aid students’ transition to 1st grade, with a special emphasis on skills for reading—including, vocabulary, comprehension, and letter recognition—fine motor skills, social and emotional skills, communication, and creative play. The workshops also would train teachers in involving parents more in their children’s education to help them internalize the importance of preschool. With this component, CARS planned to reach 2,000 preschool students in 70 preschools. • First- to 3rd-grade private school. Under this program, local NGOs would train 1st-, 2nd-, and 3rd-grade teachers at private schools on active teaching-learning methodologies and formative reading assessments. In addition, the NGOs would offer teachers coaching support and follow-up through classroom observation, with an emphasis on conducting formative reading assessments, interpreting the assessments correctly, and modifying class activities to correct deficiencies identified in the assessments. To complement teacher training, CARS would contextualize and disseminate early grade reading educational approaches and instructional materials to use in class and donate books and other titles to form mini libraries at participating schools. With this component, CARS planned to reach 2,500 students in grades 1 to 3 in 40 privately managed schools in five municipalities of the RACCS. The duration of training and follow-up would be 20 months from start to finish. Under this component, the NGOs also would offer teachers a weekend certification (or diploma) program in innovative methodologies for teaching early grade reading. Drawing primarily from Aprendo, Practico, Aplico (APA) and Fónico, Analítico, Sintético (FAS) teaching methods, the diploma program would emphasize the use of proven, simple, and effective teaching and learning techniques that teachers could use to improve early grade reading among their students. • EpCs. Under this program, NGOs would establish after-school programs in communities with public and private schools. These after-school programs, or EpCs, would serve children who are having difficulties at school, have dropped out of school, or have never attended school. EpCs are led by teachers called “facilitators.” For three hours before or after each 9 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH school day, EpCs offer activities designed to support early reading (through reading time and help with homework), socialization (through playtime), and individual growth. The EpCs work with Quantum Learning (QL) methodology, whose learning principles are based on education theories such as accelerated learning, neurolinguistic programming, experimental and cooperative learning, and effective instruction. Working through local NGOs, CARS planned to serve a total of 8,000 children with EpCs: 6,500 public school students in grades 1 to 3 who were failing, had learning difficulties, or had dropped out and 1,500 children who had never attended school.11 CARS’ initial goal was to open 270 EpCs in stages, beginning with 10 pilot EpCs. Each of these EpCs would provide approximately 30 children with 18 months of programming. However, due to capacity constraints and the need to first pilot EpCs in the region, the EpCs were initiated in staggered cohorts over the span of three years. “[The EpC] is where the child can be exposed to another environment where teaching is focused on games, activities, music, etc. EpCs give [children] an opportunity to fall in love with reading and education. . . . It’s a space to share their experiences, and do work in a fun way.” — DevTech representative 2. Community engagement activities The objective of this component was to promote community involvement in education and security. In all schools participating in formal and nonformal reading programs, CARS envisioned establishing Escuelas para Padres, or parent schools, as well as executing the CAPs to make critical school and community improvements. Similar to the reading programs discussed above, local NGOs would be trained on community engagement activities, with support from DevTech, and would then train EpC facilitators and teachers (under the train-the￾trainer model) to administer the parent schools and facilitate the CAPs. We provide more detail on these activities below. “It was important that parents and schools started communicating. . . . We want parents’ involvement to be a learning process to first and foremost communicate to parents why we’re placing so much importance on reading and writing, and how they can complement [our] work in schools.” — USAID representative • Parent schools. DevTech would work with NGOs to establish and administer parent schools, consisting of eight monthly sessions designed to facilitate parents’ reflection on reading and security and to provide parents with knowledge and skills that would allow them to become more involved and committed to the development of their community, their children, and their children’s school. These schools would aim to bring together parents and community leaders to discuss security issues and plan and carry out actions to promote reading and ways to upgrade or improve security in the community. According to the program design, DevTech and NGO staff would help facilitators establish parent schools at all EpCs and help teachers and principals establish parent schools at all preschools and primary schools that received training and CARS materials. Parent schools would provide a 11 CARS does not work directly with public schools in Nicaragua. For this reason, EpCs can.support public school students who might need additional help, but they provide this support separate from the public school. This is why in the context of EpCs, the program works not with schools but with “educational communities,” which comprise the children in the community who attend the public school, but also children who have left the public and private schooling system, children who never attended school, and the surrounding community of parents and leaders. 10 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH forum for community leaders to speak with parents about important topics related to health and education. Topics to be discussed at parent schools could include strategies to promote reading, organizing to keep schools clean, the importance of good hygiene and health, good relations between parents and their children, and strategies to deal with bullying inside a school and build self-esteem. • CAPs. In consultation with CARS and trained NGOs, parents involved in parent schools and community leaders would develop CAPs, which would be used to seek municipal resources and implement investments in schools and the larger community. CAP improvements could encompass many areas, including additional teacher training, didactic materials, school infrastructure improvements, and school security improvements such as fences and gates. This is particularly important in the context of the communities that CARS works with, which have large infrastructure needs. Some interventions that could potentially be funded by CAPS included fencing, cleaning of public areas, improvements of schools and public libraries, and neighborhood beautification. CAP-funded improvements were often designed to help children feel safer and more motivated to attend school, as well as to contribute to a stronger sense of security in the larger community. The CAP development process itself could potentially be valuable for participants, as it would empower them to solve community problems and help leverage local and private sector resources to do so 3. Local capacity development DevTech would issue grants to several local NGOs to implement formal and nonformal reading programs, as well as to conduct community engagement activities. DevTech would provide these NGOs with technical support to help administer reading and community engagement activities, as well as technical assistance and training focused on building NGOs’ overarching financial and administrative capacity. Under this component, DevTech would also develop and disseminate communication products to promote CARS and early reading in general and to establish a community of learning among participating NGOs around the themes of early grade reading and citizen security. We provide more detail on these activities below. • Technical and financial assistance to local NGOs. In preparation for EpC implementation, DevTech staff would train contracted NGOs in quantum learning (QL) methods and enrichment activities geared toward improving students’ socioemotional skills and school attachment. Similarly, DevTech would provide technical training for NGOs that were contracted to administer the diploma program, the private and subsidized school programs, and the EpCs. This training would enable the NGOs to train EpC facilitators and teachers in these methodologies and activities. CARS would also support the NGOs with their financial and administrative responsibilities through regular check-ins; provide NGOs with basic trainings in financial management, human resources, and organizational structure; and offer them the opportunity to access institutional strengthening assistance through outside consultants. • Information and experience exchange. Under this component, DevTech also planned to develop and distribute radio spots and programs, bulletins, and other means of communication directed at parents, teachers, school children, families, and leaders. The objective of these communication products was to spur community discussions around 11 I. INTRODUCTION AND CARS DESIGN MATHEMATICA POLICY RESEARCH education, security, empowerment, participation, inclusion, and sustained community development from the bottom up. Moreover, CARS planned to organize periodic NGO workshops with a focus on early childhood education and citizen security to share experiences, lessons learned, and best practices and to propose adjustments in future endeavors. 4. Knowledge generation and management With this component, USAID planned to improve the use of evidence to drive decision making by providing data and information support to policymakers and policy influencers. Under this component, CARS planned to make information on CARS activities, outputs, and outcomes available to donors and policymakers on an ongoing basis. This information would provide USAID with a consolidated vision of activities that affect targeted municipalities in the RACCS and, thus, allow for better donor harmonization and alignment of initiatives related to reading and security in the region. Relationships between components. CARS envisioned potential overlap between formal and nonformal reading programs in some schools, such that it was possible for a single school to have an EpC as well as preschool and grade 1 to 3 components. However, in general, public schools would have access to the support of EpCs (but not preschool and primary school components) and private schools would have preschool and primary school components but not necessarily EpCs. (The lack of formal reading programs in public schools reflected USAID’s policy of not working directly with public schools in the region.) CARS also envisioned full overlap between reading programs and community engagement activities: in communities where CARS and its partner NGOs established an EpC or a formal reading program, they also planned to conduct community engagement activities (including, CAPs and parent schools). 12 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH II. RESEARCH QUESTIONS AND METHODS In this chapter, we discuss the performance evaluation’s primary research questions, as well as the data collection and analysis we conducted to answer these questions. A. Evaluation questions, study design, and data Motivated by a desire to inform CARS’ remaining activities and program expansion plans and USAID’s future projects in the region, USAID/Nicaragua representatives identified several research questions for this evaluation. These questions can be divided into five global sets of questions: 1. How is CARS being implemented? 2. Is implementation occurring as planned? 3. What are the key facilitators and barriers to CARS implementation? 4. Does CARS appear to be generating its desired effects? 5. What adjustments could improve CARS implementation? To answer these core research questions, Mathematica conducted a performance evaluation of CARS. A performance evaluation is designed to describe and assess the implementation of a program, including whether the program is meeting its performance targets, which aspects are working well, and what could be improved in the future. An important characteristic of performance evaluations is that they are not designed to detect program impacts; that is, they cannot be used to attribute changes in participants’ outcomes to the program (or to specific program components) because they do not have a suitable comparison group (or counterfactual) to simulate what would have happened in the absence of the intervention. (In contrast, the forthcoming EpC impact evaluation is designed to detect program impacts, given its reliance on random assignment.) However, performance evaluations often provide insights into program implementation and potential effects of large-scale programs, as well as actionable recommendations to improve such programs. Study design. This is a mixed-methods evaluation: it uses a mix of quantitative data sources (such as available CARS monitoring and evaluation [M&E] indicators, child assessment data, and data from structured interviews) and qualitative data sources (such as programmatic reports and stakeholder focus groups) to answer the research questions. Using both qualitative and quantitative methods allows us to address each research question with all available data sources and to triangulate qualitative and quantitative findings. Data sources. We used four types of quantitative data to better understand program implementation and the potential effects of CARS: (1) CARS M&E data, (2) baseline and follow-up survey data of students selected to be served by the EpCs (collected for the EpC impact evaluation)12, (3) student assessment data (cited by CARS in quarterly reports as well as collected by a third-party data collector for the EpC impact evaluation), and (4) numeric data from structured educator interviews. In general, the quantitative data answered performance 12 These data include baseline profiles for all children assigned to EpCs in Cohorts 1 and 2, but household information for only children assigned to Cohort 2 EpCs, and follow up data for only children assigned to Cohort 1 EpCs. 13 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH questions that could be described in numeric form, such as whether CARS met implementation targets and whether students reached desired reading outcomes. To complement the findings generated by the quantitative data, we reviewed programmatic reports; held focus groups and semi-structured interviews with parents, community leaders, CARS and NGO representatives, and Sistema Educativo Autonómico Regional (SEAR) and USAID representatives; and asked open-ended (qualitative) questions in structured educator interviews with principals, teachers, and EpC facilitators. Qualitative data collection helped us understand how the CARS program works in practice, stakeholders’ perceptions of the quality of program implementation and its effects, what barriers may be impeding implementation, and what adjustments might improve the program in the future. (See Table II.1 for a summary of which data sources were used to answer which research questions.) Table II.1. Data sources for research questions Research questions Quantitative sources Qualitative sources CARS M&E dataa Baseline and follow-up survey datab Student assessment dataa,b (EGRA ) Structured educator interviewsb CARS reportsa Stakeholder interviews and focus groupsb 1. How is CARS being implemented? 1.1. What key activities have been implemented? When, where, and by whom? . . . . X X 1.2. What organizations are involved in CARS? X . . . X X 1.3. What communities, schools, and children are participating in CARS? X . . . X X 2. Is implementation occurring as planned? 2.1. Are services and materials reaching the desired target populations, particularly in the case of EpCs? . X X X X X 2.2. Are services and materials meeting implementation goals?c Why or why not? X . . . X X 3. What are the key barriers and facilitators of project implementation? 3.1. What are the key facilitators of project implementation? . . . X X X 3.2. What are the key barriers of project implementation? . . . X X X 4. Does CARS appear to be generating its desired effects? 4.1. Are teachers applying early grade reading approaches, including the use of new materials and assessment tools? If so, what is their perceived effect on student performance? X . . X . X 14 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH Table II.1. (continued) Research questions Quantitative sources Qualitative sources CARS M&E dataa Baseline and follow-up survey datab Student assessment dataa,b (EGRA ) Structured educator interviewsb CARS reportsa Stakeholder interviews and focus groupsb 4.2. Is there suggestive evidence that preschool services, after-school services, private school services, and teacher training are improving reading skills, socioemotional development, and other educational outcomes? X . X X X X 4.3. What are the major constraints to improving reading performance and establishing a more positive community environment? . . . X X X 4.4. Is there suggestive evidence that the project’s gender approach is reducing gender disparities? X . . X . X 4.5. Are outreach and community engagement services increasing parent and community support for preschool or education? X . . . X X 4.6. What types of outreach and awareness efforts to target populations are most successful in increasing community engagement? X . . . X X 4.7. Is the local development component increasing NGO capacity? X . . . X X 4.8. Does the knowledge generation and management component appear to be increasing knowledge, skills, and resources? X . . . X X 4.9. Are CARS activities sustainable? . . . . X X 5. What adjustments could improve program implementation? 5.1. What activities could be strengthened or modified? X . . X X X Note: Baseline and follow up survey data describe different populations: baseline data include household surveys for Cohort 2 children, whereas follow up data include household survey data for Cohort 1. aData obtained from CARS. bData collected independently of CARS. cImplementation goals are defined as CARS M&E goals, as well as internal CARS benchmarks with respect to the frequency of service provision—such as the number of school visits conducted by CARS staff. 15 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH B. Data collection Mathematica staff worked with a data collection partner, Fundación Internacional para el Desafío Económico Global (FIDEG), to conduct data collection visits in September and October 2016 to educational communities participating in CARS. For the purposes of sampling for the performance evaluation, we define an educational community as the principal and teachers at a primary school, the facilitator of any EpC that operated on or near school grounds, the parents whose children attended the school or EpC, and any community leaders involved in school affairs. 13 FIDEG visited 36 educational communities across all five CARS municipalities: 18 communities with private or subsidized schools and 18 communities associated with public or community schools. We randomly selected the educational communities that data collectors would visit, but selection was constrained so as to select (1) at least two educational communities per municipality; (2) a mix of Cohort 1 and 2 EpCs; and (3) at least three communities with each combination of CARS services (including a combination of EpC, preschool, and 1st- to 3rd￾grade activities). This sampling approach ensured that visited communities reflected the full variety of educational communities and configurations of CARS activities. To capture the diversity of linguistic contexts in which CARS is implemented, we also included two Miskitu￾speaking communities, two Creole-speaking communities, and one Ulwa-speaking community in the sample. We trained bilingual data collection personnel to conduct interviews and focus groups in Miskitu or Creole, according to the primary language spoken in the community14. (See Table II.2 for a summary of educational communities visited during data collection.) 13 This differs from the definition of educational community used by Nicaraguan authorities, which is the community composed of parents, teachers, and students whose activities coincide in the same school center (according to “Ley de Participación Educativa,” 03/12/2002). 14 We chose some communities whose primary language was not Spanish in order to be as inclusive as possible. 16 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH Characterizing the five municipalities in which CARS works Although the RACCS can be seen as homogenous when compared to the rest of the country, there are important differences across the municipalities that comprise it. Corn Island and Bluefields are predominantly urban, whereas Laguna de Perlas, Kukra Hill, and Desembocadura are predominantly rural. Accessing these remote, rural communities in the RACCS can be very challenging and require long hours of transport by land, water, animal, or by foot. Access to remote communities is difficult or impossible during the rainy season due to rising water levels. Rural CARS communities’ remote locations and limited access pose significant challenges for CARS programming, which envisions regular visits to all participating communities. All CARS municipalities have a sizable Spanish-speaking mestizo population, but Corn Island and Laguna de Perlas have around 50 percent and 40 percent of residents that identify as Creole, respectively, who primarily speak Creole. In Desembocadura del Rio Grande, around three-quarters of residents identify as Miskitu, and most of them speak Miskitu as their mother tongue. Corn Island and Laguna de Perlas also have significant Miskitu populations (Caracterización Sociodemográfica de la Región Autónoma Atlántico Sur [RAAS] 2005). Education outcomes are uneven across municipalities: adult illiteracy ranges from a low of 9 percent in Corn Island and 17 percent in Bluefields to around 30 percent in other municipalities. In terms of economic activities, Bluefields and Corn Island have the lowest percentage of people employed in agriculture, cattle, and fishing, whereas Laguna de Perlas, Kukra Hill, and Desembocadura have around 65 percent to 70 percent of their populations working in these sectors. Other income generating activities in the region include fisheries and tourism. In terms of living conditions, most families across all five municipalities live in houses (92 percent on average). However, in Laguna de Perlas and Kukra Hill, around 14 percent of families live in chozas or huts. Most households in Bluefields and Corn Island have access to electric power, half of the households in Kukra Hill and Laguna de Perlas use gas for lighting, and most households in Desembocadura use alternative sources of lighting other than electricity or gas (RAAS 2005). In addition, there are varying degrees social unrest in the communities in which CARS works. In particular, the proposed construction of a canal through the Punta Gorda areas in Bluefields has heightened tensions, to the extent that armed groups currently patrol the Punta Gorda area in the Municipality of Bluefields. 17 18 Table II.2. CARS educational communities visited for the performance evaluation Educational communities with: Municipality Bluefields Corn Island Desembocadura Kukra Hill Laguna de Perlas All municipalities Total Visited Total Visited Total Visited Total Visited Total Visited Total Visited Any EpC 105 13 7 2 5 3 30 5 30 3 177a 26 Cohort 1 EpCs 22 5 4 2 4 3 9 3 10 2 49 15 Cohort 2 EpCs 53 8 0 0 0 0 16 2 17 1 86 11 Cohort 3 EpCsb 44 5 5 1 5 3 10 2 9 1 73 12 Formal reading program in primary school 30 12 5 1 0 0 2 1 1 1 38 15 Formal reading program in preschool 68 15 6 2 4 3 5 3 6 2 89 25 Any CARS component 123 18 10 4 7 4 31 6 31 4 202 36 Note: Educational communities in Cohort 3 were sampled for their involvement in earlier EpC cohorts. aThe total number of educational communities with any EpC doesn’t equal 270—the total number of EpCs established from 2014 to 2016—because some educational communities had more than one EpC. bThis sample was designed to include only Cohort 1 and 2 communities, as only Cohort 1 and 2 communities had at least six months of EpC implementation. However, we interviewed individuals trained in 2016 to be Cohort 3 facilitators when no facilitators from prior cohorts were available for interviews. II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH At each educational community, we conducted one in-person interview with the principal, up to three interviews with teachers and EpC facilitators, and two focus group discussions—one with parents and another with community leaders (see Table II.3 for the number of interviews and focus groups that took place during the visits). Twenty-one of the 26 interviewed EpC facilitators (around 80 percent) were still serving as EpC facilitators or had worked as facilitators as recently as 2016. Thirty-five of the 54 interviewed teachers (around 65 percent) were preschool, 1st-, 2nd-, or 3rd-grade teachers who received CARS training and materials under the formal education program, whereas 19 interviewed teachers did not receive CARS training and materials because their community received only the EpC component. Interestingly, 9 of the 26 EpC facilitators (around 35 percent) also worked as preschool or primary school teachers at sampled educational communities. In these cases, respondents were interviewed as both EpC facilitators and teachers. In late 2016, Mathematica staff and their data collection partners also interviewed several key stakeholders by phone and in person, including seven DevTech staff members, seven representatives from each of the six implementing NGOs, two USAID/Nicaragua representatives, and two SEAR authorities (see Table II.3 for the total number of stakeholder interviews). To maximize the value of these in-person interviews, we targeted DevTech and NGO staff members with the most detailed knowledge of program implementation. Table II.3. Sample sizes for primary data sources Type of data source Number of interviews or focus groups Focus groups . Parent focus groups (with a sample of parents from each community) 36 Community leader focus groups (with a sample of leaders from each community) 35 Total focus groups 71 Educators . Private school principal interviews 18 Public school principal interviews 17 Teacher interviews 54 EpC facilitator interviews 26 Total educator interviews 115 Implementers and authorities . Implementing partner (DevTech) interviews 7 Implementing partner (NGO) interviews 7 USAID interviews 2 Educational authorities (SEAR) interviews 2 Total implementer/authority interviews 18 19 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH C. Data analysis General approach to quantitative analysis. In analyzing numeric data provided by CARS and quantitative data from structured interviews with educators, our general approach was to compare implementation outputs (such as the number of reading materials distributed) and program outcomes (such as the percent of students reading at grade level) to program goals and targets, whenever possible. Then, we made summary statements on whether program goals and targets were not met, met, or surpassed. General approach to qualitative analysis. Focus groups and interviews with principals, parents, community leaders, and implementers gave us a wealth of qualitative information. We analyzed this information to identify patterns of consensus and instances of divergent or contradictory views. We used two primary methods of analyzing these qualitative data to address our research questions: (1) conceptual categorization and (2) data triangulation. To best uncover patterns, themes, and issues in the qualitative data, we developed a coding framework with conceptual categories linked to the logic model (divided into the three categories of CARS design, implementation, and effects) as well CARS components (divided into formal reading programs, nonformal reading programs, community engagement efforts, local capacity development, knowledge generation and management). Organizing qualitative data into these categories allowed us to access data on a specific topic quickly and to organize information in different ways to identify themes and compile evidence that supported them. Because our qualitative analysis incorporated data from several different sources, we also used triangulation to test for consistency in the findings from these data sources. As a result of comparing and contrasting input from different types of stakeholders, the findings in this report largely reflect themes mentioned by a range of stakeholders—including educators, CARS implementers and parents—rather than a single type of stakeholder. Mixed-methods analysis. Below we provide more information on how we structured our quantitative and qualitative analyses to answer the research questions outlined above. • Question 1: How is the CARS intervention being implemented? To gain a comprehensive understanding of how the intervention is being implemented, we used programmatic reports and CARS M&E data to identify the number of children involved in each reading program, the number of educators trained, and the number of schools assisted. We also used the results of structured interviews with teachers to construct additional process indicators, including whether teachers attended training or received CARS materials for their classroom and how many materials and follow-up visits educators reported receiving from CARS. • Question 2: Is implementation taking place as planned? A key part of characterizing implementation is using programmatic reports and M&E data to determine the extent to which CARS met its implementation targets at the time of our data collection. For example, we used quarterly reports to compare the number of community action plans that were planned by mid-2016 and the actual number that were produced. In addition, we used process indicators from structured educator interviews to compare programmatic goals to actual implementation. For example, we compared the average number of CARS technical visits reported by educators to the goal of at least one visit per month. We also used 20 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH interview data to distill stakeholders’ perceptions on the quality of implementation; namely, we compared and contrasted qualitative accounts from facilitators, teachers, and principals on the content and quality of CARS training and asked stakeholders to rate their satisfaction with relevant CARS activities and services. • Question 3: What are the key facilitators and barriers to CARS implementation? We analyzed qualitative data from interviews and focus groups to assess implementation facilitators and barriers. For this exercise, we used a simple implementation effectiveness framework, in which the overall effectiveness of CARS implementation is defined as a function of: (1) CARS educational approach and materials; (2) CARS training and activities; (3) the CARS team's capacity, leadership and coordination; and (4) the school and community environment. Using this framework, we identified key barriers to and facilitators of effective implementation in each of these four categories. For this analysis, we define barriers and facilitators as factors, incentives, or circumstances that discourage or encourage the timely delivery of high quality services and goods as originally planned, as well as the desired learning and security outcomes. • Question 4: Does the CARS intervention appear to be generating its desired effects? To assess whether CARS activities are producing their desired effects, we relied on a variety of data sources, including child assessment data, stakeholder interviews, and CARS analyses. Based on structured interviews, we developed quantitative indicators of whether educators applied what they learned in training, whether they regularly applied reading assessments, whether they used CARS materials in class, and whether they believed CARS had positive effects. In addition, we constructed quantitative indicators of teaching practices that should, in theory, be influenced by the CARS formal reading program—including the use of group work, songs, and routines in class—and compared teaching practices of CARS-trained teachers (who received training and materials under the formal reading component) to teaching practices of teachers not trained by CARS (from schools that received only the EpC component, which trained facilitators, not teachers). In this analysis, the practices of nontrained teachers provides a potential counterfactual—that is, what teachers’ practices might have been in the absence of CARS. Although the difference between trained and nontrained teachers cannot be interpreted as causal impact of CARS, it does provide suggestive evidence of the potential effect of the program on teachers’ behavior in class. To complement these quantitative measures, we analyzed interview data from educators for concrete examples of behavior change and potential effects of the CARS program, as well as reasons they did not exhibit this behavior change or CARS did not have the desired effect. In addition, we analyzed qualitative data—primarily from NGO and DevTech interviews—to inform our understanding of the implementation and potential effects of the community engagement, local capacity development, and knowledge generation and management components. In addition, we relied on M&E indicators to measure the number of CAPs that were developed and implemented (under the community engagement component) and we used baseline and midline Organizational Capacity Assessment (OCA)15 scores to document 15 According to the OCA handbook (June 2012), the OCA tool was designed to enable organizations to define a capacity-building improvement plan, based on self-assessed need. This assessment was initially designed to measure 21 II. RESEARCH QUESTIONS AND METHODS MATHEMATICA POLICY RESEARCH trends in NGO capacity over the course of the intervention (under the local capacity development component). • Question 5: What early adjustments could improve program implementation? To answer the last set of questions on the possible adjustments that could benefit the future implementation of CARS, we first analyzed the barriers to effective implementation at the general CARS level, as well as at the level of each programmatic component, so as to identify areas for improvement. Then, we used the structural constraints to early reading and security initially defined by key stakeholders in interviews as a simple theoretical framework and examined the role that CARS currently plays in mitigating each of them. This allowed us to identify potential midcourse corrections—or instances in which CARS could better address key constraints to early reading and security—that could improve future CARS implementation. D. Limitations of this analysis A notable limitation of this evaluation is that it does not feature classroom observation due to time and resource constraints. We cannot verify firsthand whether educators adopted CARS teaching methods and used CARS materials in participating schools and EpCs. Because it relies largely on self-reports from educators, this analysis may overestimate the degree to which teachers and facilitators adopted CARS methodology and used CARS materials in class. Focus groups with parents and community leaders may also overestimate community and parent support for CARS activities, given respondents’ potential inclination to provide socially desirable answers, or answers that paint CARS in a positive light. Qualitative findings are also based on a sample of participating communities and thus will not necessarily fully reflect the variety of perspectives on CARS or the most common perceptions among all participating communities. In addition, all analysis of the potential effects of CARS—particularly the comparison of CARS-trained and nontrained primary school teachers’ practices—do not rely on random assignment and could be biased upward due to potential pre-existing differences between teachers trained by CARS and teachers not trained by CARS (such as potential underlying differences between private and public school teachers). Our findings can be considered as suggestive of the true effect of CARS on teacher practices and child outcomes. overall capacity of organizations funded by the President’s Emergency Plan for AIDS Relief under the New Partners Initiative. The OCA tool provides organizations with a set of criteria to assess their current management capacity to implement quality health programs, in order to identify key areas that need strengthening. 22 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH III. DESCRIPTION OF CARS STAKEHOLDERS In this chapter, we characterize the schools, communities, households, and children that are participating in CARS, highlighting key themes that emerged in interviews and focus groups, as well as relevant findings from baseline survey data. A. What organizations are participating in CARS? CARS is a joint effort between DevTech and six local NGOs. DevTech supervises overall CARS implementation, whereas the NGOs implement reading and community engagement programs with DevTech support. All of the NGOs had some experience in education or community-based development work prior to CARS. Two of the NGOs had no prior experience with formal education. DevTech staff oversee and support CARS implementation. DevTech supervises overall CARS implementation, including teacher training and follow-up, distribution of materials, and working with parents and community leaders. The core team is comprised of DevTech staff who reside in Bluefields and are supported by DevTech staff located in the United States (District of Columbia). DevTech staff in Bluefields comprise four main units: (1) education; (2) community engagement; (3) monitoring, evaluation, and knowledge management; and (4) administration (Figure III.1). The Education Unit oversees all formal and nonformal reading programs; the Community Engagement Unit oversees all community-oriented activities, including parent schools, CAPs, and events. The Monitoring, Evaluation, and Knowledge Management Unit is responsible for developing, updating, and implementing the CARS M&E plan; analyzing and disseminating implementation and outcomes data; and coordinating with external evaluations. The administration unit is responsible for administrative and financial management of all CARS activities and communications. DevTech has NGO liaisons within the Community Engagement and Education units as well as a grants and contracts specialist who coordinates closely with the program’s six NGOs to administer reading programs and community engagement activities. Together, DevTech and the six implementing NGOs are known collectively as the CARS Team.16 As illustrated in Figure III.1, DevTech’s chief of party reports directly to USAID on all CARS activities. 16 Initially, CARE Nicaragua formed part of the CARS technical team, providing technical assistance in community engagement and EpCs in Nicaragua. However, in March 2015, CARE and DevTech agreed to terminate their relationship, and DevTech hired two CARE staff as consultants to continue work on EpCs and community mobilization efforts. 23 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH Figure III.1. CARS organizational chart Source: DevTech org chart, 2016. AMC = Acción Médica Cristiana; BICU = Bluefields Indian and Caribbean University; FHR = Fundación Hermanamiento RAMA; FQSF = Fundación Yo Quiero Ser Feliz; FZT = Fundación Zamora Terán; URACCAN = Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense. Six diverse NGOs implement reading and community engagement programs with DevTech’s support. Six local NGOs form the CARS team that directly implements reading programs and community activities. These NGOs are (1) Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense (URACCAN), (2) Fundación Zamora Terán (FZT), (3) Fundación Hermanamiento RAMA (FHR), (4) Acción Médica Cristiana (AMC), (5) Fundación Yo Quiero Ser Feliz (FQSF), and (6) Bluefields Indian and Caribbean University (BICU). DevTech initiated operations at pilot EpCs in late 2014. FHR, FZT, and URACCAN took over operations shortly thereafter. FZT and URACCAN established the majority of EpCs from 2014 to 2016, with over 100 EpCs each. In addition, FHR established 41 EpCs during this time period. AMC and FQSF joined CARS in 2015 to implement preschool and 1st- through 3rd￾grade training components. BICU joined in 2016 to implement the diploma teacher training program. URACCAN and BICU are universities that operate within the Caribbean coast of Nicaragua, giving them deep knowledge of the region. In contrast, FZT and AMC are larger, national-level NGOs that have a home office in Managua and smaller offices throughout the country in different regions. FHR and FQSF are small in size and have only a local presence (see Table III.1 for additional information on NGOs). 24 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH All NGOs have some experience in education or community organizing, but two NGOs had no prior experience with formal education. When selecting NGOs to implement CARS, DevTech and USAID prioritized organizations with experience in education interventions and/or experience doing community-based work, as well as experience working in multicultural and multilingual contexts. (As an explicit goal of CARS was to build NGO capacity in the RACCS, DevTech and USAID also targeted NGOs with an existing presence in the region, or the ability to establish such a presence.) FQSF, FZT, BICU, and URACCAN all have experience in the education realm. BICU and URACCAN in particular have extensive experience in teacher training. FZT has experience with the One Laptop per Child program as well as other education￾oriented programs prior to CARS, while FQSF is a group of education and reading experts. In contrast, FHR and AMC had experience in community development and education prior to CARS, but not in the realm of formal education or reading (Table III.1). Table III.1. NGOs implementing CARS NGO AMC FHR FZT FQSF URACCAN BICU Expertise Community health and local development, with an emphasis on women, kids, and adolescents; violence prevention, and community security Community development, child labor and sexual violence prevention, technical education Social responsibility and poverty reduction, information technology Child education and human rights, community development Indigenous community development, including education and economic development Primary and secondary teacher training CARS involvement Run preschool and primary school reading programs Run 41 EpCs in Cohorts 1A, 1B, 2A, and 3 Run 107 EpCs in Cohorts 1A, 1B, 2A, 2B, and 3 Run Preschool and primary school reading programs Run 112 EpCs in Cohorts 1A, 1B, 2A, 2B, and 3; run preschool and primary school reading programs Spearhead diploma program CARS start date 2015 onward 2014 onward 2014 onward 2015 onward 2014 onward 2016 onward Municipalities Bluefields, Laguna de Perlas Kukra Hill Bluefields, Corn Island Bluefields, Corn Island Bluefields, Laguna de Perlas, Desembocadura (EpCs); Bluefields, Desembocadura (preschool and primary school) Bluefields, Laguna de Perlas, Corn Island, Desembocadura, Kukra Hill Source: DevTech quarterly reports and stakeholder interviews. B. What are the characteristics of educational communities participating in CARS? Although CARS is teaching reading in three languages, the majority of the participants receive instruction primarily in Spanish. Located primarily in Bluefields, Kukra Hill, and Laguna de Perlas, schools participating in CARS tend to serve fewer than 200 students, on average. Although formal reading programs are offered only in private and community schools-- 25 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH and EpCs are concentrated in public schools--it is common for a participating (private) school to have a formal reading program in school and an EpC. Schools served by CARS generally have few financial resources and poor school infrastructure. There are 202 educational communities that have participated in CARS to date, with some receiving both formal and nonformal reading programs concurrently. The majority of these communities (177 of the 202 participating communities) have received at least one EpC, 89 communities have received CARS assistance on 1st-grade readiness, and 38 communities have received assistance for teaching reading in grades 1 through 3. The majority (61 percent) of these educational communities were located in Bluefields, followed by around 15 percent in Kukra Hill and another 15 percent in Laguna de Perlas. Among all participating educational communities, 25 percent of communities with EpCs had either 1st through 3rd grade or transition to 1st grade activities in the sponsoring school.17 Among the sample of 36 educational communities, public and community schools generally have only the EpC component, whereas private schools generally have one or both primary or preschool CARS components. Data collectors interviewed principals and teachers at 36 schools that participated in CARS: 18 public and community schools and 18 private schools. Sixteen of the 18 public and community schools in the sample had an EpC (Table III.2), but only slightly more than half of the private schools (10 of 18) had an EpC. Whereas no public schools participated in the primary or preschool components, all private schools participated in at least one of the two. Specifically, eight private schools received both preschool and primary school formal reading programs, six received the primary school reading program only, and four received the preschool reading program only. This reflects the characteristics of the full population of educational communities participating in CARS. As noted in Chapter II, EpCs were largely envisioned as supports for students attending public schools (due to restrictions in providing public schools with USAID funds), while preschool and primary school CARS components were envisioned for private schools (which faced no restriction on the receipt of donor funds). Schools in the sample have fewer than 200 students on average, and most teach exclusively in Spanish. Most of the 36 schools serve kindergarten to 6th grade; about half are multigrade. Eighty-six percent of schools in the sample use Spanish as the only language of instruction (Table III.2). This is generally consistent with the full population of schools participating in CARS, of which around 95 percent teach primarily in Spanish. Interestingly, even though Spanish is the primary language of instruction for the vast majority of schools participating in CARS, Spanish is the mother tongue of only 60 percent of children assigned to EpCs (see Appendix A for more details). 17 Conversely, 58 percent of educational communities with 1st through 3rd grade or transition to 1st grade activities also had EpCs. 26 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH Table III.2. Characteristics of schools visited during data collection Public and community schools Private schools All schools Offer preschool (%)b 78% 50% 64% Multigrade (%)b 59% 41% 50% Spanish is the only language of instruction (%)b 78% 94% 86% Have an EpC (%)a 89% 56% 72% Have CARS grade 1–3 component (%)a 6% 78% 42% Have CARS 1st-grade readiness component 22% 67% 44% Any CARS component (%)a 100% 100% 100% Average school sizeb 163 210 187 Note: Sample size is 36 schools visited during data collection: 18 public and 18 private schools. aSource is CARS report. bSource is school director interview. Schools in educational communities served by CARS—particularly public schools— often lack basic resources. In focus groups, community leaders, teachers, and principals reported that schools didn’t have enough desks or chairs for the students; that they lacked kitchens, latrines, and handwashing stations; and that they had leaking roofs. They said that some school buildings were falling down and needed to be rebuilt. Public schools in particular appear to be in need of new or improved infrastructure, including electricity and classrooms. Related to this issue, fewer than 60 percent of teachers and principals agreed that they received adequate support from the broader community—particularly the mayor’s office—to keep their schools fully functional and adequately equipped. This elevates the importance of the community engagement elements of CARS, such as the CAPs, that can focus on improving the school infrastructure. C. What are the characteristics of households and children participating in CARS? Households in the sample generally came from poor agriculture and farming communities, where parents tended to have low levels of education themselves. As a result, the children often started school with low reading levels and little exposure to reading materials, which puts them at risk of falling behind and dropping out. Households are generally large, low income, and poorly educated. A baseline survey of the households of Cohort 2 EpC students found that the households tended to be large—with nearly seven people per household, on average—and that monthly household income was low— under 4,000 córdobas on average (equivalent to approximately $137 U.S. dollars). According to the latest available census, the average national wage was C$5339.5 or approximately US$275 in 2008 (INIDE, 2008). Heads of household and caregivers also have low educational attainment, on average. According to the baseline survey, heads of household tended to be around 42 years old, with only around half being literate—literacy rates vary from 56 percent in Kukra Hill to 43 percent in Laguna de Perlas (compared to 78 percent nationally, UNICEF 2013). Furthermore, fewer than one in 10 of the children’s primary caregivers were literate. This has serious 27 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH implications for the parents’ ability to practice reading at home with their children, as envisioned by CARS. Most parents who participated in the focus groups worked in the agriculture or fishing industries, or they were homemakers. The majority of men who participated in the parent focus groups worked in agriculture or fishing. The high participation of men in fishing in particular may necessitate family migration throughout the year. In a baseline survey of households with children assigned to Cohort 2 EpCs (Bagby et al. 2017), nearly a quarter of households reported migrating at least once in the past year. The majority of women in focus groups reported being homemakers; a small number of women identified as teachers. In the focus groups with community leaders, men and women reported playing a diversity of roles in their communities. Men’s roles were often related to religious institutions (for example, church pastor, president of church board), whereas women’s roles were primarily related to educational institutions (for example, school board member). The EpCs appear to be serving the target population of kids who either are not in school or are at risk of dropping out. The expected profile for EpC students is 5 years old to 15 years old, enrolled in grades 1 to 4, and either experiencing learning difficulties or at risk of dropping out of school. Intake data collected before students were assigned to EpCs corroborate that all students assigned to Cohort 1 and 2 EpCs were between the ages of 5 and 15 (with an average age of between 8 and 9). Intake data also show that 14 percent of students assigned to EpCs were not enrolled in school prior to being assigned to EpCs. However, this varies from 30 percent of kids in Kukra Hill to only 2 percent of kids in Corn Island (see Table III.3). Children assigned to attend EpCs also had learning difficulties and were at high risk of dropping out of school. Nearly half of enrolled students were overage for their grade, over half were frequently absent, and more than four in five were low performers in school at intake (Table III.3). However, it is unclear to what extent EpCs are covering the full demand for EpCs in the region. In other words, there are no reliable data on the percentage of all at-risk students in the region who are currently being served by EpCs. Reading deficiencies appear to be more pronounced among students who attend EpCs compared to the larger population of students. Schoolteachers at private and public schools said most of their students can read fluently by 3rd grade but that they often lack comprehension and analysis skills. Teachers noted that children in 1st grade can read syllables and words, while 2nd- and 3rd-grade students can often read fluently. However, when teachers ask these students about a passage the students just read, the children have difficulty summarizing or interpreting it. EpC facilitators noted even more serious deficiencies among the children who attended the EpCs. One facilitator noted that children enrolled in EpCs have difficulty sounding out basic words in 2nd and 3rd grade. In particular, EpC facilitators noted that students who have a different mother tongue than the dominant language at school are the least likely to read at grade level. According to one facilitator, kids who speak Spanish as a second language have progressed as far as 3rd grade without being able to read well in Spanish. Child assessment data collected for the EpC impact evaluation corroborate that over 20 percent of 3rd-grade students assigned to Cohort 1 EpCs have poor reading comprehension, even after the conclusion of EpC activities. (See Appendix A for more detail on students’ reading practices, fluency, and comprehension following the close of Cohort 1 EpCs.) 28 III. DESCRIPTION OF CARS STAKEHOLDERS MATHEMATICA POLICY RESEARCH Table III.3. Characteristics of children assigned to attend EpCs (percentages unless otherwise indicated) . Bluefields Corn Island Desembocadura del Rio Grande Laguna de Perlas Kukra Hill Overall average Average age (years) 8.6 8.2 8.5 9.0 8.8 8.7 Female 46 53 44 47 47 47 Cohort 1 39 100 100 48 43 47 Cohort 2 62 0 0 52 57 53 Enrolled in school 85 98 93 86 70 83 Among those enrolled in school: . . . . . . Preschool 0 0 0 3 1 1 Grade 1 45 38 43 53 40 46 Grade 2 29 39 30 24 33 29 Grade 3 26 24 27 20 26 25 Among those enrolled in school: . . . . . . Overage for grade 44 33 40 56 53 47 Frequently absent 44 66 53 47 56 48 Low performance 87 93 88 75 71 83 Source: Impact study of Espacios para Crecer—2015 intake data for 4,596 children assigned to attend Cohort 1 and 2 EpCs. 29 This page has been left blank for double-sided copying. IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH IV. IMPLEMENTATION FINDINGS In this chapter, we distill information from reports, administrative data, interviews, and focus groups with stakeholders into a summary and analysis of CARS implementation. First, we summarize implementation of each programmatic component. Then, we discuss progress toward implementation goals. Last, we present an analysis of facilitators and barriers to successful CARS implementation. A. How is CARS being implemented? CARS is being implemented largely according to its initial design and timeline, with some delays. CARS implementation has followed its train-the-trainer design, in which NGO staff are trained and supported by DevTech staff as they implement reading programs and parent schools. EpC implementation kicked off with 10 pilot EpCs in 2014 and expanded dramatically in 2015 and 2016 to reach 270 EpCs across Cohorts 1, 2, and 3 in 177 communities. Primary and preschool reading programs were first implemented in 2015—extending to around 40 primary schools and 40 preschools by the end of the year. BICU held the first training workshop for the diploma program in mid-2016 (Figure IV.1). This workshop was somewhat delayed due to the need to subcontract the development of teaching materials and the process of revising these materials. CARS distributed thousands of educational titles to EpCs and participating schools from 2014 to 2016, but experienced some delays in contextualizing materials and distributing them; these delays were linked to several factors, including lengthy approval processes for newly contextualized materials. For example, EpC books were distributed to EpCs in May 2015 – 12 and 7 months after the start of Cohorts 1A and 1B, respectively (CARS, 2016a). NGOs began implementing parent schools in 2015 in EpCs. They expanded these parent schools to complement formal reading programs in private schools shortly thereafter. Although very few CAPs were developed and approved in 2015, the NGOs and DevTech found ways of simplifying the CAP development and approval process, eventually producing more and higher quality plans in 2016. Throughout implementation, DevTech staff have worked closely with all six NGOs to strengthen their administrative, financial, organizational, management, and technical areas. 31 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Figure IV.1. Global CARS implementation timeline 1. Formal reading programs DevTech contracted and trained the NGOs in late 2015 for the formal reading programs. In April 2015, DevTech signed grant agreements with three local NGOs—FQSF, URACCAN, and AMC—to implement the CARS formal reading program in private and subsidized primary schools and preschools. In April and May of 2015, the three NGOs received (1) financial and administrative training, (2) training on APA methods, (3) training on the use and purpose of a mini EGRA instrument, and (4) training on running parent schools.18 By late 2015, FQSF, URACCAN, and AMC were active in around 40 primary schools and 40 preschools, providing teachers with instruction in APA methodology, reading assessments, and parent schools. CARS developed and distributed a large volume of materials to private and subsidized schools. In 2015, DevTech received large shipments of books that were inspected, inventoried, and delivered to the respective NGOs, which delivered them to schools participating in formal reading programs in preschools and primary schools. From April to June 2015 alone, this included 17,760 books (14,520 for primary schools and 3,240 for preschools) and large quantities of classroom materials, which were sent to the three implementing NGOs. In early 2016, school students participating in CARS formal reading programs received Nacho Nicaragüense workbooks and storybooks. These books belong to the students, and students are encouraged to use them at school as well as at home with their parents. In May 2014, DevTech began developing 14 teacher guides for the formal reading program in Spanish. It planned to later adapt and translate the guides to English and Miskitu. The development and approval process lasted over two years, given the complexities of translating and adapting content to the Nicaraguan context, obtaining necessary approvals, and printing and disseminating materials. 18 The mini EGRA is a pared-down version of the full EGRA designed to be easily administered and interpreted by teachers. 32 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH BICU held the first diploma program training in mid-2016. In 2015, BICU was selected as the sole implementer of the diploma training program for 1st- through 3rd-grade teachers based on a competitive procurement process with other local institutions. The diploma program is offered as part of the CARS formal reading program. In June 2016, BICU conducted the first of three hands-on training sessions to strengthen the skills and aptitudes of 177 preschool and primary school teachers and educators working in the area of early grade reading and writing. Attendees included teachers from preschool and primary schools that were predominantly privately managed, more than 20 Ministry of Education (MINED) teachers and pedagogical advisors, and NGO staff who worked closely with teachers in the field. Training for primary school teachers emphasized APA methods, but also included elements of QL and FAS. In contrast, training for preschool teachers focused primarily on QL—similar to that of EpC facilitators. A snapshot of educator training for CARS reading programs corroborates the focus on QL for facilitators and APA for teachers. From January to September 2016, DevTech and NGO staff trained 630 unique teachers, directors, and EpC facilitators in APA, QL, the mini EGRA, and parent schools (Figure IV.2). Reflecting the reading programs’ focus on APA in primary schools and QL in EpCs, primary school teachers were primarily trained in APA methodology and community organizing, including parent schools, whereas EpC facilitators were primarily trained in QL, reading assessments, and parent schools. In contrast, principals were most commonly trained in the use of teacher and student workbooks for private preschools and primary schools, while MINED/SEAR staff were most commonly trained in APA and parent schools. “When the teachers graduate from school, they aren’t trained in early reading. Many of their teaching methods don’t have a theoretical basis. With the diploma program, they’ve been able to understand the basic concepts of reading and writing.” – DevTech representative 33 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Figure IV.2. CARS training participants, January–September 2016 EpC teacher￾facilitators, 1% MINED representatives, 3% Others, 1% EpC Facilitators, 42% Teachers, 38% CARS staff, 10% Principals, 5% Source: DevTech M&E data, 2016. N = 630 individuals trained by CARS from January to September 2016. Note: January-September 2016 is the only period for which detailed training data were available. 2. EpCs EpC implementation started in early 2014 with 10 pilot communities. In mid-2014, DevTech started EpCs in 10 pilot communities with public schools in Bluefields, Corn Island, and Kukra Hill. (These pilot communities comprised Cohort 1A; see Figure IV.2). Prior to establishing the EpCs, DevTech gathered data from MINED and communities; compiled student lists and school data; ordered materials; planned community assemblies; obtained communities’ commitment; and conducted a weeklong facilitator training session for 32 EpC facilitators, principals, and local education leaders. DevTech also trained facilitators participating in the pilot phase in the use of instruments to measure student progress, and then conducted follow-up visits with facilitators after training was complete. Once EpCs were established, DevTech staff also worked with EpC facilitators to initiate parent schools at each of the 10 pilot EpCs. DevTech transferred direct EpC implementation to NGOs in late 2014. In November 2014, DevTech transferred the 10 pilot EpCs to three local NGOs: (1) URACCAN, (2) FZT, and (3) FHR. At the end of 2014, DevTech assumed an advisory role as each of these three NGOs expanded the number of EpCs in their respective municipalities, for a total of 63 new EpCs (Cohort 1B) in addition to the 10 pilot EpCs (in Cohort 1A) (Figure IV.3). These new EpCs were located near public schools, with the exception of 15 EpCs located in private or subsidized 34 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH schools.19 With DevTech’s assistance, NGOs gained community buy-in for EpCs, contracted and trained EpC facilitators, provided facilitators with adequate in-class follow-up, and helped facilitators start parent schools. Figure IV.3. EpCs by cohort and municipality 5 28 43 14 63 2 4 5 8 9 3 9 16 16 14 12 4 15 10 63 71 18 108 0 20 40 60 80 100 120 1A 1B 2A 2B 3 Number of EpCs Cohort Bluefields Corn Island Pearl Lagoon Kukra Hill Desembocadura Source: DevTech M&E data, 2016. CARS established an additional 197 EpCs in 2015 and 2016, mostly near public schools. These EpCs comprised Cohorts 2A, 2B, and 3 (Figure IV.2). URACCAN, FZT, and FHR administered these EpCs, with URACCAN and FZT establishing over 100 EpCs apiece and FHR establishing 41 EpCs during 2015 and 2016, primarily in public schools. Overall, NGOs established EpCs according to the initial CARS timeline, meeting FY 2015 and FY 2016 targets for number of EpCs established. EpCs were generally concentrated in Bluefields, particularly in the second and third cohorts (Figure IV.3). On average, EpCs were implemented under a 20- month schedule, which spanned two school years (Figure IV.4). However, cohort 1B EpCs had a nearly two-year schedule, spanning from late 2014 to late 2016. 19 This was not planned in the original CARS design, which envisioned EpCs primarily in public schools. 35 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Figure IV.4. Timeline of EpC implementation Cohort Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun 1A X X X X X X X 1B X X X X X X X X X 2A X X X X X X X X 2B X X X X X X X X 3 X X X X X X 2014 2015 2016 2017 Source: DevTech M&E data, 2016. Note: Current as of late 2016. The 2017 activities are shaded because they had not been implemented by the time of the analysis. EpCs received books, guides, and other teaching materials from 2014 to 2016, including some materials developed specifically for the RACCS. In 2015 and 2016, CARS received large shipments of books for EpCs that were inspected, inventoried, and delivered to the respective NGOs, which delivered the books to the EpCs. For example, 7,984 books in Spanish and English were delivered to EpCs in March 2016 alone (CARS 2016b). DevTech also contracted with a consulting firm to contextualize an EpC facilitator’s guide and student modules and workbooks for the RACCS. In all, the firm adapted five student workbooks for students in 1st and 2nd grades and five student workbooks for students in 3rd grade. The full process of contextualization took around a year; some guides and workbooks were still being translated into English and Miskitu as of mid-2016—several months after the EpCs were established in English￾and Miskitu-speaking communities. 3. Community engagement efforts NGOs began implementing parent schools in 2015 as a complement to the EpCs. Parent schools were implemented a few months after Cohort 1 and 2 EpCs were established, under the premise that CARS community engagement efforts could begin only after EpCs were established in participating communities. In early 2015, DevTech staff developed didactic materials for parent schools and trained NGO staff on these materials. NGO staff then trained EpC facilitators to administer parent schools. Facilitators began holding parent schools in early 2015 with guidance from DevTech and NGOs. In late 2015, NGOs began helping private school teachers implement parent schools as well, using the same didactic materials developed for EpCs the prior year. In 2015 and early 2016, parent school attendance was generally poor across municipalities. Among those parents who attended parent schools, nine out of 10 attendees were mothers. However, father attendance at parent schools increased in 2016, to the extent that one out of every three parent school attendees were fathers in the second quarter of 2016 (CARS 2016b). 36 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH CAPs largely stalled in 2015. In all communities in which EpCs were established and where NGOs worked in preschools and primary schools, the NGOs were expected to facilitate the development of school diagnostics, improvement plans, community intervention plans, and CAPs. (School diagnostics, improvement plans, and community intervention plans are critical inputs to the CAPs, and must be completed before the CAPs are submitted to DevTech). By mid￾2015, CARS had only submitted three CAPs to USAID. The small number of CAPs developed in 2015 was due to several reasons. First, DevTech and the NGOs found that the initial months of implementation were consumed by simply establishing the EpCs and formal reading programs in each community. Community engagement activities could not commence until some level of CARS presence had been established. Second, the NGOs noted that during the first six months of CARS implementation, they were unclear as to the basic objectives of the CAPs, the basic steps to develop them with the community, and the core criteria on which the CAPs would be assessed for approval. In particular, stakeholders noted that initially some plans requested funds to improve infrastructure at public schools; this was not allowed under the CAPs, as CARS could not support improvements at public schools. Third, several CAPs had to be modified after they were submitted to DevTech because they had critical weaknesses, including incomplete documentation, poor presentation of information, and internal contradictions. Over time, NGOs and DevTech found ways of simplifying the CAP development and approval process, eventually producing more and higher quality plans, according to interviewees. Faced with multiple steps to develop CAPs, the NGOs collaborated to simplify the CAP development process with a template that outlined the requirements of each step. One NGO worked collaboratively with other NGOs to develop a template to identify problems in stakeholder consultations, identify the action that must be taken, and justify the investment. To improve the CAP approval process, DevTech staff also designed a checklist that helped quickly and simply evaluate CAPs within USAID environmental guidelines as well as CARS indicators. The use of this checklist accelerated the review of CAPs submitted by the NGOs. According to CARS quarterly reports, CAP quality improved during 2016, such that during the third quarter of 2016, not a single CAP (of those submitted) was rejected by DevTech. This was a large improvement from the previous quarter, in which 27 of 35 CAPS were rejected by DevTech due to incompleteness, ineligibility, or quality concerns. Examples of CAPs submitted in 2016 included proposals to purchase classroom chairs, construct an early reading area at a school, and buy cabinets in which to house reading materials. CARS outreach activities included murals and reading events. During 2015, the CARS team helped paint several CARS murals in Bluefields, Kukra, and Pearl Lagoon. The team also distributed CARS shirts and baseball caps. In June 2015, CARS participants and staff celebrated Universal Children’s Day in Bluefields, Pearl Lagoon, Kukra Hill, and Corn Island by using art, The CAP development and approval process Community leaders and parent schools (organized around EpCs or private and subsidized schools) create an analysis commission, which interprets evidence (including, school report cards, EGRA, grades, school attendance and participation, household surveys, and observations from visits) to identify areas they’d like to improve. Next, they complete an initial assessment, which outlines the current situation or problem, the proposed solution, and how that solution will address the problem. The stakeholders then document this information in a CAP and present it to DevTech staff, who review it, request changes as necessary, and present a revised plan to USAID. 37 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH games, books, and music to highlight children’s rights and responsibilities. Over 2,000 children, parents, and teachers attended these events (CARS 2016c). Also in 2015, CARS and local nonprofits collaborated to hold a Children’s Day celebration on Corn Island, which was also supported by local authorities and MINED. All these community leveraging achievements are significant and important, considering the level of poverty of these areas. Communications products were still in development in 2016. As of late 2016, DevTech’s contractor Argomedia was still finalizing a communications plan for CARS, as well as various communication and visibility products. CARS anticipated developing several educational and communication products for TV, radio, video, Internet, and print media for 2017 (CARS 2016c). 4. Local capacity development Baseline OCA analyses revealed deficiencies in human resources and financial management for CARS local NGO partners. DevTech staff applied the OCA to all six implementing NGOs when they joined CARS in 2015 and 2016. Across NGOs, the areas that showed the greatest need for strengthening and improvement were human resources (particularly, recruitment of new personnel, salary and benefits, and performance evaluation) and financial management (including cost sharing and financial sustainability). Overall, participating NGOs also had deficiencies in control of fixed assets (in the administration domain), resource engagement (in the organizational management domain), and supervision (in the project performance management domain). These baseline OCA scores formed the basis for NGOs’ action plans, which guided institutional strengthening efforts. CARS conducted capacity building with NGOs throughout 2015 and 2016. DevTech staff worked very closely with all six NGOs to strengthen their administrative, financial, organizational, management, and technical areas. Strengthening activities included individual and collective trainings at DevTech offices. For example, DevTech provided general human resources, financial management, and administrative trainings for all NGOs, but contracted consultants to offer each NGO tailored training in specific areas. DevTech also conducted in-person visits, communicated via phone and email, and provided feedback in trip reports and during community visits. During the face-to-face monthly review meetings, DevTech and NGO representatives discussed challenges and potential solutions to issues related to financial management, budgeting, and planning, as well as technical planning and execution. CARS also worked with the six NGOs on monthly, quarterly, and annual work plans and budgets. As of late 2016, DevTech was still working through capacity development challenges with two NGOs. Notably, URACCAN had difficulties implementing its monthly plans for EpCs and meeting its project goals. In addition, DevTech noted that actual time spent by URACCAN “[The OCA] is about looking at organizational strengths and weaknesses on [a] macro level with the idea that we’re not only trying to strengthen the personnel, but also the organization itself, so it’s more sustainable when CARS leaves.” — DevTech representative “We put all of their [OCA] scores together and looked at the subcomponents across the board, and there’s maybe five that are consistently weak in all organizations. We’re dedicating consultants to target those weaknesses and creating terms of reference to address that subcomponent directly.” — DevTech representative 38 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH field staff in the community and EpC itself appeared to be less than that spent by other NGO field staff. DevTech discussed these concerns with URACCAN staff, who committed to enhancing their community presence. AMC also had deficiencies in its administrative and financial processes. As a result, CARS grant funding was withheld from AMC in early 2016, until it implemented a plan to remedy the defects. DevTech staff temporarily covered AMC’s geographical area and schools in order to maintain the provision of CARS assistance to teachers and materials to these schools. By the end of April, AMC had conducted some internal analysis and strengthening actions, and DevTech approved its action plan. In May, AMC once again began receiving grant funds. DevTech continued to work with AMC to remedy noted administrative, financial, and technical planning processes. NGOs collaborated in periodic encounters, but sometimes NGOs failed to work together in the field. Throughout the implementation period, NGO staff held several “encounters” or workshops to exchange early grade reading experiences and discuss best practices with respect to reading programs and community engagement efforts. NGO staff praised these encounters as an opportunity to learn from one another and consider improvements to their CARS activities based on other NGO experiences. However, according to DevTech reports, in communities that had EpCs as well as formal CARS reading programs implemented by different NGOs, it was unfortunately not possible to find synergies between the work of the two NGOs, either by holding joint parent school and CAP sessions, or coordinating efforts to avoid scheduling conflicts between different implementing partners. 5. Knowledge generation DevTech shared key findings and learnings with USAID throughout CARS implementation. DevTech was in continual contact with USAID during implementation via telephone, email, and in-person events. Through in-person events and formal reports, DevTech shared important child outcome findings and learning products with USAID, including a baseline report of EGRA findings in 2014 as well as subsequent analyses of mini EGRA data throughout 2015 and 2016. These analyses suggested potential positive effects of CARS reading programs' on children's reading fluency. DevTech also shared gender and security plans with USAID, both of which featured new research and suggestions for CARS implementation. The CARS team also shared findings with a variety of external stakeholders. In early 2014, DevTech held technical meetings with municipal education authorities to share summary information about EpC activities. Once CARS was under way in 2015, NGO staff also met every quarter with local authorities and business leaders to keep them apprised of CARS activities. In 2015, CARS staff participated in a knowledge sharing event, in which they discussed EGRA findings, a citizen security study, and school assessments with local and regional stakeholders. CARS also invited outside parties to the encounters, including local NGOs, foundations, and MINED to exchange experiences in early grade reading and community engagement. CARS also included MINED and other stakeholders in trainings—notably, the diploma training in 2016. B. Is CARS being implemented as planned? CARS met its ambitious target of establishing 270 EpCs in five municipalities. Although the EpCs served fewer children than planned, the formal primary school reading programs served more children than planned. By 2017, CARS is projected to surpass its initial goal of 12,500 39 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH students served by reading programs. CARS met targets for materials distributed in 2015, but fell short of targets in 2016 due in part to delays in developing new materials. Although EpC facilitator and teacher training occurred largely as expected, teacher and facilitator follow-up and coaching was much less frequent than planned, according to interviewed stakeholders. Parent schools also failed to meet attendance targets. Furthermore, CARS developed and executed far fewer CAPs than initially expected. 1. Formal reading programs and EpCs Formal reading programs and EpCs are being implemented largely according to their initial design and timeline. CARS implementation has followed its train-the-trainer design, in which NGO staff are initially trained by DevTech staff in active teaching methods (in a training called ENTRENA) and then supported by DevTech as they train and support preschool and primary school teachers and newly contracted EpC facilitators to employ these teaching methods in the classroom. EpC implementation kicked off with 10 pilot EpCs in 2014 and expanded dramatically in 2015 and 2016, as initially planned. Formal primary and preschool reading programs were first implemented in 2015, largely according to schedule. By late 2016, CARS had met its goal of 270 EpCs, but fell short of its goal of reaching 8,150 kids in EpCs. CARS planned to open 270 EpCs from 2014 to 2016. With 270 EpCs implemented by late 2016, the project had met this goal (Table IV.1). This is an impressive achievement, as it required a dramatic scale-up of EpCs in 2015 and 2016. Two NGOs established and maintained over 100 EpCs each and another, smaller NGO established and maintained around 40 EpCs. However, the total number of students served by EpCs from 2014 to 2016—7,331—fell short of the initial goal of 8,150. DevTech staff noted that the program failed to serve the desired number of students—despite opening the desired number of EpCs—due to the relatively low average number of students enrolled at EpCs in rural areas (Table V.1). DevTech staff noted that the program’s expansion into rural areas was necessary to conduct a rigorous impact evaluation, but that it was fundamentally more difficult to enroll students in rural areas to attend EpCs due to small student populations, transitory populations, and relatively long commutes in these areas. Initial assumptions were that each EpC would serve 30 students on average. In practice, CARS calculated that the EpCs served around 27 students on average. (Site visits to 26 EpCs in late 2016 revealed even lower initial enrollment numbers of around 22 students on average). CARS trained over 1,100 school staff in 70 preschools and 40 private and subsidized primary schools, but this was fewer staff and schools than originally anticipated. By mid￾2016, the program had trained over 200 fewer private school teachers than originally planned. In addition, CARS had only worked with 41 preschools (compared with the initial goal of 70 preschools) (Table IV.1). NGO staff noted that the lower number of preschools served (and teachers trained) reflected the fact that stakeholders had initially overestimated the number of private preschools in the region in which CARS could feasibly work. In addition, CARS had not established any of the 10 new preschools that were originally envisioned under the project design, citing the fact that once the intervention was under way establishing new preschools was reprioritized compared to implementing reading programs in existing preschools. CARS met its goal for students served at private and subsidized schools. Despite serving fewer preschools than planned, by late 2016, CARS had met its target of preschool 40 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH students served, and had plans to reach another 1,500 preschool students in 2017 (CARS 2016d). According to DevTech staff, they were able to serve the target number of preschool students— despite serving fewer preschools than planned—because preschool enrollment in the 41 participating schools was higher than expected. Similarly, CARS exceeded its target of serving 2,440 private school students in grades 1 to 3 (Table IV.1). DevTech staff attributed the program’s ability to meet grade 1 to 3 targets to relatively accurate enrollment data available for private and subsidized schools, which it used to construct initial goals. CARS is projected to surpass its initial goal for total students served by reading programs. By late 2016, the total number of CARS student participants was 11,759 across all reading programs—short of the initial goal of 12,500 students (Table IV.1). However, the program is projected to serve a large number of students in preschool and grades 1 to 3 in 2017, arriving at a total of 12,900 participating students by the end of CARS implementation in the RACCS. Table IV.1. CARS reading program implementation progress, as of late 2016 . . 2014 2015 2016 Total Goal Goal met? EpCs EpCs established 73 85 112 270 270 Yes (100% execution) Students served by EpCs 290 4,250 2,791 7,331 8,150 No (90% execution) Preschools Number of preschools established 0 0 0 0 10 No (0% execution) Number of private and subsidized preschools served 0 41 41 41 70 No (59% execution) Participating preschool students 0 978 911 1,889 1,910 Nearly (99% execution) Private and subsidized primary schools Number of primary schools served 0 39 39 39 40 Nearly (98% execution) Private school personnel trained 131 235 527 893 1,134 No (79% execution) Participating private school students in grades 1–3 0 2,175 2,539 2,539 2,440 Yes (104% execution) Students served 290 7,403 11,759 11,759 12,500 No (94% execution) Source: CARS indicators tracking sheet, December 2016. 41 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH 2. CARS materials CARS met targets for materials distributed in 2015, but fell short of targets in 2016 due to delays in developing new materials. In 2015, CARS nearly met its goal of distributing around 27,000 textbooks and learning materials to EpCs and participating schools. However, CARS fell short of a similar target in 2016, likely reflecting delays experienced in developing and contextualizing materials for formal and nonformal reading programs. In an interview, an NGO representative noted that adapting new materials to the local context was a slow process that required review by multiple parties, which ultimately led to delays in disseminating materials. Reflecting on the major delays in finalizing materials for the program, a USAID representative suggested that perhaps CARS had overinvested in developing new materials for the program, particularly when existing materials might have been sufficient in some cases. Most educators reported receiving ample materials, but some reported delays and language mismatches. In-person interviews with over 100 teachers, principals, and facilitators in 36 educational communities participating in CARS corroborate that CARS succeeded in disseminating a large volume of teaching materials to EpCs and schools (Table IV.2). In particular, facilitators primarily from Cohort 1 and 2 EpCs reported receiving 38 books for EpCs, on average. This surpassed CARS’ internal goal of at least 25 titles provided to each EpC. (However, there was some variation in the number of titles reported. Some facilitators reported receiving as few as 10 books from CARS.) Similarly, preschool and 1st- through 3rd-grade teachers reported receiving 49 books apiece. This was around the same magnitude of CARS’ internal goal of 50 titles in each classroom. Although educators expressed strong appreciation for all CARS materials, several EpC facilitators and teachers noted that they received some workbooks and other materials several weeks after EpC sessions and CARS activities had started. At least some of the 80 interviewed educators also reported a mismatch between some CARS-distributed materials and students’ mother tongue. For example, one teacher received only books in English and Miskitu; her primarily Spanish-speaking students didn’t understand them and just looked at the pictures. Table IV.2. Planned and actual materials distribution Indicator Year Goal Actual Goal met? Total number of textbooks and teaching and learning materials provided to assisted schools and EpCs 2015 27,080 26,908 Nearly (99% execution) 2016 26,618 18,560 No (70% execution) Source: CARS indicator tracking sheet, December 2016. 3. Educator coaching and follow-up Nearly all EpC facilitators reported CARS follow-up visits and in-class observations, but fewer preschool and primary school teachers reported such visits and observations. A fundamental part of the CARS model is regular coaching visits from NGOs to guide educators in their transition to new teaching methodologies, including assistance with lesson planning and classroom setup. NGOs are also expected to conduct in-class observations to verify that teachers 42 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH and EpC facilitators are teaching with an explicit focus on reading fundamentals, phonetics, vocabulary, and reading comprehension. During community visits, nearly all active facilitators reported receiving a CARS visit in the past year and 89 percent of active facilitators reported in￾class observation from CARS during visits. However, fewer teachers reported such visits: around three-fourths of CARS-trained teachers reported receiving a CARS visit in the past year, while only 61 percent of teachers reported an in-class observation during a CARS visit (Table IV.3). Teachers, principals, and facilitators that received CARS visits widely viewed them as helpful. Educators expressed a general sentiment that they felt encouraged and motivated by NGO support and that this support seemed to fill a large need that had gone unmet in previous years (see Appendix B for more detail regarding educators’ perspectives on CARS coaching and follow-up visits). EpC facilitator and teacher visits were also less frequent than originally planned. CARS set the goal of one visit per month in difficult-to-reach rural settings and two visits per month in more accessible communities. However, according to interviewed teachers and facilitators, visits from NGOs or DevTech were far less frequent than once per month. On average, educators reported between three and four visits from NGOs or DevTech in the past year (Table IV.3). In interviews, teachers noted that they wanted more in-class observation and technical assistance to learn how they could improve their application of the APA educational approach. In interviews, DevTech, NGOs, and USAID representatives agreed that the program had not provided adequate follow-up. One DevTech representative noted that as a result of infrequent coaching, some EpC facilitators were still not comfortable with QL methodology. In interviews, NGO staff acknowledged that follow-up had been less frequent than planned. They cited limited capacity, multiple CARS responsibilities, and aggressive implementation targets as the primary reasons that they were unable to have a stronger presence at schools and EpCs. “This is the most challenging part of implementation in the sense that now there needs to be more time devoted to classroom follow-up, an ongoing assessment of what’s going on. . . . If I’m not applying what I’ve learned, then it’s basically the same thing. It’s important that there’s this permanent, ongoing presence in the classroom, to be an extra aid to teachers in the classroom in the process of teaching students to read well and build a base of comprehension.” — USAID representative 43 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Table IV.3. Planned and actual coaching visits by NGO staff . Internal CARS goal Reported by active facilitators Reported by trained teachers Goal met? Reported at least one CARS visit in the past year 100 percent 100 percent 76 percent Yes for facilitators, no for teachers Reported at least one classroom observation in the past year 100 percent 89 percent 61 percent No Average number of CARS visits in the past year At least 8 visits by October 2016 (1 visit per month)a 3.8 4.0 No Source: In-person interviews with principals, teachers, and facilitators in September and October 2016. Note: Sample includes all facilitators and all teachers who reported participating in at least one CARS training. Facilitators are limited to those who were active in CARS EpCs in 2016. a Data were collected largely in September and October, during the eighth and ninth months of instruction during the 2016 school year. If teachers received one visit per month, they would have received at least eight visits at the time of data collection. 4. Parent schools There was variation by implementing NGO in parent school attendance from late 2014 to 2016. Throughout CARS implementation, NGOs were expected to facilitate at least one parent school meeting per month in the schools or EpCs in which they were working, which would be attended by at least half of the parents. In general, parent school attendance did not meet initial targets in terms of number of parent school attendees (Table IV.4). However, there was significant variation in attendance by NGO. Notably, parent school attendance was lower at private schools than in EpCs (CARS 2016c). According to DevTech, some reasons for poor attendance included incomplete NGO implementation (largely due to time constraints and prioritization of other tasks, difficult access to communities, and vacations and holidays), competing priorities, as well as poor parent motivation or incentives to attend. DevTech reported that the NGOs’ lack of availability to run parent schools, at least in part, were the result of the NGOs’ efforts to identify eligible communities for the EpC impact evaluation (CARS 2015). However, some NGOs performed better than others in organizing parent schools and generating parent attendance. For example, whereas AMC, FHR, and FZT surpassed their goals for parent school attendance at EpCs for 2016, URACCAN and FQSF did not meet their targets. (CARS 2016e). Overall, parent school attendance increased steadily during 2016, as NGOs established a stronger presence in CARS communities. Parents who attended parent schools reported positive experiences. In focus groups, parents who attended CARS parent schools expressed appreciation for the schools' fruitful discussions and engaging activities related to their children’s development. Parents liked being able to pick the discussion topics themselves, and several parents mentioned that discussions on managing their children's behavior were particularly interesting or useful. (See Appendix B for more detail regarding parents’ perspectives on CARS parent schools.) 44 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Table IV.4. Planned and actual parent school attendance . Goal (2016) Actual (2016) Goal met? Number of parent school attendees 6,250 4,471 No (71% execution) Source: CARS 2016 Annual report. Note: These attendance goals assume one parent for every two enrolled children (in EpCs or participating schools) will attend parent schools on a monthly basis. Goals for parent attendance were directly based on goals for EpC enrollment and school enrollment. Around half of the schools visited during data collection reported having CARS￾affiliated parent schools. Principals from 15 of the 36 schools visited during data collection reported that their school had a CARS-affiliated parent school that met regularly—either associated with the EpC or the school itself. This is consistent with low parent school attendance reported in CARS quarterly reports. Interestingly, there were also cases of multiple parent schools at the same school—for example, it was not rare for a school to have one CARS￾affiliated parent school for the EpC as well as a MINED-affiliated parent school or a parent school run by the principal. According to DevTech, poor coordination between the NGOs also generated cases of two active CARS-affiliated parent schools operating at the same time within the same school: one run by an EpC facilitator and another run under the guidance of an NGO (presumably implementing the CARS formal reading program in 1st through 3rd grades). Multiple parent schools run the risk of creating redundancies in attendees, discussion topics, and activities, particularly in the case of more than one CARS-affiliated parent school in the same educational community. 5. CAPs and community contributions Only 5 CAPs were executed by late 2016, far below the initial goal of 90 CAPs. Due to initial confusion regarding CAP eligibility criteria and development steps, weaknesses in the first CAPs that had to be corrected, and the prolonged CAP development and approval process, only 51 CAPs had been submitted for approval and only 5 CAPs had been approved and executed by late 2016. This was well below the goal of 90 CAPs by late 2016 (Table IV.5). NGOs were somewhat uneven in their completion of CAPS, with slightly higher submission rates among NGOs working in schools (versus those working with EpCs). For example, due in part to its temporary suspension from CARS in early 2016, AMC turned in just one CAP in the first half of 2016 (toward its target of 7 CAPs for the year). In contrast, URACCAN was the highest performer, submitting a total of 18 CAPs in the first half of 2016 (toward a target of 40 CAPs for the year). However, many of URACCAN’s CAPs were missing key documents and had to be resubmitted. The other NGOs fell somewhere in the middle, submitting between 20 percent and 40 percent of the target number of CAPs completed by midyear.20 Despite hitting only a fraction of their total CAP targets for 2016, FHR and FQSF submitted complete CAPs that required fewer revisions than those of other NGOs. 20 See the CARS FY 2015 Q3 report. 45 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH CARS surpassed its goal for community contributions in 2016. Besides facilitating CAPs, CARS staff has encouraged community volunteer efforts focused on reading, provided counseling to parents (primarily through parent schools), and organized events centered on reading. To aid these efforts, CARS often elicits financial and in-kind contributions from the community, including locales for holding reading events and donated reading materials. In 2016, the dollar value of these community contributions to CARS was $71,159, surpassing the target of $60,000. These achievements underscore the ability of the CARS team to engage local stakeholders in reading and security activities, to the extent that stakeholders offer their support and take an active role in CARS activities. Table IV.5. Planned and actual CAPs and community contributions . Goal (2016) Actual (2016) Goal met? Communities with EpCs that have submitted and executed CAPs 90 CAPs submitted and executed 51 CAPs submitted and 5 executed No (6% execution) Dollar value of contributions from local partners to implement CAPs and/or reading campaigns $60,000 $71,159 Yes (119% execution) Source: CARS indicator worksheet, December 2016. C. What are the key facilitators and barriers of CARS implementation? In interviews and focus groups, stakeholders identified facilitators—factors that enhanced the CARS program’s chances of improving reading and security outcomes—as well as barriers— factors that complicated the program’s chances of improving these outcomes. In general, stakeholders saw CARS reading program educational approach and materials as far superior to existing reading curriculum and materials (facilitator), but distribution delays and language mismatches complicated the effective use of these materials (barriers). Similarly, stakeholders praised the usefulness of initial CARS educator training (facilitator), but the training’s potential to change teachers’ classroom practices was likely undermined by infrequent coaching and follow-up visits (barrier). With respect to community engagement efforts, dynamic reading events and interactive parent school sessions have successfully engaged some parents on important topics (facilitator), but the NGOs’ limited implementation of parent schools likely led to missed opportunities to influence parent behavior with respect to reading and safety (barrier). Furthermore, a lack of initial clarity with respect to CAP eligibility requirements, a long development and approval process, and deficiencies in initial CAPs contributed to a low number of executed CAPs as of late 2016 (barriers). Larger capacity constraints, coordination issues, and bureaucracy likely played a role in material distribution delays, infrequent coaching and follow-ups, and lack of progress with the CAPs (barriers). Below, we discuss these facilitators and barriers in more depth. (Table IV.6 summarizes key facilitators and barriers to implementation. See Appendix B for more detail on the stakeholder perceptions of the CARS activities that are discussed below.) 46 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Table IV.6. CARS implementation facilitators and barriers according to stakeholders Area Facilitator of effective implementation Barrier to effective implementation Reading curriculum and materials • Strong, highly structured APA/QL educational approach engages children with active learning techniques. • Didactic materials encourage learning in and out of school. • A variety of colorful manipulatives facilitate early reading activities. • Mismatch in the language of reading materials delivered to some schools and EpCs, in that some materials are not in the children’s mother tongue. • A minority of teachers and facilitators complained of delays in receiving CARS materials, in some cases weeks or months after reading programs had begun. Educator training and follow-up • Educators widely praise the quality and usefulness of the CARS training sessions. • Training is somewhat limited in intensity and scope, creating instances in which educators fail to fully internalize the methodology. • Coaching visits occur less than originally supposed, leaving educators without critical guidance and feedback. Parent and community engagement efforts • Dynamic CARS reading events and interactive parent schools successfully engage parents on important topics of early reading and their children’s development. • Due in part to NGOs' limited community presence and a lack of motivation among some parents, parent school attendance is generally low across participating communities. • A lack of clarity with respect to CAP eligibility requirements and a long development and approval process contributed to a low number of executed CAPs in 2015 and 2016. Implementer leadership, capacity, and coordination • Educators noted that CARS trainers and liaisons have a good mastery of the material. • High DevTech and NGO staff turnover have contributed to leadership and capacity gaps. • Coordination gaps exist between DevTech and NGOs, as well as among NGOs. • There are bottlenecks in formal approvals and concentrated decision making. School setting • Teachers generally have strong support from principals to participate in CARS training and to apply what they learned in the classroom. • There is a lack of coordination between EpC facilitators and teachers with respect to lesson plans, reading assessments, and homework for students who attended EpCs and school. Community support • Many parents and community leaders value the support that EpCs offer to children who struggle behaviorally and academically. • Some parents lack motivation or incentives to attend parent schools and engage on topics of reading and security. • Some parents and leaders have only vague or inaccurate ideas about the goals and activities of EpCs. Other environmental factors . • There are low EpC attendance and retention rates due to long commutes, students’ need to eat after school, household chores, migration patterns, and security concerns. 47 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Table IV.6. (continued) Source: In-person interviews with seven DevTech representatives, seven NGO representatives, two USAID representatives, two SEAR representatives, 35 principals, 54 teachers, and 26 facilitators, in addition to 36 focus groups with parents and 35 focus groups with community leaders, from June to October 2016. Reading program educational approach and materials: Attractive CARS educational approach and materials are somewhat compromised by distribution delays and language mismatches. CARS representatives and educators emphasized the natural advantage of the CARS educational approach compared to traditional teaching techniques commonly used in the region. According to these stakeholders, QL and APA educational approach engage, encourage, and motivate students, whereas traditional techniques treat them as passive participants. Educators also widely praised CARS reading and manipulatives, which they use to guide and complement reading and writing activities. Unfortunately, instances of delayed distribution of CARS materials and at least eight cases of materials that were not in students’ native language (among 80 interviewed educators) compromised the potential effectiveness of reading programs in some schools and EpCs. Educator training and follow-up: Strong initial CARS training has been undermined by inadequate coaching and follow-up. Educators praised the initial CARS training as useful and relevant to their everyday work. Furthermore, they rated CARS trainers as very knowledgeable on active learning techniques. However, some NGO representatives mentioned that training was not intensive or comprehensive enough, and several teachers requested additional training to help them fully integrate APA techniques into their daily routines. Although teachers and principals generally expressed strong appreciation for NGO follow-up visits, stakeholders widely identified NGOs’ infrequent follow-up visits as an area for improvement. In this sense, inadequate follow-up from NGOs represents a failure to reinforce the practices covered in training and to help educators implement them in the classroom. Parent and community engagement efforts: Reading events and dynamic parent schools have engaged some parents on important topics, but parent school attendance is generally low and CAPs have failed to yield tangible benefits. Dynamic CARS reading events and interactive parent school sessions have successfully engaged some parents on important topics of early reading, security, and their children’s development and have encouraged positive behavior change with respect to reading and interacting with their children. However, the NGOs’ generally limited implementation of parent schools has likely led to missed opportunities to engage parents on important topics and to influence their behavior. Furthermore, a lack of initial clarity with respect to CAP eligibility requirements, a long development and approval process, and deficiencies in the initial CAPs contributed to a low number of executed CAPs as of late 2016. As a result, parents and community members have generally failed to experience the tangible improvements to schools and communities that were initially envisioned under CARS. This is a dangerous development, as it could actually decrease the motivation of parents and community members who helped conduct school assessments and compose the CAPs. 48 IV. IMPLEMENTATION FINDINGS MATHEMATICA POLICY RESEARCH Implementer leadership and capacity: Capacity constraints, coordination issues, and bureaucracy hinder CARS’ ability to meet ambitious implementation targets. In interviews, NGO and DevTech staff noted that they did not have enough capacity to meet aggressive implementation targets for reading programs and community engagement programs, while organizing events, collecting data, facilitating the EpC impact evaluation, and submitting required reports to DevTech. Personnel changes at DevTech and the NGOs further exacerbated these capacity constraints and produced leadership gaps in instances in which replacement staff members could not be identified in a timely manner. NGO staff also complained about uncoordinated information requests and little advance notice of meetings and events from DevTech, while DevTech cited several instances in which the NGOs failed to coordinate with one another in instances in which they operated in the same communities (for example, in the case of one NGO that implemented an EpC in a community where another NGO implemented the formal reading program). In addition, the NGOs and DevTech remarked about the prolonged length of time it took program leadership to review and approve didactic materials, CAPs, and newsletters, noting that this review often created bottlenecks to distributing materials on time and disbursing funds for much-needed community projects. School environment: Principals generally support CARS activities, but there is a lack of coordination between EpC facilitators and teachers. In interviews, preschool and primary school teachers often noted that they had the full support of their principals to implement CARS teaching techniques. However, NGO staff noted that a lack of coordination between EpC facilitators and teachers led to missed opportunities for facilitators and teachers to work together to organize complementary lesson plans and activities, such that EpC activities could reinforce what children were learning in school and vice versa. Community environment: Most community members expressed strong support for the EpCs, but parents’ motivation and incentives to attend parent schools remains a concern. Most parents and community leaders noted that they value the role that EpCs play in supporting students who struggle academically or behaviorally. A minority of parents observed that children are doing “more playing than learning” in EpCs—likely in reference to EpCs’ organized play activities and active learning techniques. In addition, DevTech and NGO staff noted a lack of motivation or incentives among some parents to attend parent schools, which contributes to low parent school attendance and missed opportunities to engage parents on topics of reading and security. Other environmental factors: Low EpC attendance and retention threaten to undermine the EpC model. EpC attendance and retention are below initial assumptions due to a number of factors, including parents’ preference, migration, long commutes, and the need to eat after school. This poses a threat to the program’s ability to improve kids’ learning outcomes, as the most high-risk students are often those who fail to enroll or those who drop out of EpCs prior to the end of the 20-month term. 49 This page has been left blank for double-sided copying. V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH V. EFFECTS OF CARS In this chapter, we distill information from reports, administrative data, interviews, and focus groups with stakeholders into a summary and analysis of the potential effects of CARS reading programs as well as community engagement, local capacity strengthening, and knowledge management efforts. A. Are teachers applying early-grade reading approaches—including using new materials and assessment tools? Teachers, facilitators, and principals reported applying the techniques they learned in CARS training, as well as using CARS didactic materials and manipulative materials in class. A comparison of teacher-reported practices between CARS-trained 1st-, 2nd-, and 3rd-grade teachers and teachers of the same grades not trained by CARS suggests that CARS-trained teachers adopted several core APA practices in class, including implementing routines, singing songs, organizing activities through games, proposing group activities, using hugs and affection, and suggesting that students share what they learned at home. However, most EpC facilitators and CARS-trained teachers didn’t report administering reading assessments as often as envisioned, or using the results to inform their teaching or to identify students with poor reading skills. Teachers, facilitators, and principals overwhelmingly reported applying what they learned in training. Nearly all educators reported applying what they learned in CARS training—with the exception of only two (of 35) principals, who reported that they were too busy with administrative activities to visit classrooms. In particular, primary school teachers and facilitators reported using active learning techniques to help children learn by singing, playing, practicing, or working in groups. Preschool teachers often reported using letter flash cards to teach their students letters and basic sounds, and playing games like “Brainstorm” and “I’m also a teacher” with the kids, where kids explain something to the teacher. Teachers noted that these teaching techniques are fundamentally different from the passive techniques they previously used in class. All interviewed teachers, facilitators, and principals reported using CARS didactic materials and manipulative materials in class. All EpC facilitators and teachers and principals who teach at least one class reported using CARS materials regularly. However, teachers were more likely than EpC facilitators to report using CARS materials selectively according to kids’ grades and needs—for example, one teacher asked 2nd-grade students more difficult comprehension questions than her 1st-grade students after sharing a story from a CARS storybook. In particular, teachers and EpC facilitators use manipulative materials to complement reading lessons. EpC facilitators and teachers noted that paper, pencils, and colored paper were some of the most useful materials. Teachers reported regularly using markers and construction paper to complement reading and writing activities. For example, teachers reported asking kids “When we got down deeper into this capacitation with CARS…there is where we realized, we understand that children do not learn only just by standing in front of them sharing. But they also learn by having a lot of games, they learn by singing they even learn through movement dancing and a lot of different things they can also learned through so, is like if the method was completely more active.” –—School principal 51 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH to draw a picture about a character they just read about, or asking them to draw a picture of a person when they are learning new words for body parts. Nacho Nicaraguense workbooks, storybooks, and manipulative materials get the most use, and musical instruments and some electrical devices get the least use. Teachers noted that Nacho Nicaraguense workbooks have stimulated children’s development, and provided a strong structure to reading exercises. One teacher said the Nacho Nicaraguense workbooks helped involve parents more in their kids’ education because parents and children work together on the books’ exercises at home. An NGO representative made a similar point that the Nacho workbooks really facilitated parent participation in their children’s homework, and another teacher noted the workbooks were a critical donation given families’ economic need. Several teachers also noted that kids love CARS storybooks and that they consistently request story time during class. Teachers and facilitators alike were also pleased with CARS-donated manipulative materials. The least popular items distributed include musical instruments. At least four of 26 interviewed facilitators noted that there is rarely a teacher or facilitator available who can play the guitar or the flute. At least five of 54 interviewed teachers also noted that they could not plug in some of the musical equipment, as their school does not have electricity. Several teachers described being “awakened” to all the teaching possibilities as a result of CARS training. Once educators have grasped the concept of varying the teaching method and using variety in classroom activities, they have started thinking of new possibilities, even beyond the activities covered in training. For example, one teacher started applying active learning techniques to teaching basic math skills. Some EpC facilitators who were also teachers reported applying new techniques used in EpCs in their classrooms as well; this included techniques to get students’ attention, such as using key words and applause to focus students on the task at hand or change topics. Most educators don’t appear to be administering reading assessments themselves, but most reported that assessments were completed for their students. Overall, less than one￾third of educators reported administering a reading assessment themselves in the past year. A DevTech representative noted that some educators have trouble conducting the assessments, and this may discourage them from completing them. However, the large majority of educators (83 percent) noted that reading assessments were completed at least once in the past year for their students, either by the CARS team, themselves, or the school principal. Most educators reported using the results of reading assessments, but not for their intended purpose. In interviews, around two-thirds of interviewed facilitators, teachers, and principals who received CARS training said they used the results of reading assessments in the past year (Table V.1). This was below the 2016 goal of 85 percent across trained educators. Furthermore, educators often had difficulty articulating how they use reading assessments. Several interviewed directors and facilitators made general statements about how they use the results for “evaluation” but could not provide specific detail. Other facilitators and directors appeared to view assessments as more of a teaching or practice technique than an evaluation tool. “When we come from secondary school we just talk and talk. But now I know we can teach through radio, through television, through dramatization, through games . . . so now I’m going to go back to class with more ideas, more creativity to help the children learn even more.” -—CARS-trained teacher 52 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH For example, one teacher reported using the timed reading assessments as a sort of game in class to motivate students to read faster, not to make any adjustments or identify students who required specialized attention. In interviews, DevTech staff acknowledged that a nontrivial portion of educators aren’t using the assessments to inform their lesson plans, and noted that a barrier to using assessments is that many educators can’t do the data manipulation required to process the reading assessments themselves. To help remedy the situation, NGOs have offered educators training and hands-on practice in applying and interpreting reading assessments. However, because educators have a range of capacities and experience, DevTech staff noted that it remains a challenge for EpC facilitators and teachers to apply assessments in a valid way, properly interpret the results, and adjust their lesson plans or provide individualized attention accordingly. Table V.1. Planned and actual use of reading assessments Indicator Goal (2016) Actual (2016) Goal met? Percentage of educators who used assessment tools to track student reading progress 85% • CARS calculations: 75% • Mathematica calculations: 65% of all educators (70% of principals, 65% of facilitators, and 62% of teachers) No Source: In-person interviews with principals, teachers, and facilitators conducted in September/October 2016. Note: Sample for Mathematica calculations includes all facilitators and all teachers and principals from the 36- community sample that reported participating in at least one CARS training. There is much suggestive evidence that primary school teachers changed their classroom practices as a result of CARS. Primary school teachers at private and subsidized schools who participated in CARS training were much more likely to report using APA methods compared to primary school teachers who had no CARS training (Figure V.1). In particular, CARS-trained teachers were over 25 percentage points more likely to use hugs and affection, try to resolve conflict through dialogue, have students use the dictionary in the classroom, and read stories out loud to the boys and girls. Similarly CARS-trained teachers were over 15 percentage points more likely to encourage routines in class, do conscious breathing exercises, sing songs in class, propose group activities, suggest that students share what they learned at home, and discuss the motivation of characters in books and stories. However, it is possible that these differences reflect underlying systematic differences between teachers in public and private schools, as most interviewed teachers who got CARS training worked in private schools, and most interviewed teachers who were not trained by CARS worked in public schools. 53 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Figure V.1. Percentage of primary school teachers who practice each activity at least once per day: CARS-trained versus nontrained teachers Source: In-person interviews with 38 trained preschool and 1st- through 3rd-grade teachers and 19 nontrained preschool and 1st- through 3rd-grade teachers in September/October 2016. B. Is CARS improving children’s outcomes? In interviews, educators indicated that CARS had a positive effect on students’ reading, socialization, and attendance—and gave concrete examples of these improvements. Some CARS student assessment data corroborate reading improvements, in that some students’ reading fluency increased dramatically after one year of EpCs and formal reading programs. Despite any potential improvements linked to CARS, a large gap remains between students’ current reading performance and CARS goals. Given the serious socioeconomic and education challenges in the region, CARS’s NGO implementation model, and the short time frame to achieve fairly dramatic performance results, these original goals may have been overly ambitious. 54 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Interviewed stakeholders expressed a strong belief that CARS has registered a strong impact on students’ reading, achievement, attendance, self-control, self-esteem, and values. Nearly all principals and facilitators, and about 9 out of 10 teachers, agreed that CARS had a positive effect on a variety of academic and socioemotional outcomes (Figure V.2). Stakeholders gave examples of nonacademic improvements related to CARS—including better communication skills and stronger self-esteem as a result of EpC activities—as well as examples of academic improvements linked to CARS. Notably, CARS representatives and educators were more likely to provide concrete examples of students’ improved academic outcomes—including attendance, reading, and achievement—than parents and community leaders, who focused on students’ behavior and self-esteem. This may reflect the greater exposure to students in an academic setting that CARS teams and educators have relative to parents and community leaders. See examples and illustrative quotes regarding the effects of CARS in Table V.2. Figure V.2. Percentage of educators who believe CARS formal and nonformal reading programs have had an effect on student outcomes. 97 100 97 97 97 92 92 92 90 92 100 100 96 100 96 0 20 40 60 80 100 Reading Achievement and grades Attendance Social skills Values Percentage Principals Teachers Facilitators Source: In-person interviews with principals, teachers, and facilitators conducted in September/October 2016. Note: Sample includes all interviewed teachers and principals, including those who did not take part in CARS trainings. 55 56 Table V.2. Stakeholders’ perspectives on effects of CARS EpCs (nonformal) and formal reading programs Outcome Effects according to: CARS and NGOs Teachers, facilitators, and principals Parents and community leaders Child engagement The CARS educational approach directly engages students in reading activities. “The kids read in little groups … I’ve seen more participation and activity. It’s a pleasure to see the kids involved and doing things … different from other classes that are more passive —NGO representative Children request story time and flash card practice in particular. “When we do an innovated activity, the kids say, ‘Teacher, are we going to do it again tomorrow?’ They want to be here in class to be involved in the stories and the flashcards.” —CARS-trained teacher EpCs have nurtured children’s enjoyment of reading and encouraged them to develop good reading habits. “My daughter loves reading, and [CARS] recently gave her a story book … it’s big and she’s halfway through it, and she reads stories to her little sister.” —Community leader Behavior Boys and girls are treating each other with more respect in EpCs. “I see changes in the kids … when it’s time for them to interact, I’ve seen changes in their respect and support for each other … in participating together without having a separation between boys and girls.” —NGO representative Students are more likely to help each other out. “Our the kids have a little of them being the way, the [environment] they are the living in the neighborhood is like: you hit me I hit you!, so the teachers try [to] teach them that instead of solve a situation with violence that will give you more problems like better to use dialogue … [we don’t have] 100 percent control of that situation but we improve a lot.” —Principal Children are displaying more mature and respectful behaviors. “I see that the children … although they are small they behave like adults; they are respectful, and they are interested in their studies and we as parents see that their behavior is good.” —Parent Reading and writing Kids with reading difficulties have improved substantially with good exposure to the CARS educational approach. “We had some kids that had serious learning problems. We did regular reading assessments and those kids have improved a lot in their scores … to a lot higher categories.” —NGO representative Students have registered improvements in comprehension and analysis. “When they came to do the EGRA, they said the kids can read but they can’t analyze what they’ve read. After that we started them working on reading and writing, asking them questions about what they read … and they’ve gotten better on the analysis part.” —CARS-trained teacher EpCs have helped children develop and improve their basic reading and writing skills. “My child writes better and expresses himself better … reads better. He already knew how to read but now he corrects himself more.” —Parent 57 Table V.2. (continued) Outcome Effects according to: CARS and NGOs Teachers, facilitators, and principals Parents and community leaders Social skills Students are reported to be more sociable, less timid around each other and adults, and better listeners as a result of their time in EpCs. “We saw really self-assured kids, talking in front of a group of 20 or 30 people with no fear.” —NGO representative Facilitators intentionally take students out of their comfort zone to meet and get to know other kids. “We have one part in the quantum learning what is how to take out the person out of them comfort zone … you just teach them that when you work together it’s easier … We had different games like you say, ‘everyone who has the same shoes, make a group’ and you ask how you name … things that people learn to know one another.” —Facilitator Children are developing communication skills and self-confidence. “In the storybook group the kids used to be timid—they talked, but they were shy. Now they are more free, they talk amongst themselves, you can see they’re happy, playing, jumping … it’s nice to see them develop, even their moral character.” —Community leader Attendance EpC retention rates are low, but students who remain in EpCs have better school attendance and enrollment than when they started. “[With the EpCs] we’ve met the goal of increasing school enrollment by 10 percent.” —NGO representative The variety of interesting activities in EpCs motivates students to come to school every day. “Well the kids like going to school. They think about all the activities they’re going to do … they say ‘tomorrow we’re going to do such￾and-such thing.’ It’s almost like the kids are just waiting for tomorrow to come so they can go to school.” —Principal Community leaders and parents did not note any substantial changes in student attendance related to CARS. 58 Table V.2. (continued) Outcome Effects according to: CARS and NGOs Teachers, facilitators, and principals Parents and community leaders Achievement EpCs in particular have improved students overall academic achievement by helping them complete their schoolwork. “They’ve gotten better with their grades. Kids that attend EpCs bring their homework in … so it’s gotten better.” —NGO representative Children’s schoolwork has improved through regular assistance with homework in EpCs. “There are a lot of parents that don’t know how to read and don’t have time … so they don’t help their kids. [The EpC facilitators] help those kids do all their homework.” —CARS-trained teacher Some parents and leaders reported improvements in children’s overall academic achievement, but others reported no changes in their children’s achievement or grades as a result of EpCs. “Yes, I’ve seen my daughter’s grades improve. She’s got more interest in studying now.” —Parent “Well I see my child’s grades as the same as before. Being in the EpC, I see that they do about the same in school as before.” —Parent Source: In-person interviews with 7 DevTech representatives, 7 NGO representatives, 2 USAID representatives, 2 SEAR representatives, 35 principals, 54 teachers, and 26 facilitators, in addition to 36 focus groups with parents and 35 focus groups with community leaders from June to October 2016. CARS = Community Action for Reading and Security intervention; NGO = nongovernmental organization; EpCs = Espacios para Crecer; EGRA = early-grade reading assessment. V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Educators believe CARS has generated positive effects on children’s outcomes, but preliminary evidence shows a large gap between current reading levels and targets. In interviews, educators indicated that CARS had a positive effect on students’ reading, socialization, and attendance—and gave concrete examples of these improvements. For example, teachers noted that students have registered tangible improvements in reading comprehension as a result of reading programs, which emphasize analysis of stories and reading passages. Some CARS data corroborate these reading improvements. For example, comparing baseline mini￾EGRA evaluations to follow-up evaluations one year later for students enrolled in Cohort 1A EpCs, NGO staff measured an average increase of 28 words per minute among 1st-grade students—well above the increase of 14 words per minute that can be expected in one year21 (CARS 2016a). (However, students in 2nd and 3rd grade did not experience similar improvements.) Another NGO measured a reduction from 64 percent to 44 percent of 2nd-grade students in the high risk category after one year of CARS exposure (CARS 2016c); high risk is defined as being able to read fewer than 25 words per minute in 2nd grade. Despite any potential improvements linked to CARS, a large gap remains between students’ current reading goals and the CARS targets. Among kids enrolled in EpCs and participating primary schools, the percentage of kids that read at grade level in 2016 (33 and 34 percent for males and females, respectively) was far below the CARS goals of 53 and 62 percent for males and females, respectively (Table V.3). According to DevTech, these results obscure the positive effect of CARS on reading outcomes. Although students may have improved their reading performance under CARS, this improvement is not sufficient to qualify students as reading “at grade level,” given low baseline reading levels. Given the serious socioeconomic and education challenges in the region, CARS’s NGO implementation model, and the short time frame to achieve fairly dramatic performance results, these original goals may have been overly ambitious. Table V.3. Planned and actual student achievement and retention, CARS formal and nonformal reading programs Indicator Goal (2016) Actual (2016) Goal met? Kids that read at grade level 53 and 62 percent for males and females, respectively 33 and 34 percent for males and females, respectively No Percentage of kids that stay in EpCs 86 percent Cohorts 1–2: 70 percent No Source: CARS indicator worksheet, December 2016. EpC retention rates are lower than expected, but CARS notes high promotion rates among students who stay in EpCs. Across all NGOs in all municipalities, CARS estimated a 70 percent retention rate for EpCs during 2016, compared to the goal of 86 percent (CARS 2016d). (Interviews with 26 EpC facilitators found similar retention rates among active Cohort 1 and 2 EpCs in the data collection sample). NGO field staff and facilitators have told DevTech staff that low EpC retention rates are due to various reasons, including families migrating to seek seasonal work in fishing and agriculture, lack of parent interest in EpCs, substance abuse problems among primary caretakers, and long commutes to and from school (CARS 2016b). 21 RTI International. “Early Grade Reading Assessment toolkit. Prepared for the World Bank, Office of Human Development, under Contract No. 7141961. RTI International, Research Triangle Park, NC. 2009, p. 60. 59 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Despite generally low retention rates, there was suggestive evidence that those who remained in the EpCs had a higher probability of staying in school. For example, of all Cohort 1 EpC enrollees who completed the session, 83 percent moved on to the next grade (CARS 2016a). Private school drop-out is generally low, but there are pockets of high drop-out in Bluefields and Corn Island. Across all private and subsidized schools in which CARS works, CARS measured minimal desertion in grades 1 through 3 during 2016 (CARS 2016a). Data collection visits of 18 private schools found higher drop-out rates of around 10 percent for grades 1 through 3, according to principals.22 However, desertion rates varied across different municipalities. FQSF noted that in seven Bluefields schools there was a desertion rate of between 2 and 12 percent. On Corn Island, the desertion rate was higher: between 14 and 37 percent in six different schools (CARS 2016a). DevTech cited family migration due to limited work opportunities as a primary factor in high drop-out rates. This was particularly common among Miskitu families, who move to Puerto Cabezas during the harvest and planting seasons. Without a viable comparison group of schools that did not receive CARS formal reading programs, it is impossible to determine whether CARS had an impact on private school desertion rates. C. Is CARS reducing gender disparities? CARS may help boys reach parity with girls in reading, but stakeholders were more likely to mention another effect of CARS reading program: better gender integration as a result of EpC and classroom activities. RACCS residents report an unfavorable view of gender-based violence in 2016. CARS reading programs may help boys reach parity with girls in reading, but rigorous evidence is currently unavailable. DevTech and NGO representatives noted that boys and girls have similar enrollment and reading outcomes in grades 1 through 3, given their relatively equal access to education. However, as early as 1st and 2nd grade, boys tend to exhibit less motivation to read than girls, and their reading proficiency suffers as a result. One NGO representative noted this phenomenon while beginning to work in primary schools with CARS. With the introduction of APA methods, however, the NGO representative found that games and group work helped motivate boys in particular to start reading—to the extent that their reading outcomes were likely on par with those of girls after one year of the formal reading program. However, the NGO representative did not yet have quantitative evidence of this effect, and facilitators and teachers did not note this phenomenon in interviews. Assessment data for children assigned to EpCs illustrate that 18 months after EpCs began, girls assigned to attend EpCs had slightly better decoding and reading comprehension skills than boys assigned to attend EpCs. (See Appendix A for more details.) However, it is unclear if EpCs had any effect on these 22 These drop-out rates should be considered a rough estimate, in that they do not reflect whether children transferred to another school when they left these schools, or simply stopped attending school. “There’s always more participation and better performance among girls in some of the schools, but in terms of reading levels … it’s leveling out [between boys and girls.]” —NGO representative 60 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH gender differences in any direction. The forthcoming EpC impact analysis will be able to explore this topic in more depth. CARS may promote better gender integration. During interviews, educators noted that CARS plays a role in promoting gender integration in the classroom. Notably, facilitators and teachers use group activities and games to break down pre-existing gender lines in the classroom. Whereas boys and girls used to play exclusively among themselves in the past, they now consider it normal to play and learn in mixed groups. This outcome of greater interaction between boys and girls in EpCs and classrooms is a positive development, as it breaks down gender barriers and encourages boys and girls to have similar educational experiences. Residents of the RACCS reported an unfavorable view of gender-based violence in 2016, but whether CARS activities influenced residents’ views is unclear. In a telephone survey of residents of the RACCS, CARS found that most people surveyed in the RACCS—and particularly males—had an unfavorable view of gender violence. This is a positive development, in that it reflects low tolerance for such violence in the region. This low tolerance may have been influenced, in part, by CARS parent schools, in which gender-based violence is a primary topic. However, whether CARS actually influenced residents’ opinions on this topic is unclear, given the lack of baseline information on residents’ views on this topic. D. Is CARS improving parental and community support for early reading and security? Stakeholders generally agree that CARS has improved parental and community support for early reading, but parents and community leaders in particular failed to connect CARS activities (such as parent schools or CAPs) with increased community support for security. However, parents often mentioned that parent school discussions have helped them understand their children’s development and behavior better, and to be more thoughtful in their communication with their children. NGOs noted that community engagement efforts have increased community support for early reading, and parents and community leaders partially corroborated these accounts. In interviews, NGO representatives claimed that parents have come to place more importance on their role in their children’s education, particularly early reading, as a result of CARS-sponsored parent schools. Some community members expressed similar sentiments. For example, one focus group participant noted the positive influence of parent schools on fathers’ interest and involvement in their children’s education, and another highlighted the positive effect of a CARS-funded library (financed under a CAP) on parents’ interest in early reading (See Table V.4 for illustrative quotes). There is little evidence that community engagement efforts had any tangible effects on community support for security or school and community conditions. One NGO representative noted that as a result of CARS activities—including parent schools and CAPs— parents are more conscious of security concerns in the community, particularly within and around preschools and primary schools. However, parents and community leaders did not mention enhanced community support for security resulting from these activities. In addition, parents and community leaders noted no tangible improvements in school or community security 61 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH as a result of CARS. In part, this may reflect low rates of CAP development and execution in sampled communities. According to the initial CARS design, CAPs could provide funding for school and community improvements oriented toward greater safety and security, such protective walls for schools. Stakeholders generally agree that parent schools have helped parents participate in their child’s education and better understand their behavior. In structured interviews, over 90 percent of principals and EpC facilitators agreed that overall, CARS had succeeded in increasing parent participation in their children’s education. However, fewer teachers (around 75 percent) agreed with this statement—perhaps reflecting the lower rates of parent school attendance in private and subsidized schools. In focus groups, parents and community members noted that parent schools provide a valuable forum for parents to communicate with teachers and facilitators about their children’s behavior, academic progress, and personal development. In addition, parents, community members, and NGO staff agree that parent school discussions have helped parents understand their children’s development and behavior better—particularly the negative role that bullying and domestic abuse can play—and to be more thoughtful in their communication with their children. For example, in one focus group, a parent reflected that a parent school session helped make clear how children model their parents’ behavior, and how domestic abuse in the household can have a negative effect on children’s achievement (Table V.4). According to DevTech and NGOs, CAPs have had some positive effects, but community leaders and parents generally weren’t aware of CAPs, school assessments, or school report cards. According to DevTech and NGOs, CAPs had generated some tangible improvements at the community level, even in cases in which CAPs had yet been approved. According to program implementers, the process of developing school report cards and CAPs called some communities to reflect more deeply on their schools' deficiencies, and even helped a handful of community groups to solicit and obtain outside (non-USAID) funding for school improvements. In contrast with CARS reports, community leaders and parents had difficulties identifying changes in their communities related to CAPs. In fact, most leaders and parents had not heard about school assessments, school report cards or CAPs—particularly in Creole- and Miskitu-speaking communities. Table V.4. Changes resulting from parent schools Changes in: Reported by NGOs Corroborating quote from a parent, community leader, or educator Parents’ support for early reading and security Parents have come to place more importance on early reading, particularly the role that early reading plays in facilitating students’ success in primary school and beyond, and the influence that parents can have on their children’s reading. “There are parents who come to [parent school] meetings … there’s interest … [The schools] are sparking parents’ interest in their kids, and it’s good.” —CARS-trained teacher 62 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Changes in: Reported by NGOs Corroborating quote from a parent, community leader, or educator Parents’ participation Parents visit school more often to attend parent schools, and parent school sessions provide them an opportunity to check in with teacher and facilitators about their child’s progress. According to some reports, parents are more engaged in EpC parent schools than in regular parent meetings led by MINED. “As MINED we have a [parent school] session every month with parents with different topic that come … Some parents don't participate in the session that we have every month [at school]—[it doesn't matter] how dynamic you try to make it. And I notice that the same parents that [don't] come to [the MINED] session…are participating actively in the EpC session!” —Principal Parents’ awareness of their children’s behavior and development Parent schools raised parents’ awareness about problems that children may face in and out of school and that can be at the root of misbehavior. Parent schools have also raised parent awareness about how domestic violence can impact children’s development. “There’s always a reason a child acts a certain way … we can think a child is acting a certain way because they’re spoiled, but maybe they have a problem like bullying at school, and we’re ignorant of the issues. But in the parent schools they cover those topics and they help us understand our children better.” —Community leader “If a child’s dad pulls his mom’s hair, the child carries all that with them … all that affects their learning.” —Parent Source: In-person interviews with 7 DevTech representatives, 7 NGO representatives, 35 principals, 54 teachers, and 26 facilitators, in addition to 36 focus groups with parents and 35 focus groups with community leaders from June to October 2016. NGO = nongovernmental organization; EpC = Espacio para Crecer; MINED = Ministry of Education; CARS = Community Action for Reading and Security intervention. E. What type of outreach and awareness efforts are most successful in increasing engagement? NGOs have had some success with community engagement efforts, particularly with parent schools that generated meaningful discussions and reading events that involved parents and their children. Small-scale CARS events in participating communities that foster educational play and reading between parents and children appear to be the most successful in stimulating parent interest and increasing parent school attendance. In addition, public events recognizing businesses’ counterpart contributions to CARS have incentivized community partnerships. Best practices for community engagement include recognizing donors and holding inclusive and interactive events. For example, one NGO had a public event to recognize businesses and institutions that offered counterpart contributions to CARS activities. According to the NGO, the activity motivated local businesses to continue supporting project. Another NGO expressed plans to hold a party at the conclusion of each EpC—a modest closing ceremony where they celebrate with parents and community leaders. Such events will have some of the same activities as larger festivals that CARS has sponsored in the past—including games and entertainment—but they are designed to include all parents and community members in each 63 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH community—not just a limited number of invited guests. This innovation would reinforce an inclusive atmosphere and minimize participant travel. Another NGO holds reading clubs, or regular meetings in which parents read to and play with their children (Table V.5). The NGO reported that these clubs are very popular among parents and children alike. Table V.5. Best practices in outreach events and communication efforts Best practice Example Publicly recognizing collaborators One NGO held a public event to recognize businesses and institutions that offered counterpart contributions to CARS. Holding inclusive, small-scale celebrations in communities One NGO has plans to hold a party at the conclusion of each EpC—a mini closing festival where they celebrate with parents and community leaders. Planning activities that encourage healthy interactions between parents and children One NGO holds reading clubs, or regular meetings in which parents read to and play with their children. Source: Interviews with NGOs implementing CARS. NGO = nongovernmental organization; CARS = Community Action for Reading and Security intervention. NGOs introduced some innovations to facilitate good parent school attendance and discussion. In response to a lack of materials to guide parent school sessions, FZT developed their own didactic materials, guides, and agendas for parent schools. These materials helped structure parent school sessions around designated topics, and introduced some uniformity in the topics discussed and materials presented at schools. Facing poor attendance at parent schools, another NGO convened parent schools on days that parents were already scheduled to report to school. A best practice with parent schools is to make each session a conversation, not a lesson. One facilitator noted a useful practice with respect to parent schools—rather than “talking at” parents, facilitators and teachers could structure parent schools as a conversation or reflection on important topics. This practice reflects the spirit of parent schools, which are designed to provide a forum for parents to reflect and learn from one another, in the interest of becoming better parents and getting more involved and committed to their children’s development. This approach appears more viable than a more traditional didactic approach, in which parents are merely told that early reading is important, and not given a chance to internalize the message. Several parents and leaders also expressed that they liked the participatory approach used in parent schools. “If we have a topic, for example if we said how to educate you child, we ask them how you think said we could … and them give them opinion. Is sharing opinion, is not we going to tell them ‘so you have to do so,’ and ‘so you have to be’ you know, they give them opinion and want they understand. Yes, so is like a little conversation.” —EpC facilitator “Well thanks to Fundación Zamora Terán that give us that material because, sincerely for [us] to go and stand up and talk to parents. It's not sufficient material without the book they give us!” —EpC facilitator 64 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH F. Is CARS improving NGO capacity? Local NGOs have been strengthened as a result of CARS implementation, but it is unclear if they are capable of contracting directly with funders in the future. According to OCA scores, two of the six implementing NGOs, FQSF and FHR, made fundamental improvements on key dimensions, including financial management and human resources. However, these NGOs continued to register deficiencies at midline—particularly in human resources and project management in the case of FHR and organizational management in the case of FQSF. Another two NGOs, URACCAN and FZT, had relatively high capacity when they joined CARS (according to OCA scores) but did not improve significantly from baseline to midline. As of late 2016, NGOs average midterm OCA score was below the target of 3.0 out of 4.0. DevTech staff noted substantive incremental improvements in all NGOs, but there is room for additional improvement. One DevTech representative noted that all NGOs had improved on some subcomponents. For example, one NGO developed and made plans to adhere to a travel policy. Another NGO had difficulties with its internal controls, but it strengthened these controls under DevTech’s supervision. NGO representatives also mentioned these improvements, noting particular improvements in their financial management, administrative processes and manuals, legal status, and strategic planning. However, according to one NGO, CARS’ technical assistance resembles an audit rather than a collaborative work to solve problems. Despite the aforementioned improvements, NGOs’ average midterm OCA scores in 2016 were 2.77, below the goal of 3.0 out of 4.0 (Table V.6) (However, two NGOs, URACCAN and FZT, had OCA scores above 3.0). In interviews, DevTech staff attributed this to several factors, notably a lack of capacity, time, and motivation on the part of NGOs to make improvements articulated under baseline OCAs. Table V.6. Planned and actual average Organization Capacity Assessment total scores among CARS awardees . Goal (2016) Actual Goal Met? Average Organizational Capacity Assessment score at midline (out of 4) 3.0 2.77 No Source: CARS indicator worksheet, December 2016. “The OCA tool reflects an organization’s culture and performance. All [NGOs] have better performance in financial management, but there is still work to do. USAID regulations are more rigorous, and there is a tendency in the region to be more flexible and open. [NGO performance is] a lot better than before.” —DevTech representative 65 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Two NGOs made fundamental improvements on key dimensions but may not be well positioned to manage complex interventions. At baseline, two NGOs (FHR and FQSF) were particularly weak in financial management and human resources, including recruitment of new personnel and staff assessment. Both made substantive progress in these categories by midline. FQSF in particular registered large improvements in human resources (particularly in recruitment and staff policies), financial management (particularly in stronger internal controls and financial procedures), and administration (particularly in fixed asset controls). FHR registered large improvements in financial management (particularly in improving its financial systems, procedures and controls, and documentation) and project performance (particularly in supervision). (Figure V.3) However, at midline both NGOs had total OCA scores that were still below 2.4 out of 4, below the 2016 CARS goal of 3.0, which designates moderate administrative and financial capacity. (Table V.7) “We’re incorporating all these processes and norms into our manuals and they’ll be adopted by the board of directors … they will be a guide for us as an NGO and we’ll abide by those norms.” —NGO representative 66 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Figure V.3. Changes in Organizational Capacity Assessment scores from baseline to midline, CARS awardees 0.1 0.3 0.1 0.6 0.2 0.3 -0.2 0.3 -0.1 0.2 0.4 0.3 0.3 0.5 0.3 0.2 0.7 0.4 0.4 0.6 1.1 1.1 0.4 0.4 0.7 0.7 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 Governance Administration Human resources Financial management Organizational management Program management Project management Total score URACCAN FZT FHR FQSF Source: CARS OCA data set, 2016. Notes: Score decreases for FZT are not due to lower performance at midline than baseline—rather, they are due to the inclusion of subscores that were previously excluded from OCA scores. AMC and BICU had not yet completed midline OCAs at the time of analysis in late 2016. URACCAN = Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense; FZT = Fundación Zamora Terán; FHR = Fundación Hermanamiento RAMA; FQSF = Fundación Yo Quiero Ser Feliz. Two NGOs had relatively high initial OCA scores but did not improve much from baseline to midline. URACCAN and FZT both had baseline total scores above 3.0, but neither made a substantive improvement in the total score from baseline to midline (Figure V.3). FZT did experience some improvements in governability (including developing key documents on mission and vision, and legally constituting a board of directors), and financial management (including documentation and financial reports), but it experienced no substantive gains in other areas. Similarly, URACCAN remained unchanged in nearly all OCA scores and subscores. A DevTech representative stated that these two NGOs “look good on paper” but in some cases didn’t actually use the documents and procedures they had adopted. 67 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH Table V.7. Baseline and Midline OCA scores, CARS awardees . URACCAN FZT FHR FQSF AMC BICU Baseline Midline Baseline Midline Baseline Midline Baseline Midline Baseline Baseline Governance 3.2 3.2 3.4 4.0 2.2 2.6 2.2 2.6 3.6 3.6 Administration 2.6 2.6 3.5 3.5 1.8 2.0 1.8 2.4 3.2 2.8 Human resources 2.9 2.9 3.4 3.6 1.2 1.5 1.2 2.3 3.5 2.1 Financial management 2.9 3.0 2.9 3.2 1.4 1.9 1.3 2.4 3.4 3.1 Organizational management 3.7 3.7 3.5 3.3 2.0 2.3 1.6 2.0 3.0 3.0 Program management 3.6 3.6 3.3 3.5 1.8 2.0 1.8 2.2 3.3 2.8 Project management 3.3 3.7 3.3 3.2 1.0 1.7 1.7 2.3 3.3 2.6 Total score 3.2 3.2 3.3 3.5 1.6 2.0 1.6 2.3 3.3 2.9 Source: CARS OCA data set, 2016. Note: AMC and BICU had not yet completed midline OCAs at the time of analysis in late 2016. URACCAN = Universidad de las Regiones Autónomas de la Costa Caribe Nicaragüense; FZT = Fundación Zamora Terán; FHR = Fundación Hermanamiento RAMA; FQSF = Fundación Yo Quiero Ser Feliz; AMC = Acción Médica Cristiana; BICU = Bluefields Indian and Caribbean University. The OCA tool has key limitations, in that it may overestimate some organizational capacities and fail to capture gains in technical capacity. In interviews, DevTech staff cautioned against using the OCA tool as an objective indicator of NGO capacity because its scores may overestimate some organizational capacities. In particular, they noted that the OCA tool captures whether NGOs have draft or final human resources procedures or financial controls “on paper,” but it does not measure whether NGOs actually follow these policies or controls. USAID representatives disagreed with this sentiment, arguing that the OCA tool does measure capacity accurately if administered properly. In addition, the OCA is not designed to capture the technical expertise and other skills that NGOs have developed as a result of CARS. As noted above, NGOs absorbed and mastered the new CARS early reading educational approach and were able to impart it to educators successfully, by all accounts. By early 2016, DevTech representatives remarked that NGO training teams showed increased technical capacity and teaching skill with each additional training, DevTech representatives also noted that URACCAN and BICU staff applied the EGRA more efficiently as they acquired more experience in baseline and midline assessments. Reflecting this improvement, the assessment cost per student for CARS staff was $27 in 2016, compared to $66 per assessment in other parts of Nicaragua. “[The larger NGOs] have a lot of different things going on and their staff might be dedicating their time to the project but are also helping on other things. They have policies that look good on paper with the OCA tool, but in the real world they can be mediocre.” —DevTech 68 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH It is unclear if NGOs’ recent capacity gains will be institutionalized after CARS. All grantees working with CARS produced sustainability plans at the start of their contract, in which they detailed the actions they will take to ensure that capacity gains from CARS would be sustained after the project. For example, FHR’s sustainability plan notes that the NGO will seek private and local public funding to continue teacher training sessions. However, the implementation of these sustainability plans—and CARS efforts to monitor these plans’ implementation—is unclear. Specifically, it is uncertain whether new organizational knowledge, norms and procedures developed through CARS will be relevant or used by participating NGOs after the project’s conclusion. In theory, some of the more institutionalized NGOs (that have permanent staff and multiple concurrent projects) may have the capacity to incorporate recent gains under CARS—such as better M&E processes and official hiring practices—into their corporate culture and general practices. Under this premise, more-established NGOs such as URRACAN and FZT might have the most potential to consolidate capacity gains under CARS. However, these organizations’ recent OCA assessments revealed they had not made significant organizational improvements under CARS, at least at midline. In addition, the sustainability of capacity gains likely depends on whether participating NGOs have continued project opportunities. If NGOs continue early reading work initiated under CARS in the near term, they are more likely to sustain key staff and keep using new processes. However, if they face a prolonged period of inactivity, they are likely to lose key staff and drop new processes and plans developed under CARS. G. Is CARS succeeding in increasing knowledge, skills, and resources? NGOs and DevTech have shared student assessment data among themselves, and in some cases with teachers, principals, and parents. There appears to be some evidence that CARS is increasing teachers’ and parents’ knowledge with respect to children’s achievement—in that CARS is sharing assessment results with them—but there is less evidence that CARS is equipping parents and teachers with the skills and resources to act upon that knowledge. DevTech also shares student assessments and various analyses with USAID, and DevTech has shared its experience with some education authorities in the region during a small number of key events. However, the extent to which policymakers and education authorities have internalized or used the findings is unclear. “At an organizational level, leaving things in writing or creating policies or procedures [is important]. When I say leaving it in writing, I mean looking for ways to come up with ideas, policies, and norms but not having them be just one person who does great things and then leaves. The OCA is getting at these organizations to have things in writing that they review and update every few years. That helps those things stay there and not leave with each person.” —DevTech 69 V. EFFECTS OF CARS MATHEMATICA POLICY RESEARCH CARS has succeeded in sharing data internally for use in implementing reading programs. During interviews, NGO representatives said DevTech staff shared baseline EGRA analyses and other early reading research with them, and these analyses gave them much-needed context for their training. One NGO also reported sharing EGRA baseline results with parents in order to frame the need for reading programs and more parent engagement in their children’s reading. In addition, DevTech staff said that some teachers were able to process information such as EGRA data and use it to inform their lesson plans and reading activities. (However, in interviews, stakeholders did not provide specific examples of exactly how teachers used the EGRA data.) As such, there appears to be some evidence that CARS is increasing teachers’ and parents’ knowledge with respect to children’s achievement—in that CARS is sharing assessment results with them—but there is less evidence that CARS is equipping parents and teachers with the skills and resources to act upon that knowledge. In sharing its experience with external audiences, CARS has kept early grade reading on the regional agenda. However, there is potential to reach an even broader audience of policymakers. From 2014 to 2016, CARS and USAID shared results with external audiences in several events. For example, in 2016, the CARS leadership gave a keynote presentation on CARS to over 100 high-level governmental and educational authorities from the LAC region at the LAC Reads International Encounter of Early Grade Reading and Writing in Tegucigalpa, Honduras. In addition, USAID staff have organized at least two knowledge management sessions during the period under evaluation where CARS has shared the studies it has produced to other USG implementing partners. Reflecting on these events, a USAID source expressed that CARS has been instrumental in keeping the importance of early grade reading on the regional agenda. However, other stakeholders expressed that the knowledge management component had not fully achieved its goals of sharing data and studies with a broader audience of national and international policymakers. DevTech representatives believed that this limited achievement was a reflection of insufficient human resources within CARS M&E to share findings with a broader audience, combined with the lower priority of community engagement efforts relative to reading programs. A CARS study on the use of knowledge products (CARS, 2016f) concluded that education authorities find CARS reports hard to digest due to their length and technical character. To mitigate these issues, main findings from CARS reports could be presented in briefs or other formats designed especially for educators and decision makers. The CARS report also noted that decreasing the timeframe for approving the release of knowledge reports may make the information produced more timely and actionable, which would increase its chances of being used by decision makers. “We’ve seen a lot with the EGRA baseline; parents are alarmed about the reading results. That helped introduce the project to show it was needed. Some teachers see the value because they can evaluate their students.” —NGO representative 70 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT CARS was designed to improve reading and security outcomes in the RACCS. In this chapter, we provide a summary of major constraints to improving reading performance and security in CARS communities that appear to persist despite the presence of CARS. We also offer recommendations to improve CARS, as well as a global sustainability assessment of CARS in preparation for a planned program closeout in the RACCS in early 2019. A. What are the major constraints to improving reading performance in CARS communities? In interviews and focus groups, stakeholders reflected generally on the largest obstacles to early reading in their communities—generally structural obstacles that existed prior to CARS reading programs and continue to affect children’s learning at present. Four factors were mentioned most often and by the largest variety of stakeholders as the primary obstacles to improved early reading: poor teacher capacity, a lack of materials in school and at home, a lack of parental engagement, and poor attendance. We discuss these factors in more depth below. (Table VI.1 provides a summary of all constraints to reading performance mentioned by multiple stakeholders). Table VI.1. Constraints to improving reading performance mentioned by interviewed stakeholders Category Constraint Mentioned by: USAID SEAR NGOs DevTech Educators Parents Community leaders Human resources Lack of capacity among teachers to teach early reading and exercise discipline . X X X X X Lack of personal commitment on the part of teachers . X . . . . X Not enough teachers for each grade/section and high student-to-teacher ratios . . . . X . X Poor educator remuneration, including salary and per diems, fails to attract and retain high quality educators X . . X . . . School resources and supports Poor school infrastructure: No preschool and no electricity, and unsafe or decaying school buildings . . . . X . X 71 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Table VI.1. (continued) Category Constraint Mentioned by: USAID SEAR NGOs DevTech Educators Parents Community leaders School resources and supports Lack of school lunch or snacks for students has a detrimental effect on their ability to attend school and learn . . X . . . X Educational approach and materials Lack of reading and teaching materials at school X . X X X X X Lack of reading and teaching materials at home . . X X X X X Attitudes and Personal attributes Lack of parental engagement in their child’s development and education X X . X X X X Students are unmotivated or have behavior/disciplinary issues . . . . X X . Students lack confidence or are shy . . . . X . X Language and capacity In some areas, students struggle to dominate up to three languages concurrently . X . . X . . Attendance Poor student attendance X . X X X X X Source: Stakeholder interviews and focus groups from June to September 2016. Note: Barriers in bold were mentioned most often and by the largest variety of stakeholders in interviews and focus groups. USAID = U.S. Agency for International Development; SEAR = Sistema Educativo Autonómico Regional; NGO = nongovernmental organization. 1. Poor teacher capacity A variety of stakeholders noted a lack of capacity among teachers, including poor reading skills, outdated teaching methods, and poor disciplinary practices. DevTech and NGO staff highlighted teachers’ poor reading skills and outdated teaching methods, which often rely on teacher dictation and passive student learning. DevTech and NGO representatives also noted that teachers and principals in the RACCS had little formal training and often had no training in teaching early reading. Several teachers and principals corroborated this sentiment, noting that they do not get as much training or coaching as they would like. DevTech staff echoed this theme, saying that funding for public education was not currently a priority in Nicaragua and that lack of political will manifests itself in the form of poor teacher training. In English-speaking communities, several parents and community leaders mentioned shortcomings in teachers’ ability to exercise discipline in the classroom. According to these stakeholders, teachers should exert more authority in imposing order in the classroom, as poor discipline often resulted in bad student behavior and poor learning outcomes. 72 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH 2. Lack of materials at school and at home Educators, parents, and community leaders note little support, training, and materials from MINED. During interviews, principals of public schools in particular reported that they are in consistent need of materials—including textbooks, storybooks, workbooks, and manipulative materials—to the extent that entire classrooms of students do not have textbooks and relevant teaching materials. Several principals also mentioned lacking basic materials—including pens, pencils, and paper—to run classes and manage school affairs. In focus groups, parents and community leaders also noted a lack of or outdated learning materials, including textbooks and pencils in public schools. NGO representatives and educators also called attention to a lack of reading materials at home. One NGO staff member said that many school-aged children in the region did not have a single storybook at home at their reading level. Furthermore, students rarely have access to books that they could take home, either through a local library or their school. This highlights the critical need for basic school materials—particularly storybooks—that students could take home to read, preferably with their parents or siblings. 3. Lack of parental engagement Most educators say parents provide them with support with chores and basic needs, but they would like more support from parents with respect to their children’s education. In general, principals and teachers at public and private schools reported that parents help out with everyday school needs, including food, chlorine, gasoline, and cleaning. In fact, over 80 percent of educators reported that they got the necessary support from parents to do their job. However, educators would like more support from parents with respect to their kids’ education. Principals noted that overall, they would like to see parents get more involved in their children’s education and progress at school. Several principals said they’d like to see more support from parents with their kids’ homework, including spending just a few minutes reading with their kid a night instead of watching TV. Educators spoke of apathetic parents, but this apathy may reflect, in part, traditional views of parenting versus educating. Particularly in English-speaking communities, principals noted that some parents appear to have such little interest in their children’s education that they rarely visit school—even to pick up report cards, and have to be “hunted down” to discuss their kids’ progress. Principals noted that these parents often tell them they don’t have time to spend on their children’s education because they have more important responsibilities—particularly a responsibility to provide for their families. Several parents in focus groups expressed similar sentiments: that they are not responsible for their child’s education or that that they should not interfere with teachers’ work. This belief may reflect a more traditional understanding of “Here in school, we motivate students to learn … we give a child a book so they’ll take it home and read, and the child comes back the next day and we ask, ‘Did you read the story?’ ‘No’ they say. ‘Why not?’ ‘Because my mom said she doesn’t have time … and that that’s why we go to school so you teach us.’” —Primary school teacher “I think [MINED] should be a little more accessible because there have been a lot of changes to the curriculum…the truth is that we’ve worked alone most of the year.” —Public school principal 73 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH parents’ responsibilities versus those of teachers, in which parents are responsible for providing for and raising their children, whereas teachers are responsible for educating them. 4. Poor student attendance Students’ long commutes may affect their attendance and performance. A baseline survey of children assigned to attend Cohort 2 EpCs found that nearly four in five of these children walked to school, and the average commute from their home to the school was around 30 minutes. This relatively long commute complicates students’ school and EpC attendance, particularly in cases when kids are expected to commute twice a day: once to school and once to EpCs. Commutes are also complicated by weather conditions, particularly in communities in which kids must use major thoroughfares for a portion of their commute. Parents often pull their kids out of school due to economic and labor needs. Primary school–age children also face pressures from their parents to help out around the house or even work for pay instead of attending school. Several teachers noted this phenomenon and one teacher argued that parents don’t let their children “be kids.” USAID, DevTech and NGO staff also mentioned migration due to economic opportunities and lack of money for school materials and clothes as two main factors affecting students’ regular school attendance. NGO and USAID representatives noted that some parents keep their children out of school due to schools’ poor infrastructure—including a lack of latrines—and questionable safety conditions. Reflecting these pressures as well as the commuting difficulties mentioned above, children assigned to attend Cohort 2 EpCs were absent on four school days in the past month, on average. B. What are the major constraints to establishing a safe community environment? In interviews and focus groups, stakeholders reflected generally on the largest obstacles to a safe community environment. Similar to the obstacles to early reading discussed above, these are generally structural obstacles that existed prior to CARS and continue to affect children’s safety and learning. Four factors were mentioned most often and by the largest variety of stakeholders: dangerous commutes, crime in and around schools, ignorance about what constitutes abuse, and poor school infrastructure. Stakeholders see students’ dangerous commutes as a primary safety concern. One principal complained that motorcycle taxis go too fast around small children outside of school grounds and that some police supervision could improve the situation. According to another teacher, taxis won’t transport kids to and from school for the standard fare, and kids end up walking home in the late afternoon after dark. The teacher considered this a serious security risk. Parents and leaders in the community expressed the same concern. Community leaders also noted environmental risks faced by children in their communities. Leaders noted that animals such as cows and snakes can hurt children when they are walking to and from school. Leaders also mentioned that children face serious risks when crossing rivers on their way to school. “This is a small community. There’s violence, mistreatment, and insults toward kids. The kids are scared because sometimes their parents say they can’t go to school and they put them to work, to take care of a baby or wash clothes or fetch water or wash dishes.” —EpC facilitator 74 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Given these conditions, several educators suggested that community leaders and police play a larger role in getting kids safely to and from school. Stakeholders see crime as a pressing issue, and link risking crime rates to large societal problems. One NGO representative noted that parents in the region keep their kids out of school due to concern for their safety, particularly following burglaries, robberies, and physical assaults on school property or near school. Some community leaders voiced similar concerns about safety in their communities, including a general lack of police presence and various robberies— including children being robbed on their way to school. In interviews, teachers and NGO representatives also noted that school break-ins were somewhat common and suggested a stronger police presence to counteract these break-ins. In focus groups, stakeholders connected these security concerns with larger social problems, including drug use and a lack of employment opportunities. Community leaders posited that the lack of constructive leisure activities is partly responsible for drug use among youth, and believe that offering opportunities for youth to get involved in sports or arts could help address the issue. Follow-up surveys with students assigned to attend Cohort 1 EpCs show that children also feel a sense of insecurity in their communities: students’ average score on the security index was somewhat low, at 1.3 on a scale of 0 to 3 points (See Appendix A for more details). According to program implementers, attitudes and ignorance are a major obstacle to safer communities. Interviewed stakeholders noted that quite often, teachers, parents, and students don’t recognize bullying or gender-based discrimination as abuse, and this ignorance stands in the way of safer communities. Although CARS parent schools touch on these topics in some of the meetings, the message provided may need to be reinforced or made more explicit to change attitudes in the community. To remedy the situation, DevTech and NGO staff noted a need to first boost the promotion of awareness among parents and educators about what constitutes abuse and discrimination, and then provide them with the tools and resources to address these problems in their communities. Poor school infrastructure also poses a risk to children’s safety. In focus groups in a limited number of communities, parents and community leaders identified weak school infrastructure as a safety risk, including at least one case of a school building that was not structurally sound. Also related to students’ general safety, stakeholders noted a general lack of kitchens, latrines, and hand-washing stations, as well as poor roofs that leak when it rains. According to stakeholders, these issues pose a direct hazard to children’s safety while school is in session. “We have to work on the security piece more … it has to do with culture. Sometimes people don’t see certain things as abuse or violence, like bullying. It’s a question of time, more than anything when it’s about customs and culture.” —DevTech 75 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH C. What adjustments could improve CARS implementation? Using the constraints to early reading and a safe community environment identified by multiple stakeholders as a conceptual framework, we identify potential midcourse corrections to CARS reading and community engagement activities (see Table VI.2). Suggestions for formal and nonformal reading programs include redistributing materials to match students’ mother tongue and schools’ electricity access, enhancing modeling opportunities and direct feedback in training and coaching visits, and introducing a minimum requirement for the frequency of coaching visits. Suggestions for the community engagement component include promoting events that facilitate parents’ interactions with their children, streamlining the CAP development and approval process, and engaging the broader community and law enforcement officials to enhance school security and protect students as they commute to and from school. (See Appendix C for stakeholders’ suggestions to improve CARS, some of which overlap with suggestions presented below.) 76 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Table VI.2. CARS efforts to reduce constraints to early reading and security, and areas for improvement Constraint type Specific constraint Is CARS currently addressing this constraint? Potential midcourse corrections Current constraints to early reading Human resources Lack of capacity among teachers, including poor reading skills and outdated teaching methods that fail to engage students and meet their needs. Yes, but there is room for improvement. CARS has provided much-needed teacher training and follow-up to encourage active teaching-learning methods to boost early reading. However, educators do not appear to have fully incorporated new techniques into their day-to-day activities. In particular, teachers and facilitators do not consistently assess children’s reading and use results to improve common deficiencies or assist at-risk students. • Training and coaching could be modified to allow educators more opportunities to see techniques in action, model those techniques, and receive real-time feedback. • During follow-up visits, CARS could also provide educators with hands-on training on applying, interpreting, and using student reading assessments to inform teaching. • CARS could require coaching visits to occur a minimum of once per month and could track NGOs’ adherence to this requirement. • CARS could also strengthen relationships with teacher training institutions in the region, with the goal of increasing CARS-trained educators’ exposure to experienced educators. Educational approach and materials Lack of reading and teaching materials at school. Yes, but there is room for improvement. CARS has donated a large number of materials to EpCs and school libraries, and has provided a large volume of manipulative materials to EpCs and schools. CARS disseminates materials in a variety of languages to support students’ acquisition of reading concepts in their mother tongue. However, educators reported multiple instances in which reading materials distributed to EpCs and schools were not in children’s mother tongue. • CARS could redistribute materials among schools and EpCs according to children’s mother tongues. • To avoid future mismatches between students’ mother tongue and the language of materials delivered, CARS could conduct an initial consultation with educators in which each facilitator and teacher requests the number or portion of titles in each language, in accordance with their students’ mother tongue(s). . Lack of reading materials in the home. Yes, but there is room for improvement. CARS student workbooks facilitate students’ learning at home with parents. However, some students didn’t receive workbooks until after EpCs and reading programs had started. • CARS could coordinate future student workbook deliveries with the start date of EpCs and formal reading programs in order to maximize potential synergies between reading at home and in-school activities. 77 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Table VI.2. (continued) Constraint type Specific constraint Is CARS currently addressing this constraint? Potential midcourse corrections Stakeholder attitudes and involvement Lack of parental engagement in their child’s development and education. Yes, but there is room for improvement. Parent schools have succeeded in getting some parents more involved in reading and school affairs. However, some NGOs are more successful than others in incentivizing parents to attend parent schools and engage in their children's education. • All NGOs implementing CARS could adopt some emerging best practices with respect to community engagement efforts. This includes holding inclusive small-scale events and festivals at the community level, as well as 'reading club' sessions in which parents or siblings read to students. To boost parent school attendance, NGOs could also structure parent school sessions as conversations and discuss topics that most interest parents. Attendance Poor student attendance due to migration, chores, weather conditions, long commutes, and students’ need for sustenance. No. CARS is unable to affect some students’ attendance issues related to weather or migration, but it could strengthen EpC and school attendance by offering a snack in EpCs or between the end of school and the start of EpCs. • CARS could make efforts to encourage EpC and school attendance, including allowing each educational community to set its EpC hours and offering a snack between primary school and EpC sessions (this may need budget and/or coordination with other agencies currently providing meals in schools). Current constraints to a safe community environment Commutes Students have dangerous commutes due to weather conditions, traffic, and animals they encounter in transit. No. To date, CARS has not invested directly in improving students’ commutes as this is not a direct objective of the program. However, this work could be in the domain of community engagement activities. • As necessary, parents, teachers, and community leaders could engage community members and law enforcement in an effort to enhance children’s safety during their school commute. In particular, law enforcement officials and community members could play a role in guiding children through traffic near school property. Societal problems and crime Schools have experienced break-ins, and students have been robbed on their way to school. Yes, but there is room for improvement. Parent schools and CAPs are designed to help stakeholders mobilize to improve school safety. However, to date, there have been few efforts on the part of parents and educators to engage a broader group of stakeholders on student and school security. • As necessary, parents, teachers, and community leaders could engage municipal authorities and other potential donors to invest in school security, such as fences and walls. In some cases, these types of investments are eligible for CAPs. • Stakeholders could also engage law enforcement to help protect school grounds, particularly when school is in session. 78 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Table VI.2. (continued) Constraint type Specific constraint Is CARS currently addressing this constraint? Potential midcourse corrections Stakeholder attitudes and involvement Parents and community members often fail to recognize bullying as abuse, and act accordingly. Yes, but there is room for improvement. Parent schools are designed to create awareness among parents with respect to bullying and abuse, but to date, CARS communication efforts have not emphasized antibullying messages. • CARS should emphasize antibullying messages in communication products, including radio announcements and flyers. . Parents and community members are often unengaged in school security. Yes, but there is room for improvement. School assessments and CAPs provide an opportunity for parents and community members to organize to improve school security. However, parents and community members are generally unaware of CARS community engagement activities—and some are even disappointed with CAPs, as they see no tangible results of their efforts. • CARS could also restructure the CAP development and approval process to make it shorter and simpler, so as to generate tangible community improvements in a shorter time frame. Financial resources and infrastructure Poor school infrastructure poses a risk to children’s safety. Yes. In the case of private schools, CAPS can fund school improvements. However, public schools are not eligible for infrastructure investments. • No change recommended. Source: Authors’ analysis CARS = Community Action for Reading and Security intervention; NGO = nongovernmental organization; EpC = Espacio para Crecer; CAP = community action plan. 79 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH D. Are CARS activities sustainable? In early 2019, DevTech concludes work and service in the RACCS. Under contract, DevTech is tasked with strengthening and leaving local NGOs that can receive funds; manage funds and activities; and plan, implement, execute, and monitor early reading programs and community security work in the RACCS. Below we assess the potential sustainability of CARS operations in the region in light of four core sustainability criteria: (1) community and educator buy-in and demand, (2) interest and capacity among NGOs, (3) funding sources, and (4) political commitment. (Table VI.3 summarizes our sustainability assessment on these four criteria.) Overall, we find that a lack of funding sources and political commitment does not bode well for future early reading and security activities in the region. In these circumstances, only continued USAID funding would likely enable CARS to continue at its current scale in the region. Table VI.3. CARS sustainability assessment Criterion Discussion Sustainability potential 1. Community and educator buy-in and demand Parents and community leaders fully support CARS reading programs and parent schools, and hope that EpCs and formal reading programs continue into the future. Notably, educators reported fully incorporating QL and APA teaching methods into their daily activities, and feel CARS fills a previous void of resources and technical assistance. Strong 2. Interest and capacity among NGOs All NGOs—and particularly FHR, FQSF, and BICU—have expressed interest to DevTech in continuing CARS activities. However, installed capacity among these NGOs may not be enough to secure funding and lead complex reading and security programs. In fact, these three NGOs had the lowest OCA scores in program and project management, which measure organizations’ ability to comply with donor requests, supervise field staff, conduct quality control, and manage subcontracts. Moderate 3. Funding sources Prospects for public funding outside USAID look scarce, and there is unlikely enough private money to support the current level of CARS programming. MINED and SEAR have governmental and political authority but are lacking financial resources to fund future early reading and security programs in the region. Alternative funding sources post-CARS include municipalities, regional government, local churches, landowners, and local businesses in the RACCS. Two NGOs, FHR and FZT, have demonstrated the most outreach to the private sector and have been successful in establishing friendships and a positive, appreciative relationship with businesses and leaders in Kukra Hill and Corn Island. As a result of the friendship and appreciation, private businesses have openly contributed and supported CARS initiatives and actions in these communities. However, it would be unlikely that these funding sources could support the current scale of CARS programming in the RACCS. Weak 80 VI. CONSTRAINTS ANALYSIS, RECOMMENDED ADJUSTMENTS, AND SUSTAINABILITY ASSESSMENT MATHEMATICA POLICY RESEARCH Criterion Discussion Sustainability potential 4. Political commitment As SEAR and MINED have the fundamental mandate to provide basic education throughout the RACCS, one or both institutions would have to take ownership of CARS (or a CARS-like early reading and security initiative) to secure its continuation in the long-term. CARS has excellent relations with SEAR and MINED, and coordinates with both authorities at the sub-national level. However, it is unclear whether these achievements are sufficient to secure the program’s continuation in the region after USAID funding is exhausted. For example, CARS has made strong inroads in relations with Nicaraguan government officials and MINED in recent years. Notably, USAID officials reported that MINED has begun using CARS educational materials and has incorporated educational priorities supported by CARS into their strategic planning. In addition, SEAR has demonstrated to have strong working relationships with CARS and USAID in substantive ways, particularly with respect to data collection for the EpC impact analysis. There seem to be indications that SEAR is incorporating some of CARS’ methodologies into their work. There is potential for MINED and SEAR to appropriate CARS activities in the future. However, the recent change in the SEAR leadership may require additional USAID investments in terms of working with the new leadership to continue to secure their commitment to early reading programs in the region. Weak Source: Authors’ analysis EpC = Espacio par Crecer; QL = Quantum Learning; APA = Aprendo, Practico, Aplico; NGO = nongovernmental organization; FHR = Fundación Hermanamiento RAMA; FQSF = Fundación Yo Quiero Ser Feliz; BICU = Bluefields Indian and Caribbean University; OCA = Organizational Capacity Assessment; USAID = U.S. Agency for International Development; MINED = Ministry of Education; SEAR = Sistema Educativo Autonómico Regional; RACCS = South Caribbean Coast Autonomous Region; FZT = Fundación Zamora Terán. 81 This page has been left blank for double-sided copying. VII. REFERENCES MATHEMATICA POLICY RESEARCH VII. REFERENCES American Institutes of Research. “Check & Connect: Intervention Report.” 2006. Available at http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=78. Andrabi, T., J. Das, and A. Khwaja. “Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets.” October 2009. Available at http://www.hks.harvard.edu/fs/akhwaja/papers/RC_08Oct09Full.pdf. Bagby, E., S. Glazerman, N. Murray, V. Poggio, and J. Rodriguez. “Impact Evaluation of Espacios para Crecer: Final Evaluation Plan.” Submitted to USAID/Washington. Washington, DC: Mathematica Policy Research, September 2016. Bagby, E., C. Toledo, S. Glazerman, V. Poggio, and N. Murray. “Evaluation of Espacios para Crecer (EpC) in Nicaragua: Baseline Report.” Submitted to USAID/Washington. Washington, DC: Mathematica Policy Research, March 2017. Banerjee, A., R. Banerji, E. Duflo, R. Glennerster, and S. Khemani. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy, vol. 2, 2010, pp. 1–30. Banerjee, A., S. Cole, E. Duflo, and L. Linden. “Remedying Education: Evidence from Two Randomized Experiments in India.” Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1235–1264. Bard, M. “Prevalence of Intimate Partner Violence in Latin American Countries and the Co￾Occurrence of Physical and Inappropriate Discipline.” Monograph, 2012. Barr, A., F. Mugisha, P. Serneels, and A. Zeitlin, “Information and Collective Action in Community-Based Monitoring of Schools: Field and Lab Experimental Evidence from Uganda” August 2012. Available at http://cega.berkeley.edu/assets/cega_events/3/doc7 Zeitlin-et-al_Information-and-Collective-Action-in-Community-based-mMonitoring-of￾Schools.pdf. Accessed August 4, 2015. Barrera-Osorio, F., and L. Linden. “The Use and Misuse of Computers in Education: Evidence from a Randomized Experiment in Colombia.” Policy Research Working Paper Series, No. 4836. Washington, DC: World Bank and International Monetary Fund, February 2009. Available at http://www-wds.worldbank.org/servlet/WDSContentServer/WDSP/ IB/2009/02/11/000158349_20090211111507/Rendered/PDF/WPS4836.pdf. Blattman, C., and L. Ralston. “Generating Employment in Poor and Fragile States: Evidence from Labor Market and Entrepreneurship Programs.” World Bank Manuscript. Washington, DC: World Bank, 2015. Bridgeland, J., M. Bruce, and A. Hariharan. “The Missing Piece: A National Teacher Survey on How Social and Emotional Learning Can Empower Children and Transform Schools: A Report for CASEL.” Civic Enterprises with Peter D. Hart Research Associates. 2013. 83 VII. REFERENCES MATHEMATICA POLICY RESEARCH Bruns, B., D. Filmer, and H. Patrinos. “Making Schools Work: New Evidence on Accountability Reforms.” Washington, DC: World Bank, February 24, 2011. Bruns, B., and J. Luque. “Great Teachers: How to Raise Student Learning in Latin America and the Caribbean.” Washington, DC: World Bank, 2014. Instituto Nacional de Información de Desarrollo Nicaragua “Caracterización Sociodemográfica de la Región Autónoma Atlántico Sur (R.A.A.S.)”, 2005. Available at http://www.inide.gob.ni/censos2005/MONOGRAFIASD/RAAS.pdf. CARS Community Mobilization Study. Submitted August 28, 2014. CARS Quarterly Performance Report: April 1, 2015—June 30, 2015. Submitted July 30, 2015 (2015 Q3) CARS Quarterly Performance Report: October 1, 2015 – December 31, 2015. Submitted January 29, 2016. (2016a) CARS Quarterly Performance Report: January 1, 2016 – March 31, 2016. Submitted April 29, 2016. (2016b) CARS Quarterly Performance Report: April 1, 2016 – June 30, 2016 (Draft). Submitted July 30, 2016. (2016c) CARS indicator worksheet, December 2016. Received March 29, 2017. (2016d) CARS Annual Performance Report: October 1, 205 - September 30, 2016. Submitted October 31, 2016. (2016e) CARS OCA dataset, December 2016. (2016e) CARS Use of Knowledge Products report – FY 2016. Received August 08, 2017. (2016f) Guio, A. F., Chesterfield, R., & Siri, C. “The CERCA School Report Card: Communities Creating Quality Education.” Washington, DC: AED, 2006. Available at http://files.eric.ed.gov/fulltext/ED536777.pdf. Centro de Investigación y Acción Educativa Social (CIASES) and RTI International. “Informe de Resultados: EGRA 2008 [Report on the Results of the EGRA 2008 Pilot Assessment],” 2008. Available at https://www.eddataglobal.org/reading/index.cfm?fuseaction =pubDetail&ID=198. Chambers, B., A. Cheung, and R. Slavin. “Literacy and Language Outcomes of Balanced and Developmental Approaches to Early Childhood Education: A Systematic Review.” Best Evidence Encyclopedia, 2015. Available at http://www.bestevidence.org/word/ early_child_ed_June_19_2015.pdf. 84 VII. REFERENCES MATHEMATICA POLICY RESEARCH Chesterfield, Ray, and Adriana Abreu-Combs. “Centers for Excellence in Teacher Training (CETT): Two-Year Impact Study Report (2008-2010).” Washington, DC: U.S. Agency for International Development, January 2011. Available at http://pdf.usaid.gov/pdf_docs/PDACS248.pdf. Accessed July 9, 2012. Comings, J. “Evidence-Based Reading Projects.” Version I. Washington, DC: U.S. Agency for International Development, December 2012. Creative Associates International and Mathematica Policy Research. “Findings from the Four-Country School Dropout Prevention Pilot Program Impact Evaluation.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2015. Crouch, L., M. Korda, and D. Mumo. “Improvements in Reading Skills in Kenya: An Experiment in the Malinda District.” USAID, April, 2009. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=154. DevTech, USAID/Nicaragua Community Action for Reading and Security (CARS) Technical Proposal, 2013. Duflo, E., P. Dupas, and M. Kremer. “School Governance, Teacher Incentives, and Pupil￾Teacher Ratios: Experimental Evidence from Kenyan Primary Schools.” Journal of Public Economics, vol. 123, 2015, pp. 92–110. EDUQEMOS. "Educación Intercultural Bilingüe con Calidad y Equidad para la Autonomía. Informe de Progreso Educativo.” Informe del Foro Educativo Nicaragüense EDUQUEMOS, May 2015. Available at https://www.fundaciontelefonica.com/arte_cultura/publicaciones￾listado/pagina-item-publicaciones/itempubli/457/. Entwisle, D., and K. Alexander. “Facilitating the Transition to First Grade: The Nature of Transition and Research on Factors Affecting It.” The Elementary School Journal, vol. 98, no. 4, 1998, pp. 351–364. Esbensen, F.-A., and D. W. Osgood. “Gang Resistance Education and Training (GREAT): Results from the National Evaluation.” Journal of Research in Crime and Delinquency, vol. 36, no. 2, 1999. pp. 194–225. Esbensen, F.-A., D. Peterson, T. J. Taylor, and A. Freng. “Similarities and Differences in Risk Factors for Violent Offending and Gang Membership.” Australian & New Zealand Journal of Criminology, vol. 42, no. 3, 2009, pp. 310–335. Flotts, M., J. Manzi, D. Jimenez, A. Abarzua, C. Cayuman, and M. Garcia. “Informe de Resultados TERCE. Logros de Aprendizaje.” UNESCO, July 2015. Fundación Telefónica. “Educación Intercultural Bilingüe con Calidad y Equidad para la Autonomía,” May 2015. 85 VII. REFERENCES MATHEMATICA POLICY RESEARCH Gertler, P., H. Patrinos, and E. Rodriguez-Oreggia. “Impact Evaluation of Parental Empowerment Program in Mexico: Randomized Experiment of the ‘Apoyo a La Gestión Escolar (Age)’ in Rural Primary School: Preliminary Findings.” Washington, DC: World Bank Research Committee, June 2010. Glewwe, P., M. Kremer, and S. Moulin. “Many Children Left Behind? Textbooks and Test Scores in Kenya.” American Economic Journal: Applied Economics, vol. 1, no. 1, January 2009, pp. 112–135. Glewwe, P., E. Hanushek, S. Humpage, and R. Ravina. “School Resources and Educational Outcomes in Developing Countries: A Review of the Literature from 1990 to 2010.” NBER Working Paper No. 17554, 2010. Gove, A. and A. Wetterberg. “The Early Grade Reading Assessment: Applications and Interventions to Improve Basic Literacy.” Research Triangle Park, North Carolina: RTI International, 2011. Available at http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf. Harter, S. “The Perceived Competence Scale for Children.” Child Development, vol. 53, no. 1, 1982, pp. 87–97. Hennigan, K. M., C. L. Maxson, D. C. Sloane, K. A. Kolnick, and F. Vindel. “Identifying High￾Risk Youth for Secondary Gang Prevention. Journal of Crime and Justice, vol. 37, no. 1, 2014, pp. 104–128. Huang, D., K. Kim, J. Cho, A. Marshall, and P. Perez. “Keeping Kids in School: A Study Examining the Long-Term Impact of Afterschool Enrichment Programs on Students’ High School Dropout Rates.” Journal of Contemporary Issues in Education, vol. 6, no. 1, 2011, pp. 4–23. Available at http://socialiststudies.com/index.php/JCIE/article/view/11745. INIDE. “Mapa de Pobreza Extrema Municipal por el Método de Necesidades Básicas Insatisfechas (NBI).” Available at http://www.inide.gob.ni/censos2005/cifrasmun/mappobrezad.pdf. INIDE. “Anuario Estadístico 2008.” Available at http://www.inide.gob.ni/Anuarios/Anuario2008/ModuloII-Social/ModuloII_SeccionII.6.pdf Jackson, R., J. Burghardt, A. McCoy, C. Pistorino, A. Wilkinson, M. Clark, C. Ross, P. Schochet, and P. Swank. “National Evaluation of Early Reading First: Final Report.” (NCEE 2007-4007). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, May 2007. Available at https://www.mathematica-mpr.com/download￾media?MediaItemId={377DCACE-CF85-42A5-91FA-0797F416F9E8} Jamison, D., B. Searle, K. Glada, and S. Heyneman. “Improving Elementary Mathematics Education in Nicaragua: An Experimental Study of the Impact of Textbooks and Radio on Achievement.” Journal of Educational Psychology, vol. 73, no. 4, August 1981, pp. 556– 567. 86 VII. REFERENCES MATHEMATICA POLICY RESEARCH Jimenez, E., and Y. Sawada. “Do Community-Managed Schools Work? An Evaluation of El Salvador’s EDUCO Program.” World Bank Economic Review, vol. 13, no. 3, 1999, pp. 415–441. Korda, M. and B. Piper. “EGRA Plus: Liberia—Final Program Evaluation Report.” USAID, January 2011. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=283. Levy, D., M. Sloan, L. Linden, and K. Harounan. “Impact Evaluation of Burkina Faso’s BRIGHT program. Final Report.” Washington, DC: Mathematica Policy Research, submitted to Millennium Challenge Corporation, 2009. Linden, L. L., C. Herrera, and J. Grossman. “Achieving Academic Success After School: A Randomized Evaluation of the Higher Achievement Program.” Cambridge, MA: Abdul Latif Jameel Poverty Action Lab, 2011. Lipsey, M., D. Farran, and K. Hofer. “A Randomized Control Trial of a Statewide Voluntary Prekindergarten Program on Children’s Skills and Behaviors Through Third Grade.” Peabody Research Institute, Vanderbilt University, 2015. Available at: https://my.vanderbilt.edu/tnprekevaluation/files/2013/10/VPKthrough3rd_final_ withcover.pdf. Magna Children at Risk. “Nicaragua Report for Year 2012.” Maldonado, A., D. Orces, and E. Saunders. “Final Baseline Report on Citizen Security in the Southern Caribbean Coast in Nicaragua.” Vanderbilt University, March 2015. McCormick, M., E. O’Connor, E. Cappella, and S. McClowry. “Teacher–Child Relationships and Academic Achievement: A Multilevel Propensity Score Model Approach.” Journal of School Psychology, vol. 51, no. 5, October 2013, pp. 611–624. Messner, L., S. Morel-Seytoux, K. Spratt, and A. Ladd. “Beyond Access: Toolkit for Integrating Gender-Based Violence Prevention and Response into Education Projects.” Rockville, MD: USAID’s Advancing the Agenda of Gender Equality (ADVANTAGE), Task Order 3, 2015. Available at https://www.usaid.gov/sites/default/files/documents/1865/USAID_ADVANTAGE_GBV_E ducation_Toolkit-Final.pdf. Ministerio de Educacion (2008). Censo Escolar 2007: Informes de los Departamentos y Regiones Auutonomas. Division General de Planificacion, Division Estadistica. Ministerio de Educacion. Managua, Nicaragua, May 2008. Morrow, L. M., and C. S. Weinstein. “Increasing Children’s Use of Literature Through Program and Physical Design Changes.” The Elementary School Journal, vol. 83, 1982, pp. 131–137. Mullis, I., M. Martin, P. Foy, and K. Drucker. Progress in International Reading Study (PIRLS) 2011 International Results in Reading. Chestnut Hill, MA: TIMSS and PIRLS International Study Center, Boston College, 2012. 87 VII. REFERENCES MATHEMATICA POLICY RESEARCH Murray, N., S. Glazerman, and E. Bagby, “Draft Design for the Impact Evaluation of Espacios para Crecer in Nicaragua.” PowerPoint presentation. Washington, DC: Mathematica Policy Research, 2014. Nguyen, T., and G. Lassibille. “Improving Management in Education: Evidence from a Randomized Experiment in Madagascar.” World Bank, 2008. Osher, D., D. L. Kelly, N. Tolani-Brown, L. Shors, and C.-S. Chen. “UNICEF Child Friendly Schools Programming: Global Evaluation Final Report.” Washington, DC: American Institutes for Research, 2009. Piper, B. “Impact Study of SMRS Using Early Grade Reading Assessment in South Africa.” USAID, July 2009. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=195. Pradhan, M., D. Suryadarma, A. Beatty, M. Wong, A. Gaduh, A. Alisjahbana, and R. P. Artha. “Improving Educational Quality Through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia.” American Economic Journal: Applied Economics, vol. 6, no. 2, 2014, pp. 105–126. Prinz, R. J., S. Foster, R. N. Kent, and K. D. O’Leary. Multivariate Assessment of Conflict in Distressed and Nondistressed Mother‐Adolescent Dyads. Journal of Applied Behavior Analysis, vol. 12, no. 4, 1979, pp. 691–700. Ramirez, S. “Violence and Crime in Nicaragua: A Country Profile.” Washington, DC: Inter￾American Development Bank, Office of Evaluation and Oversight. Discussion paper No. IDB-DP-306, 2013. RTI International. “Literature Review on the Intersection of Safe Learning Environments and Educational Achievement.” Washington, DC: U.S. Agency for International Development, 2013. Available at http://www.ungei.org/resources/files/Safe_Learning_and_ Achievement_FINAL.pdf. Accessed July 22, 2015. The Dialogue. “Building Quality Education: A Pact with the Future of Latin America.” Buenos Aires: Fundación Santillana, 2016. UNICEF. "At a glance: Nicaragua." Available at https://www.unicef.org/infobycountry/nicaragua_statistics.html. Accessed 04/10/2017 UNODC. “Crime and Development in Latin America: Caught in the Crossfire.” Vienna, Austria: United Nations Office on Drugs and Crime, 2007. Available at https://www.unodc.org/pdf/research/Central_America_Study_2007.pdf. USAID. “Performance Monitoring and Evaluation TIPS: Measuring Institutional Capacity.” Washington, DC: U.S. Agency for International Development, 2011. Available at http://pdf.usaid.gov/pdf_docs/Pnadw115.pdf. 88 VII. REFERENCES MATHEMATICA POLICY RESEARCH USAID. “USAID Education Strategy 2011–2015.” Washington, DC: U.S. Agency for International Development, 2011. Available at http://pdf.usaid.gov/pdf_docs/Pdacq946.pdf. USAID/Nicaragua. “Country and Development Cooperation Strategy, FY 2013–FY 2017.” Washington, DC: U.S. Agency for International Development, 2013. Available at https://www.usaid.gov/sites/default/files/documents/1862/USAID%20NICARAGUA%20FI NAL%20PUBLIC%203.11.14.pdf. World Bank. “Project Performance Assessment Report—Nicaragua Education Project.” Report No. 97131 Washington, DC: World Bank, June 12, 2015. Available at http://ieg.worldbank.org/Data/reports/Nicaragua_Education_PPAR.pdf. 89 This page has been left blank for double-sided copying. APPENDIX A CHARACTERISTICS AND OUTCOMES OF STUDENTS ASSIGNED TO ATTEND COHORT 1 EpCs This page has been left blank for double-sided copying. APPENDIX A MATHEMATICA POLICY RESEARCH This appendix presents a description of a sample of children assigned to attend EpCs as part of the impact evaluation of the CARS program’s EpC component. The evaluation was designed to determine the impact of EpCs on children’s educational and socioemotional outcomes, among other outcomes. The research design for the impact evaluation designated two groups of children: children randomly assigned to attend EpCs (the treatment group) and children randomly assigned to not attend EpCs (the control group). The impact of EpCs on children’s outcomes is defined as the difference between treatment and control groups after students attended EpCs, or at follow-up. The EpC impact evaluation includes household interview and child assessment data for two cohorts of EpCs: Cohort 1 EpCs, implemented from approximately 2014 to 2016, and Cohort 2 EpCs, implemented from approximately 2015 to 2017. Because follow-up data are not yet available for Cohort 2 treatment and control groups, we cannot measure and report the impact of EpCs at this point. In this appendix, we present early results for 652 children randomly assigned to attend Cohort 1 EpCs only (the Cohort 1 treatment group). These children and their families were surveyed in mid-2016, at least 18 months after Cohort 1 EpCs started, as part of follow-up data collection.23 These descriptive findings provide a better understanding of children’s educational and socioemotional outcomes after they attended Cohort 1 EpCs. However, these findings do not offer any insight into the impact of EpCs on children’s outcomes; those impacts (covering Cohorts 1 and 2) will be presented in a future report. A nontrivial portion of children assigned to EpCs attend school in a language other than their mother tongue. Although the official language of instruction in 80 percent of schools participating in CARS is Spanish, only 60 percent of children assigned to Cohort 1 EpCs reported Spanish as their mother tongue. Notably, 20 percent of children assigned to Cohort 1 EpCs speak English or Creole as their mother tongue, 14 percent speak Miskitu, and 2 percent speak Ulwa. As a result, a sizable portion of students in the sample whose mother tongues are English, Creole, or Miskitu are in an educational community linked to a school whose language of instruction is Spanish: 38 percent of English or Creole speakers, and 13 percent of Miskitu speakers. EpC attendance among children in the treatment group is below 50 percent. Only 43 percent of parents of children assigned to attend Cohort 1B EpCs reported that their children were currently attending EpCs, despite that Cohort 1B EpCs were in session at the time of data collection. This relatively low attendance rate reflects a lack of participation from two different groups: children assigned to EpCs who did not attend even a single EpC session, as well as children who initially attended EpCs but had dropped out by the time of data collection. Most children assigned to EpCs were enrolled in school at follow-up, even if they weren’t enrolled before EpCs started. More than 90 percent of children assigned to Cohort 1 EpCs were enrolled in school 18 months after EpCs began. In addition, most students who were not initially enrolled in school before EpCs started were enrolled in school 18 months after EpCs began. It is unclear if this reflects a positive impact of EpCs on school reintegration, or if it reflects a natural tendency of students who drop out of school to re-enroll in subsequent years (or 23 Cohort 1A EpCs started in May 2014 and concluded in November 2015. Cohort 1B EpCs began in November 2014 and wrapped up in November 2016. Data collection was done in May and June 2016. A.3 APPENDIX A MATHEMATICA POLICY RESEARCH both). We will be able to better explore whether there are any differences across treatment groups in the impact evaluation report. Children assigned to EpCs report that they are practicing reading at home, but 2nd graders in particular have serious reading deficiencies. Seventy-six percent of children assigned to Cohort 1 EpCs reported that they read at home, and 68 percent reported that someone reads to them at home (Table A.1). However, around half of 2nd graders assigned to EpCs could not decode or read a single word from a list of words in the reading assessment we administered 18 months after EpCs began (Figure A.1). Students in 3rd and 4th grade fared better: three￾fourths could decode or read at least one word and correctly answer at least one of several reading comprehension questions about what they read. Figure A.1. Reading skills by grade 52% 81% 90% 97% 52% 81% 90% 97% 49% 78% 88% 95% 36% 67% 77% 89% 0% 20% 40% 60% 80% 100% 2 3 4 5 Grade Decoded at least one word Reading fluency : at least one word Reading comprehension (passage 1): at least one correct question Reading comprehension (passage 2): at least one correct question Girls and boys have comparable reading outcomes at follow-up. Eighteen months after EpCs began, girls assigned to Cohort 1 EpCs were slightly more likely than boys to decode and read at least one word from a list of words, but reading fluency and reading comprehension were largely comparable between boys and girls (Table A.1). Whether or not EpCs affected boys’ or girls’ reading outcomes differently will be explored in more depth in the forthcoming EpC impact evaluation. Children enrolled in school before EpCs started have better fluency and comprehension than those who weren’t enrolled. Children enrolled in school when they were selected to attend EpCs were more likely to be able to decode words, read any words, and correctly answer comprehension questions at follow-up than children who were not enrolled in school before EpCs began (Table A.1). This finding suggests that time out of school has a quantifiable detrimental effect on students’ reading outcomes. However, because there were relatively few children in our sample who were not enrolled in school before EpCs began, these results should be interpreted with caution. A.4 APPENDIX A MATHEMATICA POLICY RESEARCH Children assigned to EpCs have relatively high levels of self-esteem. We assessed children’s self-esteem using five items adapted from the Global Self-worth sub-scale in Harter’s Self-perception Profile for Children (Harter 1982). Each item asks the child to decide which kind of kids he or she is most like: kids who feel good/happy with themselves or kids who do not feel good/happy with themselves. Children get a score of 1 if they identify with kids who feel good with themselves, and 0 if they identify with kids who don’t feel good about themselves. We obtained a total score by computing the sum across the five items. The average score was 3.8 (standard deviation [SD] = 1.1), indicating that the average child in our sample had a relatively high level of self-esteem (Table A.1). Children assigned to EpCs have relatively healthy levels of academic engagement and perceptions of support from adults. We measured academic engagement (or enthusiasm) and children’s perceptions of support from adults in their community using five items adapted from the Conditions for Learning Survey (Osher et al. 2009). Children were asked to rate on a scale from 0 (“not at all true”) to 3 (“very true”) whether they were interested in going to school, completing secondary school, and whether adults in the community encourage them to take school seriously and believe that all children can learn and do well in life. On average, children indicated that the statements were “somewhat true” or “very true” for themselves (around 2.5 and 2.4 for academic engagement and perceptions of support from adults, respectively; SD = 0.5 and 0.7, respectively), which corresponds to relatively high levels of self-reported academic engagement and positive perceptions of community support for education (Table A.1). Children assigned to EpCs have relatively strong relationship skills (or social competence). We used fourteen items adapted from the Conflict Behavior Questionnaire (Prinz et al. 1979) to gauge children’s relationship skills, a dimension of social competence that encompasses listening, self-regulation, and communication skills that are fundamental to establishing and maintaining healthy and rewarding relationships (Bridgeland et al. 2013). Primary caregivers indicated how often “it is easy to get along with” their child and their child “listens to what they tell him/her,” and so on, using a scale from 0 to 3 (0 “never,” 1 “a few times,” 2 “sometimes,” and 3 “always”). The average score was 2 (SD = 0.4), indicating that the average child in our sample is able to establish and maintain a healthy and rewarding relationship with his or her primary caregiver (Table A.1). Children assigned to EpCs don’t describe their communities as safe, but they generally feel safe in and around school. We asked children to rate how safe they feel engaging in various activities in their community (such as walking home) on a scale from 0 (“very safe”) to 3 (“very unsafe”). The average sense of unsafety in the sample was somewhat high at 1.7 (SD = 0.8; see Table A.1). This corresponds to around half of children reporting that their community is “a little unsafe” or “very unsafe,” and that they feel “a little unsafe” or “very unsafe” walking alone at night. We also asked children if they felt safe at school or on their way to school, using two items adapted from the Conditions for Learning Survey (Osher et al., 2009). In contrast to their perceptions of unsafety in the community, few children (14 percent) reported a sense of unsafety while at school and on their way to school. This could be explained, in part, because about half of the children in the sample (52 percent) report that they go to school with someone else (SD = 0.5). Additionally, students reported relatively low exposure to bullying at school (with an average of 0.7 on a scale of 0 to 3, with 3 being the most exposure to bullying). A.5 APPENDIX A MATHEMATICA POLICY RESEARCH Children assigned to EpCs report relatively low levels of impulsive risk taking and moral disengagement. We adapted four items from the Gang Risk of Entry Factors (Hennigan et al. 2014) to gauge children’s impulsive risk taking, defined as the tendency to engage in risky behaviors without considering the consequences (Esbensen et al. 2009). Children were asked to rate how true a series of statements were for themselves using a scale from 0 to 3 (0 “not at all true,” 1 “not very true,” 2 “somewhat true,” or 3 “very true”). Statements included whether they “like to do something dangerous just for the fun of it,” and “do things without stopping to think it they will get in trouble for it.” On average, children reported that the statements were “not very true” for themselves, suggesting low levels of impulsive risk taking attitudes (mean of 1.2; SD = 0.7; see Table A.1) Similarly, we adapted eight items from the Neutralization sub-scale in the Gang Risk of Entry Factors to assess children’s moral disengagement, or the tendency to hold beliefs that justify engaging in unlawful or unethical behavior in the presence of extenuating factors (Esbensen and Osgood 1999). Using a scale from 0 to 3 (0 “disagree a lot,” 1 “disagree a little,” 2 “agree a little,” or 3 “agree a lot”), children rated how much they agree that “it is okay not to tell the truth to keep friends from getting into trouble,” “hit people if they hit you first,” “break some rules to be popular with your friends,” and so on. The average child in our sample reported “disagreeing somewhat” with beliefs that justify engaging in unlawful or unethical behavior, including bullying (mean of 1.2; SD = 0.6). A.6 APPENDIX A MATHEMATICA POLICY RESEARCH Table A.1. Outcomes of children assigned to participate in Cohort 1 EpCs, by gender, and initial school enrollment . All Female Male Enrolled at recruitment Yes No Work (Any activity that is not play or educational) 86.7% 86.6% 86.7% 86.6% 87.2% Enrollment (at follow up) 92.6% 94.2% 91.0% 94.2% 72.3% School attendance (% of school days present in last month) 85.3% 87.5% 83.1% 86.8% 65.1% Grade progression (indicator for at least one year further than at recruitment) 93.4% 96.0% 90.7% 93.1% 97.9% Reads at home 76.5% 79.8% 73.1% 76.5% 76.1% Read to by someone at home 68.4% 68.5% 68.2% 67.8% 76.1% Decoded at least one word 76.2% 77.7% 74.7% 78.2% 51.1% Reading fluency: at least one word (All languages) 76.1% 77.7% 74.4% 78.0% 51.1% Reading comprehension (passage 1): at least one correct question 73.5% 74.4% 72.5% 75.4% 48.9% Reading comprehension (passage 2): at least one correct question 64.1% 64.9% 63.3% 65.5% 46.8% Self-esteem/self-worth (range: 0-5) 3.8 3.9 3.8 3.8 3.8 Adult support (range: 0-3) 2.2 2.2 2.2 2.2 2.1 Academic engagement (range: 0-3) 2.5 2.5 2.5 2.6 2.4 Community support for education (range: 0-3) 2.4 2.4 2.4 2.4 2.2 Social competence (range: 0-3) 2.0 2.1 2.0 2.0 2.1 Sense of unsafety in the community (range: 0-3) 1.7 1.7 1.7 1.7 1.5 Sense of unsafety in school and going to school 14.5% 15.8% 13.1% 14.4% 15.1% Exposure to violence in school - bullying (range: 0-3) 0.7 0.7 0.8 0.7 0.6 Impulsive risk taking (range: 0-3) 1.2 1.2 1.2 1.2 1.2 Moral disengagement (range: 0-3) 1.2 1.2 1.3 1.2 1.3 Attitudes towards delinquency (range: 0-3) 0.9 0.9 0.9 0.9 1.1 Number of children 652 328 324 605 47 Note: Children shown in this table are from 87 communities with at least one EpC in Cohort 2, in Kukra Hill, Laguna de Perlas, and Bluefields. A.7 This page has been left blank for double-sided copying. APPENDIX B STAKEHOLDER PERCEPTIONS ON CARS COMPONENTS This page has been left blank for double-sided copying. APPENDIX B MATHEMATICA POLICY RESEARCH A. Reading programs Teachers, principals and facilitators recalled that training focused on active learning techniques, gender, and moral values. When asked what the CARS training involved, educators often said it discussed how children learn and covered new teaching techniques designed to directly engage students in learning to read—with an emphasis on engaging all students in class—either through games, hands-on practice, or group work. Most facilitators recalled that training revolved around QL, and some facilitators recalled some training on parent schools or reading assessments. Teachers and principals often said that training involved getting kids to work in groups, creating an interesting classroom environment and using materials to promote learning, and using manipulative materials to complement reading activities. Several teachers recalled that training involved APA, and they could explain some of the basic principles of the methodology. In addition, most trainees recalled discussions and themes with respect to gender and morals. Common themes raised by educators about gender is that there is a difference between sex and gender, and that boys and girls should be treated no differently in class with respect to reading, chores, and play activities. A common themes recalled by educators is that morals should be taught by parents in the home but that educators have a role in reinforcing ethical behavior in the classroom. Educators praised training and trainers, but CARS and USAID representatives—and some educators—thought training should have been more extensive. The vast majority of teachers and facilitators had positive opinions of CARS training and trainers (Figure A.1). The only result that wasn’t resoundingly positive was the fact that fewer than 70 percent of teachers agreed that training took place at a convenient time. In some select instances, teachers noted that trainings conducted during daytime hours interfered with their teaching responsibilities, they had little advance notice of training, and reimbursements were often insufficient to cover transportation expenses. In addition, several stakeholders mentioned that training was not intensive or comprehensive enough. Several NGO representatives and some teachers noted that additional 1st through 3rd grade and diploma training would have been helpful. A DevTech representative made a similar point about EpC training and expressed the fear that some facilitators and teachers are still not comfortable with APA and QL educational approaches after only one or two training sessions. A few educators expressed similar opinions, namely that they would have liked more opportunities to see new techniques in action, over and above basic exposure to the techniques during training. “For me in the 16 years that I’ve been teaching, for me these are the best trainings I’ve had … the trainings have opened my eyes and mind to use what I’ve got, in addition to what they’ve given us—but also invent new activities that will help me strengthen children’s learning.” —CARS-trained teacher B.3 APPENDIX B MATHEMATICA POLICY RESEARCH Figure B.1. Percentage of educators that agree that CARS training: 100 100 100 86 100 97 97 69 100 100 91 100 0 20 40 60 80 100 Gave useful information or techniques Gave new information or techniques Took place in a convenient location Was scheduled at a convenient time Percentage Principals Teachers Facilitators Source: In-person interviews with CARS-trained educators, including 35 principals, 54 teachers, and 26 facilitators in September and October 2016. NGOs and teachers alike praised QL and APA methodologies. NGO representatives emphasized the relative strength of the CARS educational approach, compared to traditional teaching techniques commonly used in the region. According to these representatives, QL and APA engage, encourage, and motivate students, whereas traditional techniques treat them as passive participants. Educators made similar points, noting that when they use these active techniques of “learning by doing” and “learning while playing,” their students immediately respond with interest. NGOs also noted that the CARS educational approach is very attractive and practical for teachers because it is tailored to each grade’s needs and desired skills, it features highly structured and well-designed lesson plans, and consolidated EpC modules can be found in one simple guide. Educators corroborated NGOs’ statements in different terms, often praising the lesson plans and guides as easy to understand and clear in their instructions. Some parents and leaders appreciate that kids are learning in EpCs, but other parents and leaders see it as play without much educational benefit. In focus groups conducted in 36 participating communities, several parents and community leaders noted that that they value the role that EpCs play in supporting students who struggle academically or behaviorally. According to parents and leaders, EpCs offer support that teachers and parents may be unable to provide due to competing demands, such as having too many students in the classroom, being busy at work, or lacking the knowledge necessary to assist children’s learning. However, parents and leaders in some communities had a vague or imprecise idea about the goals and activities of B.4 APPENDIX B MATHEMATICA POLICY RESEARCH EpCs. In some cases, they mentioned the materials, games, and songs, but did not seem to understand the activities’ learning goals or the connection between play and learning. Some parents and leaders even expressed the sentiment that children attending the EpC should spend more time learning, as opposed to playing. Educators genuinely appreciate CARS visits— particularly educators at rural schools. Teachers and principals expressed a general sentiment that they felt encouraged and motivated by NGO support and that this support seemed to fill a large need that had gone unmet in previous years. One teacher really appreciated visits from CARS because NGO staff help teachers motivate parents to get involved in school and their children’s reading. In this sense, the teacher believed CARS was a key ally in getting parents’ support. B. Community engagement efforts Parents who attended parent schools expressed strong support for them. Parents appreciated that parent schools covered topics they helped to choose. Parents also noted that they valued learning about potential underlying reasons for their children’s behavior, as well about strategies for behavior management, rather than defaulting to negative beliefs about their children. Parent schools offered opportunities for parents to share and learn from the opinions of others, as well as participate in good discussions and engaging activities. NGOs feared that the lengthy CAP development and approval timelines may have resulted in lost community goodwill. In quarterly reports, DevTech noted that due to initial misunderstandings of what CAPs could finance, some NGOs committed to infrastructure improvements in public schools, which were not eligible for funding. The NGOs thus lost some credibility with communities as a result of the mistake. In addition, CAPs currently take between two and three months to complete and involve several steps, including (1) preparation of the registry and school report, (2) creation of the analysis committee, (3) documentation of data in order to prepare the CAP, and (4) review of content of the proposed CAP. Once they are submitted, CAPs must be reviewed by DevTech—usually requiring some revisions—and then reviewed by multiple USAID representatives. Moreover, schools and parents often request infrastructure investments, which require an additional environmental assessment. One NGO (FZT) remarked that the multiple steps to producing CAPs require several weeks and place a large burden on their staff. What’s more, the NGOs and communities would often wait several months for DevTech and USAID revisions and approval. This long process has resulted in negative repercussions for NGOs, who have lost face with community members over long delays in the submission, review, and approval process. NGOs and DevTech noted that CAPs may define community needs and goals too narrowly. One NGO noted that the schools in which they worked had multiple needs and “In these communities we’re off the beaten path, so they forget a bit about us … so I feel supported that [CARS] comes and they visit and Help us. They’re thinking of us.” —Principal “It’s already been one year and six months and they’ve only approved two PACs. We see people from the community every day and they ask us, ‘What’s happening with that little project we had planned?’ and we tell them we’re still waiting.” —NGO representative B.5 APPENDIX B MATHEMATICA POLICY RESEARCH priorities that would require multiple funding sources, and that defining CAPs around a specific school investment actually loses sight of the big picture of schools’ full set of needs. Furthermore, certain types of investments in public schools are not eligible for USAID funding, and thus require alternate funding sources. A DevTech representative made a similar point and suggested CAPs be refocused to define school and community priorities more generally, and make USAID funding just one potential funding option among many options community members can pursue simultaneously; these other options could include municipal authorities and businesses, as well as in-kind labor from the community. Parents and community leaders expressed satisfaction with early reading events— particularly International Book Day—and suggested more frequent events of this kind. During CARS events, children dance, play games, interpret stories aloud in discussions, and socialize and get to know children from other schools. Parents noted that these events are enjoyable for children while at the same time stimulating children’s interest in reading. As a result of these types of events, parents noted that their children are motivated to continue participating in the program and have a desire to take care of their books. Parents and leaders suggested making the events more frequent, disseminating information about the events through means other than radio announcements, such as flyers and street advertisements/signs, and taking into account children’s comfort during the events (for example, plan the presence of sheltered areas during events so that children are not exposed to the sun all day). Stakeholders noted that some invitation-only CARS events have some negative consequences. Stakeholders noted that the only negative consequence of CARS events and celebrations is jealousy among those not invited. One NGO representative noted that they are allowed to invite only 10 children, 5 parents, and 5 teachers from each community to some special events in select locations, and these limits end up creating tension within the community over who is invited and who isn’t. The NGO representative noted that an alternative to these invitation-only events (that take place outside of communities) would be to hold more inclusive, but modest events in each community. C. Knowledge generation Stakeholders expressed different perspectives on the goals and objectives of knowledge generation. A USAID representative noted that the component was supposed to entail substantive collaboration between DevTech and NGOs on producing, sharing, and interpreting data throughout the course of the implementation period, and making midcourse corrections in response to that data. A DevTech representative presented an alternate perspective: that the knowledge management component of sharing data with decision makers was not yet possible because CARS did not yet have information to share. This discrepancy between USAID and DevTech perspectives on knowledge sharing reflects a lack of clarity between the two parties on the component’s objectives and primary activities. “What’s needed is more collaboration between CARS and subgrantees. We don’t want it to be centered on a single person—for example, an M&E person— because there hasn’t been enough sharing of what is learned; there is no feedback [to subgrantees] … CARS needs to open itself to its subgrantees.” —USAID B.6 APPENDIX B MATHEMATICA POLICY RESEARCH NGOs complained of a lack of substantive assistance from DevTech. One NGO noted that had not received the external help promised under their action plan. Another NGO observed that technical assistance resembles auditing, in that DevTech technical staff are more likely to “check up” on them than work collaboratively with them to solve problems. Another NGO representative noted that the DevTech liaison in charge of giving them technical assistance got sick and simply “disappeared,” leaving them without support for an extended period. D. Management Bureaucracy and the concentration of decision making have slowed review and approval of materials and plans. NGOs and DevTech remarked at the prolonged length of time it took USAID to review and approve didactic materials, CAPs, and newsletters, noting that this review often created bottlenecks to distributing materials on time and disbursing funds for much￾needed community projects. Personnel changes and capacity limitations hinder implementation. Personnel changes at DevTech have caused inconsistent guidance, delays, and anxiety among staff. The first chief of party resigned in 2014, and an education specialist was brought in to begin EpC implementation. In late July 2014 a new chief of party arrived in Bluefields and assumed leadership of the team. In the next two years, CARS would lose formal and nonformal education specialists, a community engagement specialist, an administrative and finance specialist, and an M&E data manager. A former DevTech employee estimated that 11 members of the core DevTech team had quit as of mid-2016. The resignations have created leadership gaps, capacity constraints, and implementation delays, particularly when key staff cannot be replaced in a timely manner. Even as of late 2016, DevTech staff reported being critically understaffed given their workload. Unfortunately, NGOs had similarly high staff turnover and limited capacity. A DevTech employee estimated that 12 key staff from NGOs had resigned since the start of the project, and noted that NGOs have general weaknesses in administrative capacity and financial management; these further complicate CARS implementation. Overall, stakeholders disagreed about the reasons for high staff turnover within CARS, with reasons ranging from relatively low salaries to frustration with program leadership. “For me technical assistance is being right by your side … and saying we have this activity, what do you propose? How can we do it together? And if not, at least listen to you … more than a regulator that says did you comply or not comply with the activity.” —NGO representative “I feel supported but I don’t feel like I can feasibly complete the work I’m tasked within a normal work week. Whether that means I haven’t been sufficiently trained or being asked to do too much, I’m not sure.” —DevTech representative B.7 APPENDIX B MATHEMATICA POLICY RESEARCH NGOs gave mixed reviews of DevTech coordination and communication. Some DevTech staff try to set expectations and be aware of the burden of their requests of NGOs. They have also attempted to be consistent in their dealings with NGOs in the areas of finance and accounting, such that they give no preferential treatment to any NGO. However, NGOs complain that they often get uncoordinated requests from DevTech’s many units, which they cannot cover at once. One NGO also noted that DevTech staff have found issues with their deliverables and failed to notify them of the issue until several months later. In addition, a DevTech employee noted that DevTech often failed to plan events ahead of time with NGOs. NGOs agreed, saying they were often forced to reschedule events at the last minute due to a lack of communication from DevTech about upcoming activities. NGOs expressed frustration with large service delivery targets, multiple program components, and uncoordinated requests. CARS set large targets for the number of students, schools, and communities served, as well as the number of EpCs established. In addition, the scope of work in each community was extensive, in that CARS was expected to implement reading programs and community engagement programs concurrently, in addition to organizing events and collecting necessary data. Two NGOs in particular had large burdens of establishing more than 100 EpCs each, and one of these NGOs had the additional burden of working in communities with a high level of civil unrest. Due in part to their large portfolio of EpCs, these NGOs had difficulty providing an adequate level of coaching and follow-up to facilitators and meeting the goals outlined in their work plan. The EpC impact evaluation spread NGO staff even thinner, as they were tasked with identifying and visiting a larger number of eligible communities for EpCs than would be served under CARS. DevTech staff noted a lack of control over NGOs as a result of CARS’s grant structure. In interviews, DevTech staff expressed frustration with NGO performance—particularly regarding some NGOs’ failure to meet their work plan goals and overarching capacity constraints. One DevTech representative noted that the CARS program’s grant mechanism didn’t allow them enough influence over NGOs as would be available through a contract. Another DevTech representative noted benefits and drawbacks to issuing grants to NGOs versus the alternative of contracting with private firms: whereas grant agreements help foster local NGO development and autonomy in the long term, it is often at the expense of timely implementation in the short term. “One of my roles is that I see emails and what we’re asking them to do, and I always put myself in their shoes to understand from the DevTech side on how to be less overwhelming to these people and ask for things in a controlled and consistent way.” —DevTech representative “Each one of these [DevTech] offices in every moment are asking for things, demanding monitoring and evaluation, statistical information, completed instruments … that takes a ton of time.” —NGO representatives “If you go with contractors you get results. If you go with grants and NGOs it will take longer and it is not as easy to get the immediate results … Many [NGOs] are weak in terms of strategic planning [and] transparency.” —DevTech representative B.8 APPENDIX C STAKEHOLDER SUGGESTIONS FOR MIDCOURSE CORRECTIONS TO CARS This page has been left blank for double-sided copying. APPENDIX C MATHEMATICA POLICY RESEARCH A. Reading programs Stakeholders varied in their suggestions for midcourse corrections to CARS reading programs, but most agreed that additional training and follow-up are needed. NGOs and DevTech made several suggestions to improve CARS reading programs, ranging from better integration of APA and QL principles in EpC handbooks to providing a snack at EpCs to encourage attendance among those kids who have long commutes or get hungry between classes and the EpC. Suggestions from educators included more variety in the types of didactic materials and earlier notification when the CARS team intends to visit. Interestingly, the most common suggestion among NGOs, CARS, and educators was more intensive training and more frequent follow-up visits to help educators fully adopt the new educational approach. In contrast, parent and community leaders’ suggestions focused on increasing the support offered by EpCs, including serving more children, offering instructional supports in more subjects, and increasing instruction time. Table C.1. Stakeholders’ suggestions for midcourse corrections: CARS reading programs Suggestions from: NGOs and DevTech Teachers, facilitators, and principals Parents and community leaders • More frequent training sessions and follow-up visits with teachers and facilitators. • Further consolidate APA and QL principles in EpC modules and handbooks. • Offer a snack to students at EpCs to encourage attendance and good nutrition. • Monthly visits (at a minimum) to support teachers as they implement the new educational approach. • More varied learning materials, including audio books and videos. • More advanced notice when CARS intends to visit. • NGO staff could demonstrate more activities firsthand in training or during follow-up visits. This would help teachers see the techniques in action. • Expand EpC to include more children from the community, children from higher grades, and subjects outside of reading. • Adapt the EpC schedule to improve attendance and avoid instances in which kids return home at a late hour. Source: In-person interviews with 7 DevTech representatives, 7 NGO representatives, 35 principals, 54 teachers, and 26 facilitators, in addition to 36 focus groups with parents and 35 focus groups with community leaders from June to October 2016. NGO = nongovernmental organization; APA = Aprendo, Practico, Aplico; QL = Quantum Learning; EpC = Espacio para Crecer; CARS = Community Action for Reading and Security intervention. B. Community engagement efforts NGO representatives suggested that CAPs include a streamlined development and approval process, and a broader definition of community needs. Several interviewed NGO representatives would welcome fewer steps to CAP development and approval, to reduce the overall development and approval timeline and minimize NGO burden. NGOs also suggested that CAPS be reformulated to define multiple priority areas for schools and communities, and to lay out a plan to secure commitments from a variety of funders and community members, C.3 APPENDIX C MATHEMATICA POLICY RESEARCH including USAID. In this way, the CAP could more comprehensively respond to school and community needs and thus be more useful in planning a series of school and community improvements. Parents suggested that facilitators and teachers should fulfill their commitments to attend parent school and that sessions should be held at a more convenient time. Despite expressing general support, parents had some suggestions to improve parent schools. The most cited suggestions were that parent schools should be held at times that are more convenient for parents (for example, mornings are difficult due to work obligations), educators and NGOs should fulfill commitments to attend parent schools when they are scheduled, subject matter experts should teach the topics instead of regular teachers who may have limited knowledge of the topics, and the sessions should dive more deeply into topics about parent-child conflict and provide parents with more tools to improve their communication skills and ability to teach children. C.4 This page has been left blank for double-sided copying.