December 2019 A partnership with American Institutes for Research Contract No. AID 663-C-15-00001 An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO) Project Conducted by READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED MONITORING AND EVALUATION (READ M&E) Disclaimer: This report was made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the American Institutes for Research (AIR) and do not necessarily reflect the views of USAID or the United States Government. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project Conducted by READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED MONITORING AND EVALUATION (READ M&E) American Institutes for Research 1000 Thomas Jefferson Street, NW Washington, DC 20007 Submitted to: U.S. Agency for International Development/Ethiopia Addis Yigzaw, Contracting Officer’s Representative Under Contract No. AID-663-C-15-00001 Table of Contents Acronyms and Abbreviations ................................................................................................................iii Executive Summary................................................................................................................................1 Introduction........................................................................................................................................1 Methods.............................................................................................................................................1 Summary of Key Findings.................................................................................................................2 Introduction.............................................................................................................................................6 Ethiopian Educational Context ..........................................................................................................6 The READ CO Project ......................................................................................................................6 Evaluation Purpose and Questions ....................................................................................................8 Literature Review....................................................................................................................................9 Study Methods........................................................................................................................................9 Defining the Evaluation Criteria......................................................................................................10 Data Collection Instruments ............................................................................................................13 Limitations and Constraints.............................................................................................................16 Findings ................................................................................................................................................16 IR 1. Access to Supplemental Reading Materials Increased ...........................................................17 IR 2. Opportunities for Children to Practice Reading Created ........................................................23 IR 3. Parental and Community Engagement to Support Learning to Read Strengthened ...............46 Implications, Recommendations, and Prospects for Sustainability ......................................................50 Perception of READ CO Impact .....................................................................................................51 Promising READ CO Activities......................................................................................................52 Emerging READ CO Activities.......................................................................................................60 Conclusion ............................................................................................................................................66 References.............................................................................................................................................68 Appendix A. The READ M&E Project ................................................................................................70 Appendix B. Data Collection Tools......................................................................................................71 Figures Figure 1. READ CO Intermediate Results..............................................................................................7 Figure 2. SRMs: Recommendations for Overcoming Barriers.............................................................53 Figure 3. School-Based Reading Activities: Recommendations for Overcoming Barriers..................54 Figure 4. Development of a Reading Culture: Recommendations for Overcoming Barriers...............56 Figure 5. Capacity Development of Stakeholders: Recommendations for Overcoming Barriers ........59 Figure 6. Youth Reading Volunteers: Recommendations for Overcoming Barriers............................61 Figure 7. Community-Based Reading Activities: Recommendations for Overcoming Barriers..........62 Figure 8. Locally and School-Based SRM Production: Recommendations for Overcoming Barriers and Sustaining Successes......................................................................................................................64 Figure 9. Government Ownership and Engagement: Recommendations for Overcoming Barriers and Sustaining Successes......................................................................................................................65 Tables Table 1. Sampling of Zones, Woredas, and Schools.............................................................................12 Table 2. Local Respondents Sampled...................................................................................................13 Table 3. Examples of SRMs Most and Least Read by Students, by Language ....................................19 An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project iii Acronyms and Abbreviations ADA Amhara Development Association ANFEAE Adult and Non-Formal Education Association of Ethiopia AIR American Institutes for Research EGRA Early Grade Reading Assessment FGD focus group discussion KETB Kebele Education and Training Board PTSA Parent Teacher Student Association MT mother tongue OWDA Organization for Welfare Development in Action READ CO Reading for Ethiopia’s Achievement Developed Community Outreach READ M&E Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation USAID United States Agency for International Development RIP regional implementing partners SNNPR Southern Nations, Nationalities, and Peoples Region SRM supplemental reading materials TDA Tigray Development Association An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 1 Executive Summary Introduction In April 2019, the READ Monitoring and Evaluation (READ M&E) project, funded by the United States Agency for International Development (USAID) and implemented by the American Institutes for Research, conducted an evaluation of the READ Community Outreach (READ CO) project.1 The purpose of the evaluation was to understand the merit of the project’s approach, inputs, and activities from the perspective of the READ CO partners. The merit of READ CO was manifest in the project’s relevance, effectiveness, perceived impact, and prospects for sustainability (Chianca, 2008). A second purpose of this evaluation was to learn lessons from READ CO implementation that would enable USAID, local governments, and practitioners in community development to make informed decisions about the design and implementation of future community programming. The evaluation questions reflected the evaluation’s purpose. The two questions guiding this evaluation were as follows: (a) What outcomes has READ CO achieved as a result of the implementation of the activities? and (b) What knowledge and lessons can be drawn from the activities of READ CO related to project implementation and accomplishments? The evidence generated from this evaluation also enabled an informed interpretation of project successes, barriers to success, and prospects for sustainability in the selected communities. Careful probing with stakeholders revealed their level of satisfaction with READ CO and beliefs about whether READ CO goals had been met. Methods READ M&E conducted the evaluation using a qualitative design. Qualitative methods fostered the generation of richly descriptive data on the activities, implementation challenges, and perceived impact of READ CO as understood by the implementing stakeholders (Miles, Huberman, & Saldana, 2014). Informed by a Utilization-Focused Evaluation approach, the READ M&E team aspired to derive useful insights that, although not representative of all communities in Ethiopia, could nonetheless be valuable to practitioners engaged in community development endeavors (Patton, 2008). From April 1, 2019, through April 12, 2019, 30 researchers collected data from 41 communities in the five targeted READ CO regions and in seven languages (Tigrigna, Amharic, Afaan Oromoo, Sidaamu Afoo, Wolaytatto, Hadiyissa, and Af-Somali). Researchers employed semistructured interviews and focus group discussion protocols to collect the data. The researchers collected data from nine types of respondents, 1 READ CO improved reading outcomes by (a) building capacity to enable Ethiopian communities to develop and distribute high-quality supplemental reading materials, (b) increasing the number and quality of opportunities for primary-age children to read through school- and community-based activities, and (c) fostering greater parental and community engagement in reading activities by promoting a culture of reading. Save the Children International implemented READ CO in two regions, and Regional Implementing Partners implemented the project in three regions. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 2 including primary-age students, in each of the target regions. The respondents provided rich data from multiple perspectives, which enabled triangulation and cross-referencing. Indicators of READ CO Success READ CO achieved many objectives in the participating communities. The activities created opportunities for children to improve reading through various school- and community-based endeavors; increased students’ access to high-quality supplemental reading materials (SRMs); enabled students to receive enhanced instructional support through youth volunteers and mother￾tongue (MT) teachers; and empowered parents to create the conditions necessary to foster the development of a reading culture in many homes and communities. These activities and new conditions yielded observable improvement in student reading skills, especially for struggling readers and girls. A partial list of indicators of success in terms of the relevance, effectiveness, and perceived impact of READ CO endeavors included the active use of new SRMs, improved student performance (as perceived by teachers and youth volunteers based on the results of diagnostic assessments), better performance compared to non-READ CO participants (as perceived by MT teachers, principals, cluster supervisors, and woreda [district] education office experts—and as indicated by higher scores at community and school-based reading competitions), increased participation rates in reading activities, increased parental support for participation in READ CO activities, students’ lower reliance on MT teachers and volunteers for the most basic reading support, enhanced collaborative norms across stakeholders, and evidence of community investment in sustaining the reading activities, such as fundraising and provision of facilities for reading. There were, however, differences in the relevance, effectiveness, and perceived impact of project inputs and activities across and within communities. Despite READ CO successes, there were barriers to the effectiveness and impact of some endeavors. These barriers included the prevailing school-centered norms and culture, high levels of youth volunteer and teacher turnover, lack of appropriate infrastructure, shortages of materials, domestic (home) challenges to participation, dependency on outside support, and access and participation issues. Both the indicators of and barriers to success for each of the intermediate results are presented in detail in the Findings section of this report. Summary of Key Findings Supplemental Reading Materials. The distribution of new SRMs to communities and schools was effectively executed in most communities. Students found SRMs engaging and cognitively, linguistically, and culturally relevant. Teachers, parents, and youth volunteers believed that the SRMs contributed to improving children’s reading skills. Principals, teachers, and cluster supervisors believed that the SRMs were closely aligned with the textbooks and MT curricula and helped enhance children’s understanding of their lessons. The SRMs were easy for students to carry, and the fonts, text in the books, and pictures were attractive. The stories in the SRMs were appropriate for the most part, although children expressed concern about the content of some stories that contained upsetting references. In some regions, children had to wait to borrow SRMs. There were also indications that materials were not sufficient in number or were held in An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 3 storage in some schools. Overall, however, most communities are currently using the SRMs with effective results and have plans to expand SRMs access through investment. To ensure the consistent availability of SRMs, some schools and local government offices already have raised funds for printing and distributing new editions. School-Based Reading Activities. Evidence indicated that READ CO school-based activities, such as reading corners and reading clubs, were effective across the diverse communities. Improvements in test scores, increased participation rates in activities, and child and parent enthusiasm and motivation were attributed to the effectiveness of school-based activities. Stakeholders perceived schools as the natural place to improve reading skills. Schools provided a safe environment for activities and access to teachers and resources. Youth volunteers working in partnership with schools received moral, professional, and logistical support from schools when necessary. Youth volunteers were especially effective when working in close partnership with teachers to support struggling readers in schools. Most stakeholders preferred school-based activities to community-based activities because schools were making a commitment to sustaining these activities, whereas this commitment was lacking in many of the communities. Currently, the prospects are high for the sustainability of these school-based endeavors. The school-based reading activities, such as the reading clubs and reading corners, are now functioning in communities in which schools have incorporated these activities into their annual budget plans. Community-Based Reading Activities. Core community activities included the creation of reading camps and book banks. Community activities provided opportunities for students to practice their reading in some, though not all, communities. Some teachers and volunteers believed these activities were beneficial because out-of-school environments were more suitable for struggling readers who were free from the pressure of competition in school classrooms. When adequate facilities were available, book banks were successful in many communities. Stakeholders attributed the success of community activities to high-quality SRMs, well-trained book bank managers, and clear borrowing rules and polices. However, the reading camps were not as relevant or as effective as the school-based reading activities. Primary barriers to success included low government support for buildings or rooms for activities and a lack of support for the youth volunteers who coordinated activities. Without greater support for and commitment to providing reading services outside of school, in terms of both space and well-supported personnel, the prospects for sustainability of these activities are modest. Development of a Reading Culture. Parents, principals, MT teachers, and youth volunteers in many participating communities indicated that READ CO had changed their community’s reading culture. A reading culture evolved in communities in which students actively used the established reading rooms and reading corners in schools, freely accessed the book banks containing the new SRMs, and participated in reading activities such as book festival days. Parents developed positive attitudes toward reading when they saw that their children clearly benefited from the close collaboration of MT language teachers and the youth volunteers. Parents indicated that both they and their children were motivated to read, and they eagerly participated in community activities. Nevertheless, while READ CO activities have fostered a new culture in many communities, the prospects for long-term sustainability remain in question. Evidence An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 4 showed, for instance, that many community members still thought the education of children (including their own) was squarely the responsibility of the school system. While parents now have clubs and camps to send their children to, it remains to be seen how the opportunity cost calculation may be altered once the READ CO project is no longer directly supporting the school and the community. Youth Reading Volunteers. The preparation of the youth volunteers equipped them well to support children with reading difficulties. In explaining the effectiveness and perceived impact of READ CO activities, woreda officials and principals cited the support of the volunteers as a crucial factor. The volunteers accurately assessed students and supported their learning through the application of the SRMs and instructional techniques they had learned during training. Volunteers initially were tasked with leading reading camps, managing book banks, and supporting other community activities. However, in many kebeles (wards or neighborhood associations), officials made little or no effort to support community reading camps and book banks by procuring the necessary facilities or providing incentives to volunteers. Thus, the quality of the volunteers’ work varied across the regions. Perhaps unsurprisingly, some of the volunteers’ best work occurred when they provided their services at schools, where they received support from the teachers. Stakeholders consistently noted that when volunteers worked in the same spaces as teachers and principals, their collaboration encouraged cohesive vision and goals, alignment of instructional activities, and stronger communication about the needs of the individual students receiving support. Capacity Development. READ CO developed the capacity of the Kebele Education and Training Board (KETB), the Parent Teacher Student Association (PTSA), reading volunteers, MT language teachers, parents, woreda education offices, supervisors, and school principals to support reading activities in both the school and the community. Across most of the communities, there was evidence of enhanced skills and knowledge about reading instruction and ways to build and sustain reading activities. Most stakeholders expressed satisfaction with the training sessions and consistently cited their relevance and effectiveness for the implementation and monitoring of the reading activities. The continued capacity building and follow-up have facilitated a smooth working relationship between MT language teachers and reading volunteers in many communities. Further, there is some evidence that woreda level experts have learned new monitoring and reporting approaches (such as the use of diagnostic assessments) that will support their capacity to continue the progress monitoring of primary students’ reading performance . In many communities, these types of activities are already included in the work tasks of woreda supervisors and education office experts. Local and School-Based SRM Production. In most regions, MT teachers and youth volunteers used the READ CO project’s material development guidelines to prepare additional SRMs. They accomplished this material production by collecting poems, proverbs, stories, tales, and other oral literature. Selected MT teachers, volunteers, students, and community members participated in the development process. Students also participated in the collection of stories and tales from the community. However, there were challenges to sustaining this work during the project. The motivation to do the work declined when the professional-quality SRMs appeared in some An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 5 communities. Further, in some woredas, researchers noted that many of the handwritten SRMs were not accessible to children because (a) these endeavors had started near the end of the project, and (b) in non-cluster schools, it was difficult to change handwritten materials to print copies due to the lack of computers and printers. The prospects for the sustainability of these activities are moderate unless targeted support is provided to motivate teachers to continue developing these materials. Government Ownership and Engagement. From the government schools up to the Regional Education Bureau, government officials expressed positive attitudes toward the reading activities. Woreda education office experts and cluster supervisors actively participated in the implementation and evaluation processes, took part in student assessment activities, observed and monitored community reading activities, and occasionally led instruction and even personally read aloud to children. They actively in the resource mobilization efforts and supported principals in budgeting and using available resources for the duplication, purchase, or preparation of school-based SRMs. However, it was not always clear whether woreda experts were accountable for the success of the reading activities. In some communities they were involved in ensuring that activities were successful; in others, less so. There was strong demand from the community for the full engagement of government offices, mainly the woreda education office and their woreda cluster supervisors. As participants of the evaluation indicated, one of the greatest shortcomings of READ CO was the lack of consistent collaboration that resulted in sustained local governments procurement of appropriate facilities for community activities such as the reading camps and book banks. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 6 Introduction Ethiopian Educational Context The READ Monitoring and Evaluation (READ M&E) 2018 Early Grade Reading Assessment (EGRA) found that 37.2% of Grades 2 and 3 students were nonreaders. According to a 2018 USAID report, only 8% of Grade 3 students reached their grade-appropriate reading level (the ability to read fluently with full or almost full comprehension) (AIR, 2019). Children who do not learn to read in the early grades are more likely to repeat grades and ultimately to drop out of school. In 2008 EC (2015–16), the Ethiopian Ministry of Education (MoE) found that 18% of children dropped out of school by Grade 1 and slightly more than 50% of students completed primary school (Trines, 2018). Yet, reading skills are essential in the development and education of children. These skills not only underpin success in a given grade level but also determine the educational progress of children over the course of their studies. Thus, improving early grade reading is a priority in Ethiopia. Accordingly, the United States Agency for International Development (USAID) supported the Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO) project to improve early grade reading outcomes by supporting the critical stakeholders directly responsible for children’s learning. The READ CO Project Funded and launched by USAID in January 2015, READ Community Outreach (READ CO) worked with MoE, the Regional State Education Bureaus (RSEBs), woreda officials, and kebele￾and school-level stakeholders to develop and sustain community- and school-based early grade reading initiatives. READ CO improved reading outcomes by ensuring the provision of high￾quality supplemental reading materials (SRMs), increasing the number and quality of opportunities for primary-age children to read and write, and fostering greater parental and community engagement in reading activities. READ CO worked in approximately 2,300 schools in the Amhara, Oromia, and Tigray regions; the Southern Nations, Nationalities, and Peoples Region (SNNPR); and the regions of Somalia. These schools represented about 8% of the total number of schools in Ethiopia. Save the Children International directly implemented READ CO in two regions, while regional implementing partners implemented READ CO in three regions. 2 The expected READ CO intermediate results (IRs) and corresponding sub-intermediate results (sub-IRs) appear in Figure 1. 2 Regional implementing partners included the following: Amhara Development Association, Adult and Non￾Formal Education Association of Ethiopia, Organization for Welfare and Development in Action, and Tigray Development Association. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 7 Figure 1. READ CO Intermediate Results IR 1. Access to Supplemental Reading Materials in Vernacular Languages Increased Before the READ CO project was introduced, children’s reading materials in mother-tongue (MT) languages were rarely available in Ethiopia. Through IR 1, READ CO increased access to high￾quality, culturally relevant, curriculum-based supplemental reading materials (SRMs), tailored to the appropriate reading levels of students in Grades 1 through 4.3 The project produced and distributed SRMs to a wide range of stakeholders for school and community use in the targeted regions. READ CO trained teachers and youth volunteers to effectively use the materials in their instructional activities. In addition, READ CO built ministry and local-level (teacher) capacity to develop SRMs aligned with newly developed MT curricula, and produced SRM guidelines for decentralizing SRM development, review, and production to the community level. IR 2. Opportunities for Children to Practice Reading Created The purpose of IR 2 was to develop and institutionalize school- and community-based opportunities for students to practice their reading to improve their learning outcomes. READ CO trained zonal, regional, and woreda officials as well as school cluster supervisors on the use of SRMs, community fundraising, school- and community-level reading skills development activity manuals, and gender and disability inclusion strategies. Local stakeholders then trained school- and community-level stakeholders (PTSA and KETB members, MT language teachers, and principals). Language teachers and principals trained youth reading volunteers to coordinate book borrowing and to lead reading clubs and reading camps. School- and community-level stakeholders created reading rooms, reading corners, and reading clubs in schools and 3 READ CO originally was designed for children in Grades 1 through 8. USAID and the Ethiopian MoE agreed to target only children in Grades 1 through 4 in mid-2017 because of delays in revising the MT language curricula for Grades 5 through 8 (USAID, 2019). An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 8 established book banks and reading camps in communities to create more opportunities for children to practice reading. These activities included a combination of independent reading opportunities, peer-to-peer support, support from MT teachers, and youth reading volunteer initiatives inside and outside of school. Regional and woreda-level supervision teams monitored schools’ progress and provided technical support when needed. IR 3. Parent and Community Engagement to Support Reading Strengthened READ CO prepared PTSA and KETB members to build a culture of reading in their communities. PTSA and KETB members used their training to transform communities’ reading culture by encouraging parents to attend and actively participate in parental awareness sessions, holding regular community and school reading awareness meetings, and advising parents on ways to create a home environment conducive to reading. PTSA and KETB members mobilized parents and the wider community to attend community reading festivals, competitions, and other events. They raised funds for reading activities and to support youth reading volunteers. Woreda supervision teams strengthened the capacity of PTSAs and encouraged them to incorporate READ CO activities into their school improvement plans. Evaluation Purpose and Questions The purpose of this evaluation was to understand the merit of the READ CO project approach, inputs, activities, and outcomes from the perspective(s) of the primary project beneficiaries. The qualitative methodology generated evidence to enable an informed interpretation of project successes, barriers to success, and prospects for sustainability in the communities in which the evaluation was conducted. Careful probing revealed stakeholders’ satisfaction with READ CO and their beliefs as to whether the project had met READ CO goals. This approach enabled the research team to shed light on the community-level activities, and reveal important nuances in terms of contextual details and complexity from the perspective of actors at multiple levels within the system (Miles et al., 2014). The evaluation provides detailed descriptions of how the students, teachers, high-level officials, and other stakeholders experienced and made sense of the READ CO project endeavors. A secondary purpose of this evaluation was to generate lessons from READ CO implementation that could enable USAID’s Ethiopia Education Office, local governments, and practitioners in community development to make informed decisions about the design and implementation of future community-support programming. Four questions guided the evaluation: 1. What outcomes has READ CO achieved as a result of the project implementation? 2. What is the degree of stakeholder satisfaction with READ CO’s performance and level of achievement? 3. How do stakeholders explain the READ CO project? What is stakeholder experience like in the program? If READ CO is working, why is it working? If it is not working, why not? 4. What knowledge and lessons can be drawn from the activities of READ CO related to project implementation, barriers, and accomplishments? What, if any, improvements can be made? An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 9 Literature Review READ CO makes important contributions in Ethiopia and the broader field of international development education efforts to enhance community support for education in order to improve learning outcomes. At the outset of this evaluation, the READ M&E team reviewed the literature on community-based, primary-level education interventions. A broader understanding of the literature on community-based interventions frames the results of this evaluation for stakeholders in a broader context of similar activities. Some of the studies reviewed employed experimental conditions in which the reading outcomes control and treatment groups were analyzed; others were qualitative studies that relied heavily on stakeholder perceptions of the effectiveness of the interventions. Since the Coleman Report (1966), education studies have shown that the experiences and activities of children both before formal learning begins and outside of school can influence learning. Parents, caregivers, home environments, and communities play an essential role in promoting and sustaining school readiness and performance. More recent studies also indicate that out-of-school learning interventions that increase parental involvement and/or establish after-school and summer learning may improve children’s learning outcomes (Weiss et al., 2009; National Opinion Research Center [NORC], University of Chicago, 2014). Some studies have demonstrated a strong correlation between parental involvement and education outcomes (Bus, van Ijzendoorn, & Pellegrini, 1995; Scarborough & Dobrich, 1994; NORC, 2014). Research has also shown that parents with positive attitudes toward education and high expectations for their children’s school performance and future positively affect children’s performance in school (Bronfenbrenner, 1992; Henderson & Berla, 1994). Children who spend time learning outside of school and children with home learning resources and opportunities perform better than children without those opportunities (Clark, 1990; Alexander, Entwisle & Olson, 2001). A review by Bus and colleagues found that joint (parent and child) book reading had a positive effect on children’s reading achievement (1995). Parents listening to their children as they practice reading is also correlated with positive reading outcomes. Further, parents who are provided with specific techniques for listening to their children read showed greater outcomes than parents who were provided with general information (Toomey, 1993). Nye and Schwartz (2006) confirm that interventions that give parents specific training and instructions produce more positive results. A meta-analysis by Senechal and Young (2008) found that interventions that taught parents to teach their children to read were the most effective, whereas interventions in which parents were trained to listen to their children read had a lesser effect, and parents who read to their child saw no statistically significant effects. Study Methods While many of the studies reviewed employed experimental conditions, the benefit of a qualitative study is that qualitative studies add deep contextual understanding of why or why not a particular community-based intervention was effective or not. In addition to examining assessment scores (which experimental studies do), it is also essential to understand the nuances An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 10 of how participants experienced the intervention. READ M&E conducted this evaluation using a qualitative design in order to capture the full complexity of the READ CO activities. The sophisticated set of interventions and the inter-relations between the various components could best be understood through deep probing into the unique experiences of those who directly participated in the activities. READ CO involved stakeholders with a multitude of perspectives and positions (parents, children, government officials, teachers, and other personnel), and in order to develop a coherent picture of the merit of READ CO it was essential to engage with all these participants. The qualitative methodology employed in this study generated rich descriptions of the complexity of project implementation and enabled researchers to capture critical perceptions and experiences of key stakeholders. The new knowledge from this evaluation will inform future decisions about community engagement interventions. Data were collected through participatory means using semistructured interviews and focus group discussion (FGD) protocols. The research team collected data from nine different types of stakeholders—including primary students—in each of the target regions and zones. The complete data collection protocols are presented in the appendices of this report. READ M&E sought perspectives from multiple levels of stakeholders to gain a better understanding of the ways in which the various participants experienced the READ CO activities. The team customized questions and prompts for each stakeholder group and aligned each of the protocols with the purpose of the evaluation. Informed by a Utilization-Focused Evaluation (UFE) approach (Patton, 2008), the READ R&E team sought to derive useful insights that could be valuable to others engaged in community development work. A core tenet of the UFE approach is that the quality of an evaluation should be judged on its usefulness to its intended users—in this case, the USAID mission in Ethiopia and Ethiopian stakeholders active in the design and implementation of community education programs. To learn as much as possible about the READ CO project, the research team collected data from a diverse spectrum of communities to move the analyses beyond “the unique experiences” of just a few stakeholders. We sought patterns in the data and commonality of experiences across communities to highlight useful lessons. Defining the Evaluation Criteria The AIR team posed open-ended questions to participants, who were invited to elaborate freely on their experiences and perceptions. However, researchers employed four a priori evaluation criteria to facilitate coherence in the data analysis and interpretation of the findings. We used these four criteria to organize the presentation of the findings under each of the IRs. Because a core research question of the study focused on determining the merit of the READ CO project, we needed a clear definition of merit. For the purposes of this evaluation, merit was determined by the READ CO’s relevance, effectiveness, perceived impact, and prospects for sustainability. These four criteria align with the evaluation criteria established by the Organisation for Economic Co-operation and Development for international development projects. Further, the criteria provide the evaluation user with organizing concepts around which to clearly interpret the findings (Chianca, 2008). Next, we briefly elaborate on the meaning of each of these criteria. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 11 Relevance refers to the extent to which an activity, product, or output is suited to the needs, priorities, and policies of the target group. In evaluating the relevance of a project or components of a project, it is useful to consider whether the expected activities and outputs are consistent with the overall goal and the attainment of its objectives. Are materials culturally, linguistically, developmentally, and institutionally appropriate for their intended recipients? Are the activities and outputs of the program likely to stimulate the intended impacts and effects? Effectiveness refers to the extent to which an activity attains its objectives. Are processes and endeavors functioning the way their creators intended? Are the products produced of high quality? According to Chianca (2008): “In evaluating the effectiveness of a program or a project, it is useful to consider the following questions: To what extent were the objectives achieved or are likely to be achieved? What were the major factors influencing the achievement or nonachievement of the objectives?” In the case of READ CO, for example, we wanted to understand the effectiveness of key inputs such as reading corners, resources and materials, collaboration mechanisms, activities, and methods as they pertained to ensuring that objectives were met as outlined in the IRs and sub-IRs. Perceived Impact4 refers to the perceived positive and negative changes produced by an intervention, whether directly or indirectly, intended or unintended. In READ CO, the primary variable of interest was the perceived impact of activities and resources on student outcomes in reading, with a primary measurable indicator being test scores and results of progress assessments. However, other variables of interest were changes in adult behavior, changes in cooperative norms in the communities, or other perceived changes believed to have resulted from the project. When assessing for perceived impact, we answer the following basic question: What do informed beneficiaries perceive has happened as a result of the project that has made a difference to them? Because this evaluation did not employ a design that would substantiate claims of statistically discernible impact, use of the term perceived impact is intentional. Qualitative studies frequently rely on self-report and perception data. However, claims that student outcomes have increased because of an intervention can be plausible when a teacher has conducted baseline and end line student assessments. In this evaluation, the fact that impact refers to a teacher’s (or others’) perception of impact does not mean that the notion of impact loses all credibility. Sustainability refers to whether the benefits of an activity are likely to continue after donor funding. Projects need to be institutionally, culturally, and financially sustainable. When evaluating the sustainability of a program or a project, it is useful to consider the following questions: To what extent did the benefits of a program or project continue after donor funding ceased? To what extent have the institutional arrangements (in schooling, for example) been changed such that they sustain 4 The term impact in evaluation studies is frequently used in a strictly statistical sense, i.e., in experimental conditions where impact refers to a measure of effect size (a coefficient representing a portion of change, typically in standard deviation units) that can be plausibly attributed to an intervention. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 12 and maintain activities without reliance on external support (Chianca, 2008)? What major factors influenced the achievement or nonachievement of program or project sustainability? Selection of Zones, Woredas, and Schools The five participating READ CO regions in Ethiopia were stratified into 17 zones, 94 woredas, and 2,312 schools. The team randomly selected zones for all regions except SNNPR, in which the selection of zones was based on the languages of intervention (recall that there were five regions but seven languages of evaluation). After the selection of zones, woredas from selected zones were chosen randomly through a lottery method. A similar approach was followed to pick schools from selected woredas. All schools and replacement schools were selected randomly before the training. The data collectors sampled from two replacement schools because of logistical constraints for collecting from two of the original schools. Based on our experience with other qualitative evaluation studies, we believed that 41 schools would provide more than adequate data saturation and that beyond this number, fewer and fewer new lessons were likely to be learned from the addition of schools. The large and diverse sample ensures robustness of the data and allows for the exploration of the diversity of the communities. There were rationales for selecting schools and communities with a modicum of proportional representation and randomization. First, in regard to proportionality—with the exception of the SNNPR—more schools from Oromia and Amhara were selected because these were the two regions with the largest participating populations from the five regions. Second, the five regions were culturally and linguistically different, and the researchers sought to ensure that the possibility for regional variation based on these differences would be captured in the sample in order to add richness. Third, there were different implementing partners in the regions, and by ensuring sample coverage of all five regions, we ensured some inclusion of all implementing partners. Fourth, an element of random selection would better ensure that schools with varying degrees of “success” would be included in the sample. That is, had we relied on “recommendations from education stakeholders” or “snowballing” for community/school selection, we might have ended up exclusively with communities that had highly successful experiences with READ CO. Thus, random selection at each administrative level was done to ensure that schools with varying quality of READ CO experiences would make their way into the sample. Table 1 shows the distribution of zones, woredas, and number of schools selected for this evaluation. Table 1. Sampling of Zones, Woredas, and Schools Regions Participating in READ CO Sample Selected Zones Woredas Schools Zones Woredas Schools 1 Tigray 4 10 336 1 2 4 2 Amhara 4 26 647 2 7 9 3 Oromia 5 36 775 3 9 12 4 SNNPR 3 18 434 3 6 12 5 Ethiopia Somali 1 4 120 1 2 4 Total 17 94 2,312 10 26 41 An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 13 Selection of Respondents It was important that the selected schools engaged in some way with the READ CO project and that knowledgeable personnel were available to meet with the data collectors. After contacting the school/community and confirming this basic level of understanding, the AIR team contacted the individual participants who were directly involved in the READ CO project implementation. That is, we purposively selected respondents who either had directly or indirectly received training from READ CO (woreda experts, KETB members, PTSA members, principals, MT language teachers, youth reading volunteers, and parents) or were beneficiaries (students in Grades 1 through 4) of the READ CO program. Table 2 presents the breakdown of respondents by number and type of respondent. Table 2. Local Respondents Sampled Region Woreda KETB PTSA Youth Volunteer Principals Supervisor MT Teachers Students Parents Tigray 2 4 8 7 4 4 10 47 28 Amhara 7 21 35 21 9 8 38 108 71 Oromia 8 29 30 23 11 12 35 132 92 SNNPR 6 34 45 26 10 12 29 137 97 Ethiopia Somali 2 5 17 7 4 3 8 47 37 Total 25 93 135 84 38 39 120 471 325 Notes. KETB is Kebele Education and Training Board. PTSA is Parent Teacher Student Association. MT is mother tongue. Data Collection Instruments As READ CO was a complex endeavor involving stakeholders at multiple levels, it was essential to gather data from participants with different perspectives and experiences in the sampled communities to ensure a holistic understanding of the READ CO project. Data from multiple sources enabled triangulation and either confirmation or rejection of claims and versions of events. Seven data collection tools were used: 1. Interview Protocol for Experts From Woreda Education Office. This tool was designed as a semistructured interview to collect qualitative data from experts working for woreda education offices as a READ CO focal person. 2. Interview Protocol for Principals and Supervisors. This semistructured interview protocol was developed to explore principals’ and supervisors’ knowledge of literacy activities implemented by READ CO or other implementing partners, their roles in the implementation of the project, their perceptions of the project’s effectiveness, and their views on improvements that could be made. 3. FGD Protocol for MT Teachers. This tool enabled the collection of data from group discussions among MT teachers working in schools that supported the READ CO program. The FGDs enabled the research team to understand how MT teachers understood their role in project implementation. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 14 4. Interview Protocol for KETB and PTSA. This tool enabled data collection from individuals or group interviews of selected members of Kebele Education and Training Board (KETB) and the PTSA of each school. 5. Interview Protocol for Volunteers. This interview tool was designed to collect data from volunteers working to support reading activities in after-school activities and schools. 6. FGD Protocol for Students. This tool was designed to collect data from group discussions among pupils who attend READ CO program and after-school activities. 7. FGD Protocol for Parents. This tool elicited data from group discussions among parents to learn whether and how they supported their children’s reading habits. During the FGDs, the READ M&E team also used the title pages of Grades 3 and 4 SRMs with students, MT teachers, and parents to collect evidence of students’ and teachers’ use of SRMs and parents’ familiarity with or exposure to the SRMs as proxy indicators of the effective use of SRMs. Piloting the Data Collection Instruments READ M&E piloted the tools with the aim of assessing the appropriateness of the FGD and interview protocols for the evaluation’s purpose. All tools were used and data were collected from six woredas and six schools in three regions (Amhara, Oromia, and SNNPR). Woreda education office experts, principals, cluster supervisors, KETB and PTSA members, MT teachers, youth volunteers, students, and parents participated in the data collection process. READ M&E collected data from 173 (female = 57) participants. The data collection teams audio-recorded all the FGDs and interviews, and they took detailed notes. According to the analyses of the pilot data, the tools in general were appropriate for the evaluation’s purpose. However, some of the tools required editing (for example, adding the names of implementing partners for the project sites at which READ CO was operating through partner organizations). The analysis also uncovered the presence of questions that could be improved or removed. For example, the researchers identified some questions that would be better answered by one type of respondent type rather than another (e.g., the question on the SRM most preferred by students), different numbered questions with similar ideas, questions that could be combined, and questions least understood by respondents. Based on the results of the pilot, the team deleted some items, rephrased ambiguous questions, combined some questions with those containing similar ideas, and reordered the positions of some questions. The researchers then sent the final tools to the AIR home office in Washington, DC, and to USAID for approval. Initially, the FGD and interview protocols for Sidama, Wolayta, and Hadiya (except for the children) were in Amharic, the working language of the region. From the observation of the data collection process in Sidama Zone, the READ M&E team decided to translate the tools for these languages into their local languages, as was done for the other regions. Data Collector Training and Data Collection The READ M&E team recruited 20 experienced, qualitative data collectors and ten independent supervisors from universities, teacher training colleges, Regional Education Bureaus, and the MoE. The minimum qualification for participation was a Master of Arts degree in education, An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 15 psychology, or social sciences and fluency in the local language. Ten data collection teams (three teams for Afaan Oromoo, two teams for Amharic, and one team each for the rest of the languages), each consisting of two members, were formed to collect the data. The READ M&E team conducted a 2-day data collector training session on March 28 and 29, 2019. The training introduced the READ CO activities and expected project results, the purpose of the evaluation, and the guiding questions for the evaluation. Next, trainers covered the content and purpose of each data collection tool, how to approach the respondents, and how to use the data collection tools, including the SRMs for the children’s FGD. Trainers led a discussion on each question for each of the seven tools and led the group to a consensus per the intended meaning of each question. The READ M&E team then presented lessons learned from previous data collection experiences, including the pilot. In the training session, the READ M&E team also covered how to record the data, maintain the privacy of the respondents, and take accurate notes. During the afternoon session of the second day, the trainees participated in a simulated data collection exercise and discussed strategies for troubleshooting problems. The teams collected operational data from April 1 to 12, 2019, in the five regions and in seven languages (Tigrigna, Amharic, Afaan Oromoo, Sidaamu Afoo, Wolaytatto, Hadiyissa, and Af￾Somali). One member of the team conducted the interviews and facilitated the FGDs while the other took notes. The data collectors conducted 79 FGDs with students and parents, 102 individual interviews, and 158 group interviews with MT teachers, members of KETBs and PTSAs, and youth volunteers. Data collection supervisors, READ M&E staff, and staff from the AIR HO ensured data quality through daily debriefs with each team. The teams completed data collection in 10 working days. Coding and Data Analyses The field team first translated all the recorded transcripts from the local languages into English before conducting an initial content analysis of the collected data using NVivo 12 software. Trained data coders used a codebook to sort and organize the data into broad categories by themes and organizing principles. The team treated the three READ CO IRs as parent nodes; an additional 28 child nodes were originally created to sort the data by categories. The main headings of the findings were derived from USAID’s IRs for READ CO, and the subheadings were the criteria upon which the researchers evaluated the merit of the READ CO program. Then, in the text under each subheading, we derived additional categories from those that emerged from the data analyses. To ensure coder reliability before the analyses began, we convened as a team to share assumptions. Before coding the data, in order to improve the inter-rater reliability of coders, data coders practiced coding the same one or two pieces of text selected from each of the seven tools representing a particular IR and evaluation criterion. The coders worked individually and then discussed in a group how they each had coded the practice texts. They repeated this practice until the variation among coders was reduced. Then, two READ M &E team members and two researchers from the AIR U.S. office analyzed the coded data with NVivo 12 software. The data analyzers identified important subthemes that emerged from the data analyses under each IR and held another discussion before commencing the study write-up. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 16 Limitations and Constraints One limitation of the evaluation is that interviews and focus groups ultimately rely on the credibility of self-reported data. It would not be unusual for some participants to slant responses based on what they believe data collectors want to hear. However, READ M&E data collectors were trained to probe deeply and to seek not just surface-level responses but also the evidence that respondents used to support their claims. Second, because the goal was to assess READ CO program achievement against the four evaluation criteria, we do not present a detailed description of what happened in each of the communities over the life of the project (from start to finish), as might be expected in some qualitative studies with a smaller sample. Findings are not presented separately by region, community, or language but holistically by thematic area. Data are presented in various thematic aggregations from the nine types of respondents to paint an overall picture of what happened in the communities in terms of what outcomes were achieved across the communities sampled, rather than within a single community. Third, we do not present the findings grouped by respondent type, although the data were triangulated to verify that no major contradictions existed between what two or more different stakeholders reported from within the same community: We analyzed the data by respondent type and found general consistency in the reports across the different levels of stakeholders regarding the impact of READ CO in their communities. Finally, as an evaluation with a UFE approach, the study focused on the stakeholders’ perceptions of outcomes attained and the corresponding lessons that could be learned for future work in community development. READ CO was a 5-year project, and many respondents were not with the project at the outset. Major obstacles may have been overcome in the first few years of the project, and experiences and lessons learned in overcoming those initial barriers may be under-represented in this report because most stakeholders were familiar with the project only during the last several years of implementation—either through memory loss or given the fact they were not in their present position at READ CO’s inception. An additional point in regard to the analysis of the findings is a constraint rather than a limitation. There are constraints tied to the general difference in use of qualitative vs. quantitative research methodologies (Miles et al., 2014). The findings, while informative, are not generalizable across all participating READ CO schools. We have looked for patterns within the 41 schools sampled but these same patterns may not be apparent across all READ CO schools and communities. Thus, caution should be used when making claims or drawing inferences about the utility of results from a single community or this group of communities to the broader domain of all READ CO schools. Findings We present the evaluation findings organized by the three READ CO intermediate results, divided by findings for each of the three major criteria of relevance, effectiveness, and perceived impacts of the READ CO interventions. We address the prospects for sustainability of the various project components in the final section of this report. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 17 IR 1. Access to Supplemental Reading Materials Increased Access to and Use of SRMs Distribution and Sufficiency. The Tigray Development Association (TDA), Amhara Development Association (ADA), and Organization for Welfare Development in Action (OWDA) provided SRMs for Tigray, Amhara, and Ethiopia Somali, respectively, while READ CO regional project offices provided SRMs for Oromia and SNNPR. Experts from Wolayta woredas indicated that large quantities of three types of SRMs (read-aloud, leveled readers, and decodables) were supplied by the READ CO project office. Approximately one-half of principals in the sample reported that an adequate number of SRMs were distributed to their respective schools. Some principals stated that the number of SRMs that reached the schools was insufficient compared to the number of students who needed the materials. As a result, in some cases children had to wait for their turn to borrow SRMs. One reason reported for the wait time was that the number of children joining primary school increased every year, thus making it a challenge to match the number of children with the appropriate number of SRMs available. This situation was reported by some woreda education office experts in the Somali, Hadiya, Amhara, Oromia, and Sidama regions. Two principals (one from Tigray and one from Wolayta) explained that there was a lack of materials for Grades 3 and 4. A woreda expert in Tigray also reported that the number of SRMs distributed to Grades 1 and 2 students was sufficient, but that it was insufficient for the students in Grades 3 and 4. A few respondents complained that there were not enough copies of the most popular SRMs. A principal from Tigray stated: “I believe that the number of SRMs distributed is enough. For me, the problem is related to variety in the titles of the SRMs. The SRMs received are limited in type.” Despite the perception of shortages in some cases, other woreda education office experts from Amhara, Somali, Hadiya, and Oromia believed that the number of SRMs distributed to their respective woreda was adequate. A woreda expert from Amhara observed that there was no scarcity of SRMs. Rather, in some rural schools, the SRMs were kept in storage and therefore went unused because of a lack of proper supervision. Woreda education experts, cluster supervisors, and members of KETBs oversaw the distribution of SRMs, monitored and followed guidelines on the effective use of SRMs, made checklists, and conducted site visits. A woreda expert from the Amhara region explained: “We work in collaboration with ADA and principals of the schools. ADA brought the SRMs to the woreda education office. Then we distribute the SRMs to the cluster schools and principals and mobilize the community to collect the SRMs and make [them] available to mother-tongue teachers and volunteers.” An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 18 Members of a Wolayta KETB noted that whenever they visited schools and reading camps, they asked about the sufficiency of SRMs. Wherever scarcity was noted, they worked with the school principal and woreda education office expert to solve the problem. In Amhara, administrators used checklists, which proved to be an effective tool during supportive supervision. The administrators used these checklists to evaluate the ways in which the SRM materials were being used. Relevance of SRMs to Community, Teacher, and Student Needs Student Use of SRMs. Students gave different reasons for selecting the SRMs they chose to read. These included liking the content (e.g., the reading passage was attractive), easy to understand, related to their actual life experiences, provided better knowledge (e.g., the SRM taught students more about traffic rules, respecting elders, and how marriage is conducted in a given culture), were visually attractive, or were assigned by MT teachers to read. Hadiya students noted that they were least attracted to SRMs related to the death of people and cattle. Students who participated in the FGDs in one of the schools of Oromia identified SRMs that scared them, such as those that contained information about car crashes and transmitted disease (dhukkuboota daddarboo). Unsurprisingly, the students identified these SRMs as the least attractive. During the FGDs, students were asked, “From where do you get these SRMs? Where do you read? Can you take them home?” The responses were similar for all languages. Students explained that they received the SRMs from teachers and youth volunteers at both school-based (reading rooms, reading clubs, reading corners) and community-based locations (kebele reading centers, reading camps, community book banks, and libraries). They responded that they read them in their school classrooms, at camps, in the library, under the shade of trees in the school compound, and in their homes. Most students said that they borrowed the SRMs for weekends and took them home for a few days at a time. In four of the FGDs (two Hadiya and two Sidama), participants indicated that they could not borrow and take the SRMs home. Other stakeholders were questioned about student SRM use. A woreda education office expert reported that project-supplied and locally made SRMs were available in schools and reading camps. According to a Wolayta woreda expert, children could easily access the materials and were effectively using them in the schools, at reading camps, and at home. Others had similar views. For example, an expert from Tigray explained: “The SRMs are already in the schools and book banks in the community. Children access these materials either in the reading club or at the reading camp. They borrow them from their mother-tongue language teachers (reading corner) or from the book bank. Currently, children are using these SRMs effectively, unlike in the previous times where students/parents were reluctant to borrow (also some mother-tongue teachers were reluctant to lend) in fear of loss or damage of the SRMs. This time, students frequently borrow SRMs and take them home to practice reading.” An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 19 An MT teacher from Oromia reported: “Students are using the SRMs effectively. Each student borrows the SRM from the MT language teacher for two days in a week. There is [a] format prepared in the school to write the name of borrowers, the title of the SRMs, lending date, and date of return.” Adding to this point, an MT teacher from Hadiya reported that students practiced reading either in the reading camp or the reading club. They also read SRMs at home with the support of family members. In the village, the children frequently read in pairs. Table 3 presents the titles of the most and least read SRMs, as indicated by the students. Table 3. Examples of SRMs Most and Least Read by Students, by Language Language Most Read SRMs Least Read SRMs Title Title Tigrigna ድሕንነት ብሽክሌታ ኣንበሳን ማንትለን Af-Soomaali Cigaal shiidaad Iyo Hashiisii Webi Cudurada Faafa Amharic አባባ ትልቁ ብርሃን እና አረጋዊው እቴነሽና ስንታየሁ Sidaamu Afoo Qarqaru Agarooshshe Bude Macciishshinanni Niwaawe Wolayttatto Ishali Shoorota Qaxxara Bacaallaa Buukulo Koshshaa Hadiyissa Doolleebaam Ga'nanne Afu Hawwo Gaalshawwaa Shirdeemmii Afaan Oromo Arfaaseefi Dannabaa Kaayyoo Milkoofte MT Teachers’ Use of SRMs. Overall, teachers were positive about the quality of the SRMs and the effectiveness of the book-lending process for home reading. One MT teacher from Oromia facilitated book borrowing according to student interest and need. After students returned the books, the teacher asked questions, listened to student reflections, and provided feedback and corrections. The teacher believed that the school benefited considerably from the SRMs provided. However, creative planning was sometimes required to ensure effectiveness. For example, the teacher noted that for 43 students in the class, there were only 20 SRMs. Thus, the teacher had to divide the students into groups and adhered to a tight schedule for lending the SRMs. The teacher also worked hard to ensure that the books were not lost or damaged so that they could be transferred to the next generation. She noted: “We lock the books once the students are done with them. I teach the afternoon class, so in the mornings, the principal takes the keys and gives out the books. We organize the books according to their weekly schedule. We make sure the books are not torn.” An MT teacher from Amhara reported that the quality of the SRMs that her school received was excellent. The teacher observed, “As a fourth-grade teacher, I haven’t seen a student who doesn’t An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 20 want to read the books. All of them raise their hands; they don’t say they won’t participate. All of them are eager to read.” This teacher gave students 30 minutes before the end of each school day to read the SRMs. She also lent the SRMs twice a week (on Tuesdays and Fridays) so that students returned them on Saturdays and Sundays. The teacher explained that a form provided by READ CO for tracking the borrowed books was effective in keeping the book checkout process organized. Similarly, an MT teacher from Amhara described how MT language teachers used the SRMs: “READ CO books are our supplemental tools, and they support the students a lot. The books will be given out according to their reading level. The SRMs are not like other books that we toss away or give away once we are done with them. They go back to the store. . . . We have schedules where we give out the SRMs turn by turn. If some of them get one on Monday, then the rest will have one on Tuesday. This is how we compromise. They have read a lot, but we are not done reading yet, as every topic has its own week.” Appropriateness of SRM Level and Cultural Relevance In addition to the adequacy of SRM provision and frequency of use, special attention was given to learning about how relevant the SRMs were for teachers and students. Mother-tongue teachers and youth volunteers found the SRMs’ level of complexity and content well-tailored to their students’ grade levels and cultures. For example, MT teachers in Oromia explained that the SRMs increase in complexity over the grade levels. Grade 1 SRMs have large text, games, and teach letters; Grades 2 and 3 SRMs include short words and sentences; and Grade 4 SRMs advance to longer sentences. In general, most of the principals, cluster supervisors, and woreda experts agreed that the SRMs were appropriate for the developmental needs of children. One of the cluster supervisors from Tigray stated, “The fonts of the texts in the SRMs are big, the pictures are attractive, and the stories are also very funny for the children.” While describing the relevance of SRMs, a principal from Amhara reported, “We have seen students read those beautiful books with great pleasure.” Stakeholders perceived the SRMs as relevant to students’ needs, daily lives, and culture. According to parents, the presence of familiar stories in the materials facilitated children’s understanding of the material and increased their motivation to read. Alignment of SRMs With MT Textbooks Another important issue was the extent to which the SRMs were aligned to core textbooks. An MT teacher from Amhara remarked, “For us, the contents of SRMs and the regular textbooks are the same and consistent with [the] Amharic weekly and yearly teaching plan.” Mother-tongue teachers, principals, and cluster supervisors favorably judged the alignment of SRM content with the new MT instructional materials and reported that the SRMs complement MT textbooks used in the classroom. An MT teacher from Wolayta explained: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 21 “The SRMs used at the reading camp complement classroom activities. Similarly, the materials used in the reading club go well with the materials we use in the classroom. For example, if the textbook talks about river water, the other one discusses the ocean water. The topic remains on water in both reading materials. The match between the SRMs and the new student textbooks enhanced children’s understanding of the lessons.” Some KETB and PTSA members from Amhara also believed there was a link between the daily lesson from the textbook and the contents of the SRMs. MT teachers used the SRMs that complemented the lesson needed at the appropriate time. Principals, cluster supervisors, and members of KETBs and PTSAs of different regions and zones held similar views and used different terms to express the alignment. Some of the topics were also related to non-language subjects, such as environmental science. A Wolayta principal and a Hadiya cluster supervisor indicated the presence of alignment between the contents of SRMs and the classroom textbooks for Grades 1 through 4. Overall, the evidence indicated that the SRMs distributed in the activity were well aligned with the new student textbooks prepared by the READ CO program. Effectiveness of Training to Use SRMs Training Organization and Participation. READ CO and its implementing partners, such as TDA, ADA, and OWDA, organized training sessions that focused on multiple topics, including how to use SRMs, coordinate reading activities, implement plans, communicate with children, screen children who need support, support children’s reading activities, and prepare contextually relevant SRMs. Almost all respondents who attended training events could name the other community participants who also attended. In most of the training sessions, the trainees were diverse: Youth volunteers, MT teachers, cluster supervisors, school principals, PTSA and KETB members, and woreda education office experts or READ CO team members all frequently attended the training events together, though in some cases they attended them separately.5 Training Effectiveness (woreda and government experts). Woreda education office experts and cluster supervisors gained knowledge about how to follow up, supervise, and monitor reading activities with SRMs in their communities. The woreda and kebele training participants expressed satisfaction with the training sessions and consistently cited their effectiveness in supporting their implementation and monitoring of activities. A woreda education expert from Wolayta observed, “The training helped us in coaching teachers and principals. It capacitated us even to solve reading-related problems at [the] school level and checking students’ participation in the reading activities.” 5 There was considerable variability in the number of training days that stakeholders reported receiving. A review of the data indicates that it was not uncommon for some stakeholders to have attended a total of 3 to 4 days of training for the project, while others attended up to 8 days of training. In Amhara, a woreda expert attended only two days of training, whereas one Somali World Economic Outlook expert attended 7 days of training in preparation for the READ CO activities. This overall random pattern of participation may be attributable to the high turnover of parties involved in the reading activities. For instance, volunteers, KETB and PTSA members, woreda focal persons, and others joined the reading activities long after the project had commenced and after the earlier training sessions had been completed. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 22 Woreda education experts and cluster supervisors noted that they were able to cascade trainings they received to principals, MT teachers, and selected members of the community in an attempt to fill training gaps. For example, a cluster supervisor from Oromia organized trainings for MT teachers, youth volunteers, and other community members who did not receive training. The same supervisor mentioned that the stakeholders training organized by READ CO created a collaborative spirit among participants: “We got the opportunity to know each other better and thereby work together collaboratively.” Members of the PTSA from Ofa and Wolayta expressed a similar view: “All the stakeholders who took part in the training have good working relationships.” One Oromia woreda expert, however, expressed dissatisfaction with the organization and effectiveness of some training events in the early years of the project but declared that organization had improved over the last few years: “At the beginning of the program, there were a lot of defects. For example, all stakeholders were not invited to [the] training, and activities implemented were not well organized. However, this year READ CO has been implementing tangible activities (for instance, rewarding those who perform well). So, significant effort was made by this organization this year.” Training Effectiveness (teachers and principals). Mother-tongue teachers and reading volunteers from Hadiya, Amhara, and Oromia emphatically cited the relevance and effectiveness of the training they received for implementing READ CO activities and using the SRMs. A reading volunteer from Hadiya remarked: “The trainings we received at different times built our capacity to implement the reading activities. We did not know anything about components of reading activities before. The trainings were helpful in this regard. We gained knowledge about how to assess and grade children based on their reading performance.” Likewise, an MT teacher from Amhara observed: “Yes, we partook in the training organized by ADA. The training was on reading activities. It helped us in implementing our plan as per our schedule, communicate with children, and use the SRMs. For example, we learnt to keep the most frequently used SRMs to the front so that children can easily access [them]. The training also helped us to become more aware of voluntary services.” Mother-tongue teachers in South Gondar, Amhara, shared the following: “ADA provided the trainings to some of us. Those of us who took the training cascaded the training to MT teachers who did not. The training helped us improve how to help children in their reading activities. Before the training, we were using the SRMs as we wanted. But after the training, we started to use the SRMs properly, as per the weekly regular schedule.” An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 23 Many MT teachers from Tigray, Wolayta, and Oromia explained that the trainings helped them use the materials and align the SRMs with classroom instruction. For example, an MT teacher from Wolayta stated: “A single training helped me a lot. Before the training, I did not know what to do and what is expected of me. Last year I was not clear, but this year I am helping children with better knowledge. I have seen changes in the way I use SRMs this year.” Similarly, MT teachers from South Gondar, Amhara, noted that the training they attended improved their instructional practices and helped them share experiences with other MT teachers. An MT teacher from Oromia also participated in trainings to develop local and school-based SRMs. In connection with this, an MT teacher stated, “The trainings we received were effective and helped us to prepare SRMs by collecting proverbs, stories, and sayings from members of the community. Students also participated in the collection of stories and tales from the community.” An MT teacher from Somali confirmed that OWDA conducted training about how to implement reading activities to all parties involved in the activities. Training Effectiveness (youth volunteers). Reading volunteers from the Wolayta Zone expressed in specific terms the ways in which they benefited from the training provided. After the training sessions, they were able to help children use the SRMs and assess their knowledge. The training also helped them avoid the use of mixed languages during instruction. The Oromia reading volunteers believed the training was highly effective in supporting children with reading difficulties. They learned how to plan and identify children’s level of reading and to rank children in different level of ability, from D (lowest) to A (highest). In addition, the reading volunteers reported that they had received training on how to use the forms and procedures for lending books from the book banks. The volunteers reported considerable skill development during training events relevant to the tasks they were to carry out. IR 2. Opportunities for Children to Practice Reading Created READ CO aimed to increase opportunities for Grades 1 to 4 students to practice reading in structured activities: reading clubs and reading corners in schools, and reading camps and book banks in participating communities. As highlighted under IR 1, READ CO created and distributed SRMs and trained teachers, volunteers, and other stakeholders to carry out these activities. Once prepared, educators and volunteers collaborated with local officials to establish school-based reading rooms and located spaces in the community to conduct the activities. Under the IR 2 activities, educators and volunteers aligned their teaching activities with the MT reading curricula and SRMs, increased their skills in identifying struggling readers, improved their reading instruction technique, built communities of peer learners, and monitored student learning progress over time. Teachers and volunteers focused on the needs of struggling readers and girls to close achievement gaps. In this section, we present the evaluation findings pertinent to IR 2. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 24 Relevance Community and Student Needs. The reading activities implemented were relevant across the regions based on the poor performance of Ethiopia’s school children on the 2014 EGRA. Teachers and administrators explicitly indicated that low EGRA scores were a primary rationale for the introduction of the READ CO program, making it highly relevant to their needs. One woreda official from the Sidama region noted that the approach, which relied on youth volunteers and new SRMs, was a good way to get to “the root of the problem,” as the many practicing teachers were not effective instructional leaders. Parents in Oromia and other regions noted that the poor reading skills of Grade 6 graduates indicated that assistance was long overdue and that this program addressed their need to improve. More specifically, prior to READ CO, teachers did not have an assessment tool to quickly identify struggling readers. Project implementors learned to screen for children with the lowest reading skills so they could receive targeted support. Teachers, principals, youth reading volunteers, and woreda personnel across the regions mentioned that the student reading assessment tool enabled them to effectively identify students with the lowest skills and efficiently assign them to the appropriate support activities. Implementors considered the improvement of their capacity to accurately identify baseline reading levels as highly relevant support. The student assessments estimated oral reading fluency (words per minute read) and assigned students to a low (Group D or Group C), medium (Group B), or high (Group A) ability group. In addition to the tool to establish baselines, teachers and volunteers received appropriate tools for tracking student progress during the span of activities. They used checklists to track which student needed which type of support and book level and to follow student progress during camps on follow-up assessments. A Sidama youth volunteer explained, “Volunteers received lists of the lowest performers from the mother-tongue teachers. In some cases, children who were more advanced in skill levels also found the programming relevant to their needs.” Across the regions, among the children with the two lowest reading levels (C and D), the use of the assessment tools helped monitor which students were selected to receive support so that equal proportions of males and females participated in reading activities (i.e., assignments to groups were based on objective need, not arbitrary criteria such as gender). The system for assessing students and the use of results in decision making meant that task implementors overcame any arbitrary barriers to participation. Ownership and Access to Activities. In addition to having relevant tools to complete tasks, school-level stakeholders reported that they were actively involved in the design of the reading activities from the outset. One principal stated, “I was involved in making the reading space suitable for children.” A member of a PTSA from the Wolayta region assisted in the selection of a location for project activities to ensure that they were appropriate and quiet for reading. Respondents believed that their direct involvement in planning the work enabled them to help adjust programming to the local context and culture, ensuring that the work remained relevant to student and community needs. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 25 Throughout the life of the activities, stakeholders at the higher administrative levels accessed and participated in the reading activities through site visits, spot checks, mentoring and peer coaching meetings. Easy access to activities and monitoring progress meant that principals and/or woreda or kebele personnel could provide corrective measures or suggest adjustments in implementation to ensure that activities were meeting students’ needs. In many cases woreda, kebele, or other officials participated directly in tasks, such as assessing individual students, to learn more about the project. School-Based Versus Community-Based Activities. As highlighted in the IR 1 findings section, officials, parents, teachers, and volunteers found the SRMs used in the school and community-based reading activities relevant in terms of difficulty level as well as cultural, contextual, and linguistic appropriateness. In this section, we focus on the appropriateness of the physical spaces allocated and human resources assigned to lead the reading activities. An important aspect of relevance to student needs was the safety and comfort of facilities for small children. Educators in the Wolayta Zone believed that school-based activities were good for children because they were run in protected school compounds; that is, it was a safe environment with few distractions. As one teacher explained, “There is nothing that disturbs them in club activities.” The same teacher noted, “Children are more familiar and happier within school compounds.” School-based activities were relevant for the children for other reasons. In the presence of principals, teachers, and other support personnel, students received the professional support of trained educators who worked on the school premises. Some youth volunteers noted that many people were skeptical of their relevance to the project: They received questions such as, “How will you be able to assess and teach students with no teaching experience?” Activities on school grounds also promoted student access to other resources, such as bookshelves, book boxes, and floor mats. As we highlight in the section on the effectiveness of the IR 2 activities, there was considerable debate over the extent to which community activities were relevant to student learning in a cultural context that traditionally has relied on the schools to solve educational problems. The data indicate that in comparative perspective, many principals, supervisors, teachers, and parents strongly believed (at least initially) that school-based reading activities were more relevant to student needs than community-based activities. The relevance of the youth volunteer work was also a contested aspect of IR 2; a discussion of youth volunteer effectiveness is presented later. Effectiveness School-Based Reading Activities In this section, we present findings related to the effectiveness of school-based activities such as reading corners and reading clubs, the effectiveness of MT teachers, and efforts to monitor project and learning progress.6 The evidence indicated that the school-based reading activities 6 The activities differed in important ways in each community context, as did the ways in which stakeholders talked about the activities. For example, sometimes book banks were created in both An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 26 (reading corners and reading clubs) were effective. Mother-tongue teachers frequently credited the attractiveness of the SRMs as key to their own effectiveness in leading these activities. In addition, school-based activities were most successful when connected to the work of the youth volunteers. Tigray teachers in one woreda, for example, noted that the most effective way to improve student learning was to tightly align the SRMs, methods, and reading activities with the community-based work. Such alignment involved communicating and collaborating closely with the youth volunteers. Student Perspectives on Reading Corners and Reading Clubs. The student focus groups offered insight into how students made sense of the reading activities. The data across students from the regions confirmed what adult stakeholders had reported: Children enjoyed the SRMs, frequently borrowed and read them at home, and actively participated in reading corners and reading clubs. In the focus groups, students clearly articulated the purpose of these activities, and they were aware of the activity norms, such as days and times of attendance, the rules for checking out books, and the consequences for not returning them on time. Because students borrowed books both from community book banks and at schools, we discuss these activities in both the school-based and community-based activity sections of this report. Borrowing Books. There were similar patterns of book borrowing from the school reading activities across the regions. Students commonly checked out a single book and returned it after 2 or 3 days. At this pace, they could read one or two books within a week. Many students read these books on the school premises in their free time. Family members helped students read when they brought the SRMs home. One student explained the SRM borrowing process at her school: “We can get the book that we ask the school to provide us. We read quickly and return the book. We can borrow another book as we like. We read it at home, and when we finish, we bring it back and borrow another one.” Students indicated that reading the SRMs at home with family or older siblings was helpful. However, some students reported that reading in clubs was preferable to home reading because in clubs there was always someone present to correct their mistakes. One student stated, “We practice the reading and writing and support each other,” at clubs and in the corners. Reading Corners and Clubs. Students actively engaged with the SRMs and teachers through school activities. In one Oromia region, students mostly read in their reading club on Tuesdays and Thursdays, from 8:00 to 9:30 a.m. Oromia students read at what they called “a reading park” (reading corner), and the book checkout was in the principal’s office. Students there noted, “There wasn’t a book that [we] did not like.” School-based reading clubs sometimes served as an schools and community locations; other times, activities like reading camps (intended to be organized outside of school) were held at schools due to a lack of infrastructure in the community. The role of youth volunteers, discussed under the section on community activities, could also vary from appearing to be “school based” or “community based,” depending on the region and community context. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 27 extension of the regular classroom activities. In one Oromia woreda, a representative described what she observed during reading club activities: “SRMs were arranged in the reading room by grade level. Students read them to one another in front of the class, which enabled the students to learn from one another and be more confident in their reading and writing. The teachers assessed the activities of the students in 90 minutes and gave feedback to the students immediately. With support from the teacher, the students increased their skill.” The Role of MT Teachers in School-Based Activities. Teacher participants across the diverse regions played similar roles in the implementation of the reading activities. After receiving initial training, they began by assessing their students, sometimes with the volunteers. They typically assigned the struggling readers to reading clubs and referred them to the community reading camps, coordinating and organizing schedules and activities with the volunteers. They integrated the new SRMs into their daily and weekly lessons. Teachers frequently held activities in reading corners and reading rooms twice a week. They lead the reading activities, encouraged students to practice, invited parents to competitions and reading events, and monitored student progress. Teachers encouraged pair and group work to create opportunities for peer help. Teachers mentored youth volunteers on strategies for working with the students outside of school in the reading camps. A teacher described a typical routine: “Both a reading club and reading corner are available in our school. We assign students to different groups. Pairs are formed from lower and higher level students. The pairs are gender based: girls with girls and boys with boys. Students who score low are sent to the youth volunteers. Twenty-four students with low scores are selected at each grade. The struggling readers stay with the volunteers until they can read properly.” In addition to instruction, teachers took other steps to promote reading. They prepared (by hand) materials such as the “small sized” SRMs to enhance children’s reading skills. Teachers also encouraged students to borrow books from the school or community book bank that aligned with the objectives they were aiming to accomplish during their lessons. Teacher leadership in the school-based activities was a key factor in their effectiveness. Stakeholders across the regions at the principal, cluster supervisor, and kebele levels consistently indicated that the primary reason for reading corner effectiveness was that the activities took place at schools with teacher support. Several woreda specialists and cluster supervisors noted that MT teachers were showing increased ownership of school-based activities from previous years. MT Teacher Collaboration with Volunteers. A crucial dimension of teacher effectiveness in the school-based activities was that many teachers collaborated closely with the youth volunteers. Teachers reported that their own instructional endeavors were the most effective when they were able to coordinate methodological approaches with youth volunteers and focus An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 28 more support on individual students with special needs. In some cases, especially in community facilities that lacked community-based activities, teachers and volunteers collaborated at the schools, and some reading camps originally intended for other locations ultimately were held on school grounds. In the Amhara region, one reading club was considered especially effective by the woreda cluster supervisor because of teacher-volunteer collaboration. In the reading centers, volunteers not only worked with primary students in Grades 1 through 4 but also with language teachers to improve their instructional methods and individualized approaches. The supervisor in this area attributed the effectiveness of these particular reading centers to the availability of teachers to support the volunteers in their activities. Teachers also assisted the volunteers with the assessment of students and maintenance of reading progress checklists. This same pattern of collaboration was found in many areas. A Tigray teacher explained: “I arrange and fix programs for the students to borrow the books once a week [every Tuesday]. We also make children participate in the reading clubs together with the youth volunteer. I also support the youth volunteer to align the reading club activities with my weekly lessons to make the support method coherent and aligned. We also determine if the children are truly understanding the stories in the books by allowing them to do questioning and answering activities in the class.” In many cases, volunteers worked on the weekends and gave the MT teacher student progress reports on the following Monday. In those situations, students in essence received “extra school days,” when reading activities took place on the weekend. Participation Levels in School-Based Activities. Multiple stakeholders indicated that the number of students participating in reading activities had increased over time, especially the activity in the school reading rooms. However, while this sometimes occurred because of the enthusiasm of educators, in some cases this was due to the fact that volunteers and kebele officials were not able to locate spaces for the community activities. Thus, some activities, by default, moved back into the schools. Nonetheless, many teachers and volunteers collaborated across multiple settings and spaces in challenging circumstances. Stakeholders noted important challenges with the school-based reading corners and clubs. Reading corners were not always operational for Grades 3 and 4, due to classroom and material shortages. In regard to their overall effectiveness, however, a kebele representative from Amhara explained: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 29 “The most efficient activity, I think, is the reading corner run by the teachers. Because teachers organize competitions in their class during spare time and students want to compete, this leads them to engage in reading practice. The board sometimes awards pens and small materials to the winners, which motivates students to read and attend the competition sessions. If we see the community reading camp as only volunteers working with individual students through book checkouts, that limits the effectiveness…. The reading camps we have in the school compounds are under the trees near the benches. Since we don’t have alternatives, we prepared the reading there in the school.” Community-Based Reading Activities Reading Camps and Book Banks. The rationale for creating community opportunities to read and write was to provide the greatest number of students access to reading opportunities at a location close to their home. Many students lived far from their schools, and not all were able to attend before- or after-school activities. Thus, reading camps and book banks were frequently— though not always, as already highlighted—located outside of the school in communities close to the students’ homes. In the sections that follow, we break down the camp and book bank activities into several key dimensions of effectiveness: actors, activities and participation levels, location, the role of volunteers, teaching literacy skills, and collaboration across stakeholders. Participant Actors in Community Activities. While youth volunteers could be perceived as “the face of camps and book banks,” many other actors worked behind the scenes to make them happen. Many stakeholders led community activities, and various officials resolved technical and administrative challenges. For example, woreda officials and cluster supervisors, teachers, principals, KETB and PTSA members—and sometimes even parents and private citizens—lent or rented their property for reading activities. A principal in Tigray delivered different orientations for youth volunteers when required. In addition, the principal supported the youth volunteers’ participation in employment opportunities that generated income for them. During the student assessments, many stakeholders across the regions assessed the number of words read by the students in order to assign them to camps. Various actors also played roles in areas such as recruiting students to attend camp. According to one kebele representative: “We have convinced families of children in relation to the reading activities. We have visited each household in relation to the activities. We have motivated parents to build camps, though we did not build a house for camps. There was [a] plan to build a house for camp activities, but we failed to accomplish that. In relation to children’s activities, there was no problem. Everything went smoothly. The effectiveness was due to follow-up by stakeholders.” One Wolayta kebele administrator was directly involved in both training for and organizing reading camps. This administrator stated: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 30 “I was a member of the youth volunteer selection team. I encouraged parents to send their children to the reading activities. We coached the reading activities together with others. We checked the reading skill of children together with Wolaytatto teachers. I contributed my own share in making the reading camps conducive for reading. We supervised youth volunteers and children’s attendance. We organized reading competitions. Children read in groups. They understood the benefits of reading. However, I cannot say the participation is high—it is medium.” Active monitoring was also an important role to ensure that the camp activities were contributing constructively to reading skills. A Wolayta cluster supervisor explained how he or she maintained a feedback loop with the camp and book bank managers: “Youth volunteers notify language teachers when some children miss the reading activities. Teachers convince children of the importance of the community reading activities and send them back to the reading camps. I support and supervise the reading activities. I check whether teachers and volunteers are supporting children based on the training they got or not. I check whether they are supporting children in the two days of the week or not. There is a format for using the 90 minutes effectively. I motivate or appreciate them when their work is productive.” Activities and Participation Levels. As highlighted in the previous section, most reading camps focused on readers who needed the most support. Frequently, students who made great progress were later replaced with newly identified struggling readers. In many cases, students were grouped in a way to enable poor and stronger readers to practice together. This was done deliberately to increase their reading proficiency. Children of both sexes participated in peer reading. One youth volunteer explained: “Children are reading in mixed groups. There are high, medium, and low achievers in each group. The groups are prearranged by the school and used in the reading camps. They are supporting each other in their neighborhood. We will strengthen the coaching activities in relation to their participation. We will motivate them and bring the participation to a higher level. At first, nobody understood the activities of the project. This was actually related to an awareness problem. But after more awareness, things went smoothly.” In one Somali kebele, stakeholders believed that one reason they sustained high participation rates in community reading camps was the effectiveness of the SRMs and student interest in them. A Hadiya principal and member of a reading committee noted that children in their community preferred the reading camps to other activities because the camps were associated not only with reading but also with games and fun. However, at another Somali kebele, an An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 31 administrator observed that some students were motivated and active but that overall participation rates were not high. While perceptions of the effectiveness of reading camps varied, stakeholders across the regions tended to strongly support the idea that book banks were quite effective. An Oromia supervisor emphasized that book borrowing was effective because literate family members at home helped with the children’s reading skills. In cases in which there were no literate family members to assist, volunteers requested that family members lighten the burden of some chores to enable children to practice their reading. One Amhara cluster supervisor noted that book borrowing proved to be more effective than anticipated. Location of Community Reading Activities. A recurring challenge related to community reading activities was securing an adequate location for the activities to take place. For example, in one Wolayta community, both the book banks and the reading camps were at the kebele office to ensure access across communities and schools. In one Oromia community, youth volunteers used a private house for the reading camp and took the books home at night to protect them. This was a creative solution to the location problem, but the findings indicated that not all communities were able to effectively solve this problem. Some Sidama children from another community reported that they had no information about either a reading camp or a book bank. A group of students from another region reported that they did not know where the reading camp was located. Further, responses such as the following were common: “The assumption is that the reading camp should be in a place easily reached by most children. Under some circumstances, children could be excused for not attending camp activities due to the nature of the terrain in rural areas. It is difficult to get full involvement of children.” In one Hadiya community, kebele leaders considered their office to be the neatest and safest place for activities. As will be explored in greater depth later in this report, despite some enthusiasm for reading camps, procuring safe, accessible locations was a challenge for many communities. Some administrators were able to find creative solutions (churches, renting private homes), while others reverted to holding community-based activities at schools or let activities falter altogether. The Role of Youth Volunteers. The use of the youth volunteers was a key part of the READ CO innovation, and we present an analysis of their effectiveness in this section. There was considerable variability in terms of the quality and engagement of the volunteers across communities. Some teachers, principals, and cluster supervisors reported outstanding motivation and solid knowledge of reading instruction on the part of volunteers, while others noted a lack of interest, mainly because of poor incentives and high turnover. Some highly motivated volunteers appeared to be satisfied with the conditions and effectiveness of their work, whereas others clearly identified barriers to their own morale and effectiveness. Across the regions, youth volunteers’ primary tasks were similar. After completing training, they typically set aside 2 days a week to work with the students in need of support. Frequently they worked 3 hours per week. Volunteers mostly ran reading camps and book banks, but this was not An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 32 always the case. When facilities for these activities could not be located, volunteers sometimes worked in the schools and closely collaborated with teachers. In some instances, they used school classrooms on weekends. Volunteer Collaboration with other Actors. Volunteers worked closely with kebele officials, KETB committee members, language teachers, and school principals. Volunteers indicated that collaboration with other stakeholders was key to effective work in the reading camps in order to set schedules, procure materials, secure a location, teach, and monitor student progress. They reported that a frequent obstacle was the lack of a location suitable for the reading camps. Volunteers in one community initially used a private house in a noisy environment that distracted students before later moving to a better location. A group of Hadiya volunteers reported high satisfaction with the SRMs and their training, which enabled them to be effective in their jobs. One volunteer stated that he could see the children improving over time, which gave him great satisfaction. He collaborated well with the school principal and teachers. However, in regard to cooperation with the local authorities, he made the following observation: “The KETB did not see our work closely. They never asked us to solve any problems. We did not get the support from the kebele administration as expected. We are continuing with the help of the school and some volunteer people from the kebele.” Volunteers from Tigray, Somali, and Hadiya appeared passionate about their support to the students. A Tigray and Somali volunteer both reported that students come to their houses in search of more books to read. An enthusiastic volunteer from Amhara explained: “We don’t want a nominal volunteer, with just the name. We don’t want a volunteer that doesn’t teach on the day assigned. I think we have prepared, in our subregion, children that don’t embarrass us. What we do is help the children fill their educational gaps, reading and writing and math, so that children wouldn’t be left at home.” From their experiences working with educators and children to improve reading outcomes, youth volunteers learned as much as they taught. One Hadiya volunteer commented: “We were asked to participate in this campaign voluntarily, without any payment. I personally agreed to this idea in order to develop my mother tongue and to help our children. Since then, we are also learning many things from this program. We are the direct beneficiary of the trainings. It builds our capacity. We did not know anything about reading components before”. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 33 Comparative Perspectives on School- and Community-Based Activities When responding to questions about the school and community interventions, evaluation participants tended to make comparisons and highlight the pros and cons of the different types of reading activities. And while the consensus was that READ CO activities were effective overall, there was no consensus about which activities were the most effective. In the section that follows, we present different perspectives from the data to provide insights on how stakeholders perceived the effectiveness of the various activities. Arguments in Favor of School-Based Activities. School reading clubs were more effective than community reading camps because of the availability of existing human resources and the existence of physical spaces (i.e. school buildings). Stakeholders observed that while all activities had been effective, school-based reading clubs and corners were the most effective because materials were available to support the school-based activities. An Oromia cluster supervisor highlighted that one of his schools ensured that all 161 SRM titles distributed under the project were made accessible to students in the school reading corner. Students could also borrow these books from the school. The general perception was that most community activities could not offer that level of support. In regard to community-based camps, a woreda expert from Hadiya made the following suggestion: “The reading camp needs more supports from the community in order to be effective. There should be proper reading center and materials at reading camps. It was not as effective as expected.” There was a belief in the effectiveness of school-based activities more than community-based activities, primarily due to a more consistent commitment from the school to keep activities going. Schools provided support for the safety of the children and a sense of being on familiar territory, being in a compound with adults who knew them, with a solid physical infrastructure. Tradition and an understanding of the purpose of educational institutions were also a reason that school activities were more effective: School was considered to be the place where proper education should take place, and the role of the teacher was understood as paramount in educating children. When students were at school, they had quick access to teachers and opportunities to attend events before or after their regular shift. A cluster supervisor from a region in Tigray explained: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 34 “The reading corner is the most effective literacy activity because this activity is done in a face-to-face contact with the teacher himself and gets close follow-up. The teacher is always there to support the students in the reading corner, unlike the other activities. The second effective activity is the reading club, the third is the book bank, and the least effective is the reading camp due to its remoteness to follow and unsuitability of the space to read. To my understanding, the project was very effective and successful in the school-based activities starting from its commencement, but had a problem in implementing the community-based approaches directly. The reason was, the kebele administrators were not even trained and unaware of the program activities; to that effect, they resisted mobilizing the communities during the start-up.” Several kebele administrators believed that the school-based activities were the most effective. One administrator reported: “The literacy activities performed in school were more effective. Because in school there is follow-up from Monday to Friday. In the community, what has been implemented is, students take the books home and they [are] supported by literate family members. The reading activities performed in communities do not continue, so they are not very effective, but activities performed in school are better because the places arranged in school are appropriate.” When activities were based at the school, cluster supervisors, principals, and teachers were better able to provide the volunteers with moral, professional, and logistical support. School-based reading corners were effective because there was accountability in the system of students checking out books and returning them. Finally, the frequent turnover of volunteers from their positions made reading camps more likely to be disrupted than school-based activities. One Tigray woreda representative explained: “If we compare the school and the community-based activities effectiveness, it’s loud and clear that the school-based literacy activity is more effective than the community due to the continuous turnover of the facilitators [youth volunteers]. Children’s level of participation at school is medium but is low at the community level.” A Different Point of View. Not all participants believed that school-based activities were more effective than the community activities. In many communities, reading camps and book banks provided important learning opportunities. A Hadiya teacher reported that students actively read books in the community reading center at least twice a week and frequently borrow books from the book bank. The teacher stated that students were more interested in the camp activity and that the work there was effective. Reading camps had specific advantages over school reading corners. First, weekend or after￾school camps offered a low-pressure learning environment for students. Because most of the An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 35 students attending camps were struggling readers, there was a sense of “comfort” in a more relaxed setting. Further, given that leaders of camps were frequently youth volunteers and not teachers, students could be “free.” One Sidama supervisor noted, “In a camp, those who are equal-level readers try to read. They are not afraid of each other. They take reading as a game.” In the reading camps, singing songs and playing games helped to create a more learner-friendly environment with less pressure to perform than in the school environment. One KETB representative reported, “[T]he camps are good for children to use. They are spacious and airy. The place has no disturbing things like cars.” According to a Tigray MT teacher, the children who are afraid to participate in class activities participated better in the community-based literacy activities, and many successfully learned to read with the youth volunteer facilitators. Many activity participants felt that community camps and school-based clubs effectively complemented each other or were of equal merit. The SRMs used in the reading camps were well aligned to the students’ curricular needs. One principal from Hadiya stated, “There is no difference, whether it is community based or school based. Students participate in all reading activities equally.” An MT teacher from Tigray also cited that complementarity of the activities: “The community-based activities are so supportive to the classroom activity in a way that students who lagged behind are sent to the field-level facilitator for additional support service. Children even want to explain [in the classroom session] what they have learnt at the community-based activities [reading camp]. I understand and assume the activities [in school and out of the school] both complement each other.” Many children in Somali focus groups responded to questions about reading activities with simple statements such as “I like reading camp” or “I like reading at school/in the reading clubs.” While some student responses lacked detail, others provided evaluative insights. For example, in one focus group, students indicated that they acquired less knowledge in camps but became more proficient after attending school clubs. (This triangulates with what adults reported about the effectiveness of in-school activities). However, some had different views. One student observed: “Reading camp is the literacy activity in which we participate the most. Our participation in reading camp has enhanced our ability in reading and writing from the regular classroom. Next to the reading camp, we also actively participate in the reading club. In reading club, we write a paragraph and read turn by turn. So, we are improving both our reading and writing abilities. At the weekend, we will borrow the books.” Monitoring and Evaluation of All Reading Activities. For both in-school and community activities, stakeholders employed monitoring and evaluation techniques to ensure effectiveness. Teachers and volunteers started the reading activities with children at the D level (lowest level) in reading proficiency. After the interventions, they evaluated both the activities and the reading performance of the individual students across time (weekly, monthly, etc.) to “graduate” the children from support when they improved. One cluster supervisor explained: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 36 “I have facilitated the selection of children with reading problems in collaboration with Wolaytatto teachers. We have evaluated reading activities with teachers and youth volunteers. I have coached every reading-related activity. I have fixed things based on the supervision report of the regional and zonal level teams who came here…. We produce monthly reports and send them to the woreda. That report contains the details of the changes observed in the children. We make classroom observations and check the changes. We did that in reading camps and reading clubs.” Other supervisors recorded the activities implemented by language teachers, students, volunteers, and school directors for monitoring and evaluation purposes. They assigned roles and communicated with committees, parents, and teachers. PTSA committees monitored those activities with the school principal. At school, the principal coordinated the activities, but at the classroom level, teachers took charge of monitoring success. A Wolaytatto language teacher explained her vision for monitoring and evaluating for program effectiveness: “We cannot stop this role in the future because it is part of our responsibility. When children show improvement, we take them back and send others in their place. When children go to clubs and camps, we follow their improvement. We make decisions based on performance. We evaluate children’s performance every three months. We also follow the work of the volunteers, and we check whether they are borrowing books from the book banks. We involve children in peer reading, and sometimes a student’s peer members are five to seven in number.” Perceived Impact Teachers, supervisors, principals, and youth volunteers claimed that the impact of the activities on student learning was observable. In discussing what participants believed best explained the success they were seeing, several themes emerged. Most stakeholders tended to believe that the use of SRMs at school and through book borrowing, reading at home, in-school instructional activities, and peer help networks was critically important. Participants named the most helpful SRM titles, whether fables or decoders. They also believed that assessment and monitoring tools played a key role in providing accurate data that enabled a clear understanding of which students needed the most assistance. Another recurring, success-related theme was the evidence of reading progress visible in reading competitions and public performances involving parents. Further, close collaboration between volunteers and teachers and solid alignment between volunteer and teacher work were critical to success. In the next section, we more carefully probe the evidence for perceived impact to better understand the extent to which claims of impact might be supported. An Analysis of the Evidence for Perceived Impact. Recall from the introduction to this section that one of the most relevant aspects of the READ CO project—according to woreda officials, principals, teachers, and youth volunteers—was the provision of a simple, easy-to-use assessment tool for assigning students to one of four reading levels. This assessment tool enabled An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 37 stakeholders to take the pulse of students’ progress over time at the most local levels: their schools. While not standardized or administered at a regionally representative level, this type of assessment and monitoring nonetheless can provide revealing information about the plausible impact of the activities. At a minimum, it provides implementors with a more credible claim than “they know there was impact,” when they can provide empirical evidence from a longitudinal progress check on the students participating in the activities. Assessment of Student Progress. Participants frequently referenced test-score increases of the same children across a growth continuum (baseline, midline, endline) as the basis for their perception of impact. Stakeholders also compared other types of reading levels and student motivation before and after the interventions. They also compared the progress of students engaged in READ CO activities with those from other communities or those from higher grade levels. Stakeholders monitored and tracked progress over time by assessing students’ scores in words read per minute. In some communities, PTSA representatives, KETB members, and woreda officials reported that reading levels at Grades 1 through 4 not only improved but also surpassed those of students in higher grades once interventions had time to take hold. Stakeholders at multiple levels used assessments over time, sometimes every few weeks, and sometimes as a baseline, a midline, and an endline. Sometimes teachers and volunteers took the initiative to conduct these progress assessments, and sometimes they did so with the encouragement of woreda, kebele, or school principals. One Oromia MT teacher stated: “We assess the students daily and at the end of each week—we reinforced students who performed well. This made students motivated, and as a result, participation increased. After the reading room and reading club were introduced, students become effective in reading. These students even correctly read books at the Grade[s] 7 and 8 levels. Generally, there is significant change in the reading and writing abilities of students.” Volunteers and teachers assessed students’ words-per-minute rates at frequent intervals and tracked the movement of students across the performance categories (A, B, C, and D). By the end of a cohort’s participation in a round of support, one principal observed that most students were able to read 40 words per minute. Student responses in focus groups indicated that they were aware of their own progress, given that data about their movement across performance groups (C to A, for example) were open knowledge. In regard to measuring performance change, a Wolayta youth volunteer noted: “Children with reading problems are becoming proficient because of our intervention. Children who were at [the] D level have progressed to [the] B level after getting our support. Children who were at the C level have progressed to [the] A level after getting support.” An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 38 Participants also compared the progress of project and non-READ CO participants. An expert from an Amhara woreda reported that intentionally conducting comparative reading assessments with participating and nonparticipating students: “The test shows that there is a clear difference between project supported and non-supported schools.” Another Oromia woreda expert discussed progress monitoring in terms of comparison groups: “Children are motivated to read because the SRMs are attractive and manageable for students. We have eight Community Resource Centers from which seven are included under the project, but one is a control group. The schools under Community Resource Centers which are included in the project have seen tremendous changes in their children’s reading ability. They are far better than those schools not included in the project. There is no doubt that there is great difference between them.” Participants also compared the ability of students within the same school at different grade levels: nonparticipants (the older students) and participants the students in Grades 1 through 4). Younger students frequently did as well as—or better than—the older, nonexposed, students across the regions. One principal even reported having older students turn to a first-grader for assistance with basic reading tasks. A parent in the Amhara region explained, “Fifth grade and above haven’t used these supplemental books; but students who are in Grades 1 to 4 have better reading skills than those students who are in the Grade 5 up to Grade 8.” In one classroom observation situation, an observing cluster supervisor was surprised by the exceptional reading skills that he was observing. Convinced that the students were not actually first graders, he believed that the school principal was intentionally misleading him by having him observe older students rather than first-grade students. Amhara PTSA members, Oromia kebele representatives, principals, teachers, and woreda cluster supervisors from across the regions observed the similar phenomena of younger students in the program outperforming older students who had not been exposed to the reading activities. An Amhara supervisor stated: “In earlier times, we got students who are unable to identify the alphabet in Grade 2. In addition, there was difficulty reading words up to Grade 5. Now, after implementing this project, even Grade 1 students who have outstanding results are able to read 70 words per minute. When we come to Grade[s] 3 and 4, though, the standard is 70 words per minute; those students with outstanding results are able to read 120 words per minute. Thus, this is the change. We properly evaluated and confirmed this.” Youth volunteers across the regions were especially detailed in speaking about the impact of the READ CO program; they had conducted many of the student assessments and worked closely with students in their reading camps. Over the life of the project, an increasing number of students were reading at 90 words per minute. Volunteers provided (unsolicited) the names of specific youth and their words-per-minute scores during data collection. For example, when An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 39 asked about the evidence for intervention impact, one volunteer replied, “Rahel7 from Grade 2: 43 words per minute; Mary from Grade 3: 65 words per minute; Solomon from Grade 3: 77 words per minute….” The volunteers were confident in the validity of their perceptions of the program’s impact. Likewise, youth volunteers from Tigray mentioned students by name and could recall the increases in the students’ reading test scores. Reading Skills Beyond Words Per Minute Read. Students improved aspects of their reading and writing skills beyond the number of words per minute read. Perceived impact on specific literacy skills included alphabetic knowledge, pronunciation, spelling, phonemic awareness, word recognition, differentiation of words, and writing. Students improved in reading and listening comprehension and other academic subjects. A Wolayta kebele official reported specifics about skill improvement, such as improvement in constructing longer sentences with the use of bigger words. Basic skills such as “concept of print” improved over the course of the project. Before the program, many students did not hold their books correctly and sometimes tried to read upside down. Speaking of one young girl, a teacher noted, “X was chosen earlier for reading. When she was in Grade 2, she didn’t know how to read and did not answer any questions when asked. But now she takes her book to her brother who is in eighth grade, and they study together.” Improvement in student dictation skills corroborated the perception that students’ skills improved in several cases. Increasingly, teachers reported that they no longer always wrote tasks or assignments on the blackboard that students then copied into their notebooks. Instead, teachers dictated text (with greater speed than before), and students were able to successfully write down the text without seeing it on the board. Before the program was introduced, students could not write their names correctly, but now teachers were able to dictate text to students. In addition to test score increases and measurable progress in the major skill areas, stakeholders mentioned other ways in which READ CO endeavors made an impact. In the sections that follow, we highlight random monitoring checks, project participation levels, and parent decision making.8 Random Monitoring Checks. Most communities reported that cluster supervisors and principals conducted frequent “spot checks,” when they would meet with students engaged in reading activities. A Tigray cluster supervisor noted that children were able to construct comprehensible sentences during his spot check. One Wolayta cluster supervisor stated that the change over time was powerful: “At one point, a random student from the lowest performing group was asked to read, and she read a full page in less than 3 minutes.” Others also provided specific data on progress and impact. A Somali MT teacher said observers claimed that while it had previously taken a student 20 minutes to read one page, after some time the same child was now reading the page in 1 or 2 minutes. A Wolayta principal observed change in a previously labeled “nonreader” who now was able to read more than 60 words per minute after some months in the program. 7 The names of the students have been changed to protect their privacy. 8 For more on public reading days, holidays, and competitions, see the findings under IR 3. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 40 Recalling one such spot check, a Wolayta principal stated: “We allowed children at [the] A, B, C levels to read in front of the representatives. We showed the existence of change. After getting the support, there is much improvement in children’s reading proficiency. The change is due to the support and SRMs provided by READ CO. The support improved the reading skill of children. It increased the independent reading skill and time. It increased the participation in reading activities. Before the intervention, we saw only three or four students who could read in one class. Now we have many children in a class who can read proficiently. We see many children volunteering in a class when the teacher needs them for reading activity. More than 75% of the children in a class express their readiness to read. This is an amazing change.” During a recent school supervision visit, one Oromia supervisor asked students to read from an SRM called “The Smart Old Lady,” and asked students to retell what they understood. This supervisor stated: “Students fully understood the contents and reflected their reading for me in detail. There is change because parents are giving feedback that their children are performing better. Before the intervention of this project, parents were saying that our children couldn’t read and write. But currently some parents are very happy about the change in abilities of their children. The independent reading time of children has increased greatly. They now understand that reading is important in their education.” Participation Levels. There was a perception that there was a significant increase in the activity participation rates over the course of the project. Several participants noted that in addition to higher participation rates, more children wanted to spend more of their time reading. The widely shared view was that high-quality SRMs were a key reason for increased participation rates; both students and parents recognized the quality of the materials. Teachers and principals also noted an increase in the number of older children requesting access to the SRMs and other materials. Administrators credited their own information campaigns with getting more students involved in the READ CO program. Several stakeholders noted that retention in program activities over time was an indication of impact; after all,, students would not continue if they were not learning and improving. A student support system now existed, according to a Tigray MT teacher; in this network system, students knew other students whom they could help or who could help them. The establishment of these student networks and the availability of these appropriately designed books enhanced children’s motivation and habits and promoted deeper impact of the project. According to an Amhara Kebele official, the impact of the program was evident in increased overall school attendance. The involvement of volunteers and quality materials led to higher attendance, and parents were more engaged because they could see what books their children were bringing home and reading. Within the last 3 to 4 years, the school administration, teachers, and parents appeared to be helping the students read in their spare time at increasing rates. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 41 Parental Views on Participation. Directly connected to the issue of students’ participation level in activities was the important question of what parents thought of the READ CO activities. Because many of the communities were resource poor, the decision of a parent to forgo a child’s time invested in earning income or doing household chores was a strong vote of confidence that their child was attaining something of value in the activities (at least over the long term, when information about merit of the activities was more certain). Therefore, parents’ behavior in regard to whether or not they encouraged participation was a strong indicator of their perception of the effectiveness of the activities. Parents saw distinct changes in their children’s reading habits. They reported that their children could read better than their elders, and that “before the program, our children used books for looking at the pictures. They have improved their reading and spend more time doing it.” According to a parent in Tigray, his youngest child reads at home for more hours than his sister who is in eighth grade, and “Because of this, the younger child is a very active reader and can read whatever he is given.” One Hadiya parent explained how he encouraged his children to read: “When they come back home, I check his exercise book. If I see any problem, I will go to school and discuss with the teachers. After the READ CO project was introduced, the school and its management were working in high spirits. We have seen changes in our children since then. We are happy about what has been done so far. The teachers encourage students to come to tutorial classes. The school also encourages parents to send their children to school. As I have seen in the village, there is improvement in cooperative reading. In one of our houses, they come together and read the books. The outstanding students help others when they return to the village. The parents support them by letting them read.” Impact on Gender Equality. There was a perception that activities led to more equitable outcomes in reading by gender. Participants documented improvement in girls’ reading skills in the Amhara, Wolayta, and Oromia regions. Seven Oromia parents discussed in detail the issue of greater involvement among girls and concurred that READ CO had succeeded in closing gender gaps. One parent noted that while there was still work to be done to get more girls involved, the READ CO program had improved the gender-participation ratio. Students now frequently asked parents to listen to them while they read, and the communication between children and parents improved, including between girls and their parents. The reading camp activities in Wolayta helped girls who were struggling readers to receive support through the activities. PTSA members and youth volunteers took measures to ensure that girls were represented and not denied support if they were identified as poor readers on the initial assessments. One PTSA member explained: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 42 “All children with reading problems started reading well. Today children read well and understand easily. Children in Grade 1 read confidently. They do it with ease. This is change. If you ask community members randomly, they will tell you the same thing. There is visible change at the classroom level. Especially significant change was observed in female students.” Impact on Confidence Levels. An Amhara PTSA member noted that the confidence of girls had grown as a result of improvement in their reading skills. Girls were reading in front of people, and “girl children were equally competing with male children.” Respondents described other evidence of the READ CO program’s impact on children’s confidence, including awareness of their improvement in class standing (as a reader), winning performances at reading competitions, setting higher bars for success at subsequent events, public readings with parents confirming improvement, and more time invested in independent reading. One teacher reported that a shepherd boy herding livestock was reading as he worked. An Oromia student reported: “When we go to our regular class, we feel confident that we can read what our teachers request us to read. No more fear of reading in the classroom. Our reading ability has improved well. We read with no or few mistakes by now. We used to fear reading in the classroom, but now we raise our hands to stand up and read. Our writing ability has also improved. Our rank in [the] classroom has improved after we started to participate in reading camp.” Other Impacts on Adult Behavior. The project changed some adult behaviors through experiences with the activities as education administrators and specialists developed a sense of ownership of the activities. Amhara and Wolayta woreda representatives reported feeling more motivated to visit schools and follow up with activities. They noted that the assessment tools and capacity to monitor progress were important. They reported a shift in their role from that of a cluster supervisor who provided training and supervision to that of a monitoring and evaluation role. They also believed that parents were more motivated to send children to activities. Preparing and Using Locally and School-Based SRMs. Another important impact of the project was the interest it generated among MT teachers and volunteers in developing and using locally and school-based SRMs. In explaining this point, a respondent noted that the READ CO approach was beneficial to MT language teachers in that it helped them develop locally and school-based SRMs. Woreda education office experts noted that teachers and volunteers collected stories, proverbs, sayings, and different oral traditions from elders and other members of the community and prepared locally and school-based SRMs that resemble those prepared by READ CO. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 43 Barriers to Effectiveness and Impact While stakeholders were eager to highlight the impact of certain aspects of the IR 2 activities, they were also not hesitant to point out barriers to effective implementation and impact. In the sections that follow, we highlight barriers and challenges that emerged from the data. Institutional Norms and Existing Structures. One barrier to effectiveness and impact was existing institutional traditions and norms of the communities in which READ CO was working. Political leaders frequently considered the teaching and learning processes to be within the purview of educational institutions (i.e., schools and school oversight bodies). Thus, asking communities and “volunteers” to improve education outside the school space was not necessarily understood and accepted by all stakeholders as the correct approach to improving reading skills. For example, a Wolaytatto teacher argued that officials from the kebele or the woreda level should take over and conduct all of the reading activities. The argument was that only the government could be effective in managing improvement. If there was no government follow-up and supervision, then achievement of the READ CO educational goals would not be possible. A Wolayta principal held the same view. He argued that the necessary follow-up could come only through government oversight. He claimed: “To strengthen the activities, there should effective supervision. Supervision alone is not sufficient; there should be evaluation. There should be recognition of activities done effectively. Government bodies should do that. Trainings should be offered to all stakeholders equally. There should be a focus on supervision. Up to now, no one has recognized those who outperformed others. No one has criticized the underperformers. No one has motivated the schools. These and other similar things should be given attention. In terms of this, there was [a] problem on the side of woreda and zone. On the side of the children, there was absenteeism and tardiness. Sometimes youth volunteers have prioritized their social life.” Youth Volunteer Turnover. The reading volunteers provided a voluntary service. Volunteers left the reading activities when they procured other job opportunities; in many cases, the intrinsic rewards were not enough to keep them in place. Kebeles worked to assist with the replacement of volunteers, but the high turnover rates were believed to be an impediment to project effectiveness and impact. Many volunteers worked hard for little or no compensation. Some volunteers were not active due to a lack of motivation and the absence of rewards. Some came to work but did not put forward their best effort. Some volunteers initially worked hard, but their productivity declined over the long term. One Somali cluster supervisor was direct in his assessment: “There was a poor motivation strategy for youth volunteers, which could be said to be nil.” Although this woreda attempted to offset the lack of incentives with training, the supervisor concluded that this problem persisted. Teacher Turnover. Teacher turnover was a related barrier to the effectiveness and impact of program personnel. Trained teachers who left their assignments rarely passed on their skills and knowledge to their replacements. One of the cluster supervisors described the situation: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 44 “We are newly appointed teachers, as the senior teachers who took the trainings transferred. That is one reason for low achievement. Some of them left early in the first semester, whereas some of them left at the beginning of this semester. It is an obstacle to this activity. I was teaching English, Amharic, and civics for other grades. I came here to fill the void they left. The absence of experienced teachers is a big factor.” Lack of Appropriate Infrastructure. In many communities, there were no reading spaces outside of the schools. In one community, one room initially was provided for reading (the owner of the house was a government employee), but eventually, the community had to turn to renting. In other places, volunteers used churches and rooms in vocational and technical training colleges. There remained a need for more community reading centers and rooms. In Amhara, one PTSA member explained a difficult situation with school infrastructure: “Most students get sick because the school is very dusty. Once they leave to get medical help, their chance of coming back to school is very low. I think every three days, students are required to clean their classes because it is not appropriate for the teachers to bend and clean the floor, as it is tiring. So, the students are required to clean. Because of that, their clothing is not clean. They are not hygienic and are coming home with different kinds of diseases. We understand the school has no capacity to put in a proper floor.” Several kebele representatives reported that problems related to weather and basic structural conditions were barriers to effectiveness. Because children frequently studied outside, there was a need for roofed accommodations. According to one Oromia Kebele representative, a major challenge in some schools was the absence of clean water. The representative explained that when children got thirsty, they went home to find a familiar water source. This negatively affected their time investment and their concentration on reading activities. Many community activities originally had been planned to be held in the communities closest to the students, but they had to be shifted back into the schools. However, in some cases—even at the school—school principals still faced the challenge of inadequate space. In one instance, a room built by the community for reading activities became a regular classroom for eighth-grade students because the number of students enrolled that year had increased. In some communities, crowding worsened in the schools when the reading centers initially created in the community stopped functioning. Shortage of Materials. The number of SRMs was not sufficient in all places to carry out the reading activities, either because students read all available materials quickly or because some of the handwritten SRMs were illegible. Some schools had large class sizes (in some cases, more than 50 students) and therefore did not have enough Grade 3 or 4 books to distribute to students. A related challenge to program effectiveness was a lack of space for all who wanted to attend: An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 45 Older students or students with no reading difficulties who wanted to attend activities were sometimes turned away. In one Amhara woreda, officials had no transport vehicles to deliver and distribute materials to the schools and volunteers. There was also a shortage of bookshelves and reading corners with either nonexistent or broken windows. In the same region, the cluster supervisor noted, “Due to the large enrollment in these schools, the number of resources (e.g., Grade 4 books) was chronically not enough.” Home Issues. Many parents were active in their children’s reading, but in some communities, they clearly were not. An Oromia principal stated, “That lack of parental awareness of the READ CO aims was a barrier to effectiveness . . . students spent too much time in the home working with parents.” Absenteeism due to household chores and/or long distances between activities and home was also a barrier to program effectiveness in some communities. Bureaucracy and Dependency on Outside Support. Some local administrators were not aware of the purpose of the literacy activities and the role that the volunteers performed in facilitating the community reading spaces. Local administrators did not always support the procurement of reading spaces or support volunteers because some believed that they were paid workers of the READ CO project. An Oromia volunteer commented that he or she received no encouragement from the school or kebele, not even a certificate of recognition. Another issue raised by stakeholders was that while the project was somewhat effective, the project implementors placed heavy emphasis on materials distribution and report writing and less emphasis on human development, training, and incentives for community members to actively engage in the program. In the Somali region, the woredas noted considerable variability across their region in terms of effectiveness, as well as differences in activism between those schools that were participants in the program and those that were not. While it would seem positive to note that participants in the program were more active, some woreda representatives maintained that even those active participants were still overly dependent on project training, support, and materials. A Wolayto woreda representative agreed that the programs were still not owned by the local communities. When trainings were delayed, activities were slowed down or did not take place. Access and Participation Issues. One Amhara supervisor reported that because of the support that teachers and volunteers provided, changes were happening. However, the supervisor also reported that it “was not like that we expected it.” A PTSA member reported that some good things were happening but that the low number of participants in the community meant that not enough of the target population was being reached. She explained that more needed to be done to address the middle school classes and to ensure that students who lived long distances from the activity locations had access to support. Volunteers from Wolayta and Tigray reported the challenges that arose from children sharing a classroom but using it in different shifts (i.e., at different times of the day). Children from one class were not familiar with the reading corner, and they destroyed the box containing the books. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 46 IR 3. Parental and Community Engagement to Support Learning to Read Strengthened Parents, caregivers, home environments, and communities play an essential role in promoting and sustaining school readiness and performance. Research has shown that parents with positive attitudes toward education and high expectations for their children’s school performance and future positively influence children’s performance in school (Bronfenbrenner, 1992; Henderson & Berla, 1994). Children who spend time learning outside of school and children with home learning resources and opportunities perform better than children without those opportunities (Clark, 1990; Alexander et al., 2001). READ CO trained PTSA and KETB leaders to conduct awareness sessions for parents to encourage participation in and financial support for children’s reading activities both inside and outside of school. Only a few sampled parents reported that they had attended a parental awareness session. Many parents expressed interest in attending a parental awareness session, but they had not been informed when sessions would be held. Nonetheless, most parents reported increased participation in their children’s education after they saw marked improvements in their children’s reading ability as a result of the READ CO intervention. Relevance While many parents initially did not believe that READ CO was responsive to their needs and those of their children, they became convinced of the value of being engaged in READ CO activities after gaining awareness of the intervention and seeing improvements in their children’s reading. A parent explained their initial views on out-of-school reading activities as follows: “Before I understood what is going on with this project, I refuse[d] to send my children for additional classes. When the teachers sent letters from school in order to make sure [of] our support to my children, I asked why they always send me letter[s] since they are responsible to our children [at] school. After getting awareness, I myself take care of my children. I promise to cover their work at home and let them focus on reading.” As this parent’s comment indicates, before the READ CO program was implemented, some parents believed that the schools and teachers, rather than the parents, were responsible for educating children. Given the constraints that parents face, some parents valued their family’s livelihood over their child’s education—especially if their children had not been able to read even after attending schools for many years. As parents witnessed their children’s performance improve, such that they could read both aloud and independently whereas they previously could not, their enthusiasm for participating in the READ CO program grew. A KETB member in Amhara explained that parents often need their children at home to look after livestock, fetch water, or help during market days; thus, they kept their children from attending school or participating in reading activities. This KETB used school meetings to shift societal perspectives and decrease the number of children working during and after school hours. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 47 Likewise, before the READ CO intervention, some parents admitted that they did not prioritize the education of girls. One mother explained, “One of my daughter[s] who had dropped [out of] school before became [a] good student after coming back to school. I did not give equal chance to both boys and girls before. Now, I realize the importance of education. I motivate my daughter to read more.” Other parents confirmed that boys were given more time to read and write because girls were expected to spend more time doing domestic work with their mothers. While the unequal burden of housework for boys and girls was not resolved in all sampled households, parents reported that the READ CO program helped them become more aware of the effects of unequal treatment. Many of the parents in the study sample were entirely illiterate or were illiterate in their MT language because they had grown up learning Amharic and English in school. Despite parents’ limitations, they reported that they were able to participate in reading activities with their children, and they themselves were learning in the process. One parent explained, “I did not have any understanding about [the] book bank before. Even I don’t know how to read Hadiyissa. . . . But now, my children bring the SRMs from [the] book bank and asked me to read together. I myself become a good reader currently.” Across language groups, parents appreciated that READ CO provided their children with books in their MT language that taught their cultural heritage. A KETB member in Hadiya enthusiastically remarked, “The contents of the books make happy not only the children, but also the parents who would love to see their past culture and history reach to their offspring.” A parent in the Somali region confirmed that even illiterate parents are eager to engage in the SRMs because they document and transmit their language, culture, and heritage. Effectiveness KETB members, PTSA members, principals, teachers, and youth volunteers worked in coordination to engage parents in their children’s reading and education. KETB members reported various methods for engaging parents, such as visiting each household in their kebele, holding community meetings in schools and churches to discuss children’s reading progress, celebrating World Literacy Day, hosting reading competitions during Parents’ Day at school, and fundraising to fix or build school buildings and support youth volunteers and children. According to READ CO’s program design, KETB and PTSA members were trained to hold parental awareness sessions. Only a few of the parents in the sample study had attended the parental awareness session; not all of the parents had been informed of the location. This is possibly because, before the READ CO intervention, parents did not make frequent trips to the school without strong outreach efforts from school administrators, as a principal in Wolayta attested. A Wolayta KETB member agreed that they initially had a difficult time getting parents to participate, but as excitement grew around the program, “They owned the activities. They started An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 48 to say, ‘It is our project. It is for us, and it is for our children.’” KETB members and parents explained that seeing children participate in reading competitions and watching younger students read at a higher level than their older classmates increased their participation and enthusiasm. Parents’ Day9 reading competitions, attended by the broader community and sometimes attended by woreda and zonal officials, showcased the extent to which children’s reading skills had improved as a result of their participation in reading activities. As many parents, KETB members, principals, and teachers attested, the students in the lower grades who were able to participate in READ CO reading activities received more school-level rewards than students in higher grades. One KETB member stated, “[Reading competitions] increase the children’s competitiveness. After they see changes [to] their children [’s] ability, parents become very happy and ready to support.” Parents reported that they also rewarded their child’s performance in the competition with money, new clothes, and school supplies to encourage them to continue reading. The entire community— not just parents—shared in the enthusiasm. A KETB member explained: “Different community members [are] invited to observe the performance of the students. Some of the individual[s] invited completed Grade 6 and 7. When they observed students of Grades 1 and 2 who read different books without any fear as equal as Grade 6 or 7, they [are] surprised and even some [have] regrets because of the opportunity they lost.” Parents had been frustrated that, in many schools, children had advanced in grade level without learning to read and write or improving their ability to read and write. But the Parents’ Days event showcased that reading with ease and confidence was possible for their children. After observing the excitement around Parents’ Day, one KETB decided to host more competitions to encourage more parental engagement and to motivate the children to work harder to obtain more rewards. Support for Youth Volunteers. The KETB and the PTSA were responsible for leading fundraising efforts to support youth volunteers and establish reading corners, bookshelves, book banks, and reading clubs. Generally, the communities in the sample study used donated space such as churches, empty classrooms, homes, and tree shade to host reading activities. Some of the communities were still raising funds to build new infrastructure dedicated to reading activities. While some stakeholders were concerned about building adequate infrastructure for hosting reading activities, respondents expressed widespread concern about communities’ inability to adequately support and retain youth volunteers. The support provided to volunteers was widely perceived as inadequate and varying in quality by community. One youth volunteer explained that they had not received any reward for their work from the school, kebele, or woreda. However, they had greatly improved the reading ability of children in their community. Nor did this youth volunteer receive a certificate of recognition for work with READ CO. KETB members, supervisors, and woreda experts recognized the importance of youth volunteers, and they worry that high turnover rates and low volunteer morale will affect the sustainability of reading activities. Communities had varying approaches for supporting 9 Parents’ Day was also called Reading Day by some respondents. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 49 youth volunteers. A community in Wolayta collected 5 birr per month from households to pay youth volunteers, a community in Hadiyissa sponsored college tuition (600 birr per month) for 16 volunteers, and another community in Hadiyissa payed their youth volunteers nothing aside from the compensation they received during quarterly trainings. The Hadiyissa supervisor explained why they were not able to support youth volunteers: “The school do[es] not have [an] additional budget to support them. The community refuses to contribute so far. The woreda government should include [youth volunteers] in its budget plan.” Communities highly valued youth volunteers, but they were too impoverished to reimburse them adequately. Another supervisor explained that communities attempted to support youth volunteers: “The supports they get [is] not enough to motivate them for more work. The works done by youth volunteer[s] [are] incomparable to others. They should get some incentives that can motiv[ate] . . . we try to provide some financial support.” Parents similarly echoed that communities could contribute food and building materials but struggled to financially compensate youth volunteers even though they recognized that volunteers were doing valuable work. Perceived Impact Parents consistently attested that as a result of the READ CO intervention, their views on the importance of reading have changed. They now provide more support for their children’s out-of￾school reading practices, and they have become more engaged in their children’s schooling. Some parents felt that they were not able to be more engaged in their children’s schooling because of the distance between their homes and the school as well as heavy workloads, but their efforts to help their children increased nonetheless. Community Participation. Parents, principals, MT teachers, and youth volunteers believed that READ CO had fundamentally changed their community’s reading culture. One MT teacher explained that these cultural shifts were driven by children: “Parents changed, or their views changed, because of the change of their children.” Another MT teacher explained the shift in reading culture over their own lifetime: “We saw real change in children. I am comparing the reading activities today with the time we ourselves stayed in school. No one gave attention to reading at that time. Today, children read well and understand easily. Children in Grade 1 read confidently. They do it with ease. This is change. If you ask community members randomly, they will tell you the same thing. They will witness the existence of change.” As this MT teacher attested, the entire community changed from no one paying attention to everyone being aware of the strides that children were making by participating in reading activities. Raising awareness around the effectiveness of the reading activities was essential for encouraging more parents to engage with the READ CO program and encouraging their children to participate in reading activities. For example, parents began to send their children to reading activities after observing the positive changes occurring among other families. Parents already involved in the program explained that they sought out nonparticipating parents and encouraged them to allow their children to participate in reading activities rather than working. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 50 Parents became more involved in their children’s schooling after witnessing improvements in their reading skills. Teachers explained that previously it was difficult to encourage parents to come to the school. A teacher in Somali reported that through READ CO, parents came to the school to observe classrooms, participated in discussions, attended reading activities, and regularly came to collect SRMs for their children. Changes in the Home Reading Environment. Parents reported varying levels of ability to engage with their children’s reading at home. Many parents explained that they helped their children by taking on their child’s work responsibilities to give them more time to read. Other parents had follow-up conversations with children about what they had read, especially if they were reading an SRM on history or culture. Some parents read to their children, read with them, or listened to their children read, and the frequency of reading engagement varied from nightly rituals to whenever parents had time. If parents themselves did not have the time or ability to read with their children, they encouraged them to read together and help each other. Some parents remarked that their children’s behavior and attitudes toward reading had also changed. One mother explained, “[I]n the evening my children used to shout and play, but now they silently read their SRMs with concentration.” Another parent remarked that her children spent most of their time at home reading. Several parents reported that their children had friends over to read together and discuss their SRMs. While some parents reported that they had to encourage their children to read, others explained that their children read at home without prompting. One father explained, “There is no need to push them to practice reading after the program is introduced. They practice by themselves.” As these parents attested, their children enjoy reading and therefore self-motivate. Implications, Recommendations, and Prospects for Sustainability In many of the communities participating in this study, solid evidence demonstrated that READ CO had achieved many of its major objectives. Primary-age children had increased access to high-quality, appropriate SRMs and enhanced opportunities to improve their reading skills through school- and community-based learning activities. Parental observation of their children’s improvement was a powerful factor in generating increased community and parental support, which lead to the emergence of vibrant reading cultures in some READ CO communities. Various READ CO school and community endeavors contributed to the improvement of students’ reading skills. Stakeholders provided considerable evidence of increased interest and participation in reading activities as well as observable (at the local level) evidence of increased reading skill. At the same time, barriers to attaining all of the READ CO outcomes remain: lack of incentives for youth volunteers to stay engaged, failure of many communities to provide safe places for community reading activities outside of schools, and reluctant parents in some communities who were still not fully aware of the initiative or encouraging students’ participation. There were mixed results in terms of the number of SRM titles available, uneven SRM distribution, and kebele and community support for non-school-based activities. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 51 In the findings sections of this report, we presented the evidence for the satisfaction levels of key READ CO stakeholders in their own words. We also highlighted accomplishments and barriers under each of the three IRs. There were differences in the effectiveness and impact of project inputs and activities within and across communities. School-based activities appeared more successful in effectively reaching students, though this was not always the case. Understanding the complexity of READ CO implementation requires tolerance for ambiguity: Some activities were clearly more successful than others, and some approaches that worked in one context may not have worked in another. Nonetheless, the reader can glean important lessons learned that will improve the design of similar community-support endeavors in the future. In the sections that follow, we synthesize the evidence for the merit of READ CO in the core activities. We summarize the major findings under each core activity, provide recommendations for overcoming barriers to future implementation, and present a brief discussion on the prospects for sustainability of the key activities. In our syntheses, those activities for which there is evidence of merit are presented under the label of “promising” activities.” READ CO activities of lesser perceived merit are presented under the label of “emerging activities.” Before moving to those final sections, however, we briefly discuss the complex issue of perceived impact of the READ CO project on student learning. Perception of READ CO Impact Many stakeholders were convinced that the project’s endeavors positively influenced student learning outcomes. In many cases, teachers and youth volunteers cited the results of locally administered assessments as evidence to support their claims. Parents also frequently were confident in their claims that their children’s reading skills had improved. Yet other READ CO evaluations conducted by NORC with representative samples and treatment and control groups have found little evidence of impact on student outcomes that could be definitively attributed to READ CO. Grades 2 and 3 EGRA scores from the 2016 to the 2018 assessments also did not improve dramatically. How might we explain this phenomenon of what appear to be contradicting results? While the purpose of this evaluation was to better understand the merit of the project from the perspectives of stakeholder participants, and not to detect significant changes in student test scores, inevitably the plethora of studies on READ CO invites comparison, discussion, and further probing into barriers to improving student reading outcomes in Ethiopia. The fact that stakeholders frequently claimed “amazing turnarounds” that were not confirmed in other studies is not necessarily contradictory. Critics may claim these results point to a possible response bias on the part of stakeholder responders in this study (i.e., over-reporting of successes and impact), but there are other plausible explanations for the difference in evaluation results. One explanatory factor may have to do with the variability in the number of students engaged across the communities. Across all the communities in which READ CO was implemented, in a small percentage of the total, perhaps most students may have genuinely engaged with the program and learned, which may have led to positive changes in student outcomes in those schools. Across another section of schools in which READ CO was implemented, in a small percentage of the total classrooms, perhaps most students did absolutely nothing to engage with READ CO endeavors. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 52 Most classrooms, however, likely had participation rates that were somewhere in the middle. Because some communities may want to preserve an image that “almost everyone” in their schools are participating, it is difficult to obtain accurate data on actual participation rates. Variation in the quality of engagement and learning across schools could explain why test score increases, on average, are not visible in the larger studies. Further, gains may be happening in most places, but that change may be seen only in a relatively low proportion of the students in a given community; that is, when students are assessed representatively, those gains are not detectable because many of the high-growth students are not sampled. Thus, a lack of mean score increases in the impact studies could be a “scale up” (participation rate) problem both within classrooms and across communities. Nonetheless, this state of affairs does not necessarily threaten the credibility of individual stakeholders’ claims of the impact of the READ CO intervention in their own classrooms. For example, a teacher in a given school may have 60 primary students. If, after participation in READ CO activities, that teacher now has 12 to 16 children actively engaged and learning at a high level—instead of the “status quo” six to eight children that were performing well before project onset—from the teacher’s perspective, this change certainly could be considered dramatic: Twice as many students are now learning after the READ CO intervention. Teachers, youth volunteers, and parents reported their experiences in their classrooms and in their reading activities, not on national averages or the representativeness of any claims made. We do get a sense from some of the respondents that despite the enthusiasm and credible arguments for effectiveness and impact, in many communities it is not all of the children who are closely engaged and/or not all engaged actively when they did participate. Promising READ CO Activities Supplemental Reading Materials (SRMs) The findings indicated that the SRMs had been highly effective and well received. A pattern that emerged across regions and types of stakeholders is that the quality of the SRMs was perhaps considered the most important contributing factor to both motivation and increased student learning. Students found the SRMs engaging and appropriate in terms of cognitive level, language, and culture. The SRMs contributed to improving children’s reading skills. Principals, cluster supervisors, and members of KETBs and PTSAs confirmed that the SRMs were in alignment with the textbooks and MT curricula, and that they enhanced children’s understanding of their lessons. The SRMs were easy for children to carry, and the fonts, reading passages, and pictures were attractive. The stories were largely appropriate, though children expressed feelings of distress regarding stories about the death of people or cattle, car crashes or injuries, and transmitted diseases. In general, almost all the respondents who participated in this evaluation reported that the SRMs were relevant, effective, impactful, high quality, and eagerly used by students and parents alike. The distribution of SRMs was well executed in most communities and a large number of SRMs reached the woredas and the project schools. However, there were barriers to distribution. Some children were nonetheless forced to wait to borrow SRMs in some instances. Distribution of SRMs appeared to be more effective for Grades 1 and 2 than for Grades 3 and 4, due to delays in An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 53 SRM distribution for Grades 3 and 4. Further, there were signs in some rural communities that materials were not sufficient or were kept in storage and going unused because of lack of proper supervision. Figure 2 presents recommendations for overcoming barriers to SRM distribution, creation and use. Figure 2. SRMs: Recommendations for Overcoming Barriers • Plan to increase the production and distribution of the supplemental reading materials (SRMs) considered the most relevant and effective; plan for priority distribution to those regions still suffering shortages. • Identify and revise or remove passages or stories in the SRMs (as suggested by the children) that present disturbing stories or passages. • Investigate why SRMs for Grade 3 and 4 were not produced at the same efficient rates as the SRMs for Grades 1 and 2 (or not distributed on time). • Link with new literacy programs or like-minded nongovernmental organizations to continually support the provision of SRMs in the early grades. • Create a knowledge hub in which content material/SRM developers can network to take advantage of future economic opportunities to develop materials. • Discuss with regional education officials ways in which woreda education offices can continue to provide support and sustain the proper utilization of these materials. • Scale up SRM distribution to adjacent (non-READ CO) schools to reach more children who lacked the opportunity to engage with the materials to this point. • Ensure there are places in which books can be safely and easily stored in an accessible place in the community and that SRMs are not being “warehoused” and therefore inaccessible to children. Prospects for Sustainability of the SRM Activity Many communities are currently using the SRMs and have already planned to expand SRM use to improve children’s reading performance. For example, 25 schools in the Amhara region have already estimated the cost to their annual budget for printing more SRMs. Parents in one Amhara woreda expressed their willingness to extend the reading activities and to print and purchase more books by collecting 65 birr per parent. Thus far, they have collected 31,500 birr for this purpose. Parents in one Hadiya woreda raised 5,000 birr for the development and procurement of SRMs. The Sidama Zone also collected 75,000 birr for the same purpose. As many of the respondents indicated, nontarget schools also worked hard to get the SRMs and have already started contributing money. The efforts described above indicate that the communities understand the impact of these resources and the reading activities that helped their children acquire reading skills. Parents are currently engaged in providing some additional financial support to the schools for the reprinting of the existing materials or purchase of more SRMs. Some principals and woreda education experts also expressed their readiness to continue supporting SRM reprinting from the school revenues in the long term and planned to include this in their 3-year school budget plan. For example, a school in an Amhara woreda already started duplicating books using their own resources. These efforts, if supported by accountable authorities, could continuously support An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 54 SRM duplication and purchase of new SRMs. Based on the evidence, it appears that the prospects are high for sustaining the production, distribution, and use of the SRMs supported by the READ CO in the participating communities. School-Based Reading Activities School-based activities were strongly supported because of the traditional role of the school in Ethiopian society. Core school activities had a school base of support, access to teachers and principals, and alignment with the curricula and were perceived as effective and congruent with the traditional role of schooling. School-based activities were provided in a safe and convenient environment. Stakeholders preferred school-based activities because the school consistently demonstrated a commitment to sustaining the activities—a commitment that was lacking in some of the community-based activities. Youth volunteers were especially effective when working in close partnership with teachers in schools. In the school compounds, children’s dedication to the reading clubs was substantial; they had opportunities to access books in the club and a quiet, safe place to read. However, a shortage of bookshelves and the unsuitability of some reading spaces were areas that needed improvement. Overall, the evidence from the READ CO schools that participated in the evaluation indicated that the school-based activities were mostly relevant, effective, and impactful, and perceived as more relevant than the community-based activities, such as reading camps. At the same time, as noted in the findings section, there were some barriers to effective implementation of school-based activities. Figure 3 presents recommendations for overcoming those barriers. Figure 3. School-Based Reading Activities: Recommendations for Overcoming Barriers • Involve students above Grade 5 and nonparticipating school students in the student reading networks given that they currently are not taking advantage of school-based learning opportunities. • Approach community leaders with evidence of ways in which students have made reading progress under READ CO and continue to encourage their funding reading corners, SRMs, book banks, and reading clubs. • Consider the continued use of youth volunteers in support of mother-tongue language teachers. • Link community mobilization efforts with partner schools and recognized community structures, such as community care coalitions and iddirs. • Ensure that local administrators and parents are well informed about the project’s successes and failures as well as its challenges, as well as the role the school is playing. • Create greater awareness among the social and political leaders about the positive contribution of school-based reading activities. • Encourage more schools to serve as community reading resource centers for the whole community, not just their students. If the school organizes weekend literacy activities for adults/parents, circles of literacy that honor oral culture, the use of the library/school as a place for health awareness, etc. these “pull” factors will attract the community members to sustain their investment in literacy activities. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 55 Prospects for Sustainability of This Activity The school-based reading activities, such as the reading clubs and reading corners, are currently visible and well-functioning. Schools in one Amhara woreda, for example, have already included these reading activities in their annual budget plan and they believe that they will sustain these activities. Some of these community structures were confident about the sustainability of the project given that the trained teachers, the books, and the volunteers currently exist in their areas. They planned to reach out to more teachers in non-READ CO schools by sharing their experiences. Some teachers from READ CO participating schools in the Amhara region and the Sidama Zone have already trained teachers, principals, and volunteers in adjacent woredas and shared their experiences with the reading activities. Schools in a different Amhara woreda trained new teachers, principals, and volunteers aimed at proactively tackling challenges related to teacher and volunteer turnover. In some sites in the Sidama Zone, schools stored books for the new school year and finalized the reading volunteer selection process for the school- and community￾based reading spaces. They emphasized giving due attention to building the capacity of the responsible agents and sustaining the reading activities in their areas. Some respondents noted that they were committed to sustaining the reading activities, in the belief that the community has the capacity to continuously support the activities with in-kind and in-cash resources. They promised to bring books from their homes or buy new ones. A KETB member from an Oromia woreda said the society knew the benefits of reading activities. Therefore, they started mobilizing resources and built rooms for this purpose. He said this is one indicator to show their commitment in support of the reading activities. Parents in Hadiya Zone, for example, appreciated the project and said it helped their children to fluently read in the Hadiyissa language. From here on, it will be their responsibility to build upon what was already started. These community structures demonstrated their vested interest by continually supporting the reading activities through cooperation with stakeholders like PTSA, KETB, the woreda education office, and other key actors. Based on the evidence from the evaluated READ CO communities, the prospects for the sustainability of some school-based activities are high. Development of a Reading Culture Many principals, MT teachers, local administrator, and youth volunteers indicated that READ CO had fundamentally changed their community’s reading culture. One finding of the study was that students who passed through the school and community-based reading activities read well and understood the SRMs easily. Students benefited from these opportunities and from the follow-up and support of their MT language teachers as well as the reading volunteers. Receiving consistent follow-up from teachers and volunteers and having easy access to the SRMs changed the reading culture, boosted students’ confidence, and instilled a love for reading. READ CO children are currently not afraid to demonstrate their skills in different public reading competitions in the presence of their parents and other community members. The evaluation team discovered that the motivation, encouragement, and support from different actors in the READ CO areas enabled children to use the available resources properly, to increase their individual reading time, and to significantly improve their reading fluency. The program brought change in parental behaviors and attitudes as well. Parents and the community were resistant to An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 56 the objective of the program until they observed tangible changes in their children’s reading skills. Once parents saw their children’s progress, they not only allowed their children to read in their spare time but also cooperated with teachers and volunteers to borrow books from the book banks and reduce home tasks in some cases. The community, mainly parents, collaborated with the schools to construct reading centers/rooms and to furnish them with the necessary equipment, such as mats and bookshelves. In some areas, parents raised money to ensure the reading centers were neat and child friendly. Parents’ attention toward helping their children with reading activities at home was also very encouraging: There was a common understanding of the importance of giving equal attention to girls and boys. Teachers promoted the necessary guidelines for students to take good care of the books both in the school and at home. They created a culture that could last and that could be transferred to the younger students. An MT language teacher from an Oromia woreda explained, “We oriented students to cover the books, to put them inside their bag while taking them to their home and to keep them safe from moisture to increase the service year of the books.” Based on the evidence from the participating communities, it appears that children were clear on the importance of reading for their learning in other disciplines. They were cooperative and happy to engage in reading activities. Both the in-school and out-of-school stakeholders shared the same feelings and mobilized resources to encourage students to improve their reading skills by developing the reading habits introduced by READ CO. At the same time, as noted in the findings section, some barriers to developing a reading culture persist. Figure 4 presents recommendations for overcoming barriers to creating a reading culture. Figure 4. Development of a Reading Culture: Recommendations for Overcoming Barriers • Continue to focus on parental participation in reading celebrations and events; ensure that parents are aware of successes and challenges. • Continue organizing evidence-based awareness creation programs on the importance of parental engagement for the success of reading activities both at home and at school. • Continue recognizing students who are making a continuous effort to improve their reading skills. • Consider strengthening school reading clubs in which targeted children can continue to improve their reading skills and extend the practice to new students. • Strengthen child-friendly school libraries in which both targeted and nontargeted children have access to supplemental reading materials and engage in reading. • Assign responsible and accountable teachers to oversee libraries and support reading clubs. • Expand the support base for reading clubs to access reading materials through different means, such as organizing a book mobilization day. • Strengthen child-to-child support schemes to continuously expand the reach of the service to new school and community children. • Consider strong media involvement in the promotion of the reading culture in the country. • Involve book publishers with book fairs that offer discounted prices, as well as radio and television programs that facilitate reading and reading circles. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 57 Prospects for Sustainability of This Activity The project’s main purpose was to raise community and parental awareness about the importance of children’s reading. In creating a reading culture, it supported early grade students in accessing SRMs in their school, home, and community. A parent from Hadiyissa Zone Misha woreda supported this idea: “Everything should start at home. If parents give more attention to their children’s education, the children become more successful. I understand what my role should be. I give equal attention and chance to all of my children regardless of their gender difference.” With respect to the proper handling of SRMs, the students generally know the management of the books and are disciplined. Teachers believed this will increase the service year and utilization of the books long after the project has phased out. A principal from Oromia Zone said: We plan to make monthly competitions on their reading abilities, reading frequencies, and accuracy of reading and writing and reward them based on their performance in front of stakeholders. This is to motivate the competitors in particular and all students in general. Based on the evidence from these READ CO project schools, this component of the intervention was relevant and somewhat effective. Originally, there were expectations from the community that the woreda education office would take on all the responsibilities in organizing the activities after the project phased out. However, many parents now realize that they need to be responsible for supporting their children at home and the volunteers to continue their engagement in supporting reading activities. An MT language teacher from an Amhara woreda explained: … if it is not possible for the project to continue, we have to continue the work by ourselves. We have seen the progress in the students, and just because there isn’t a project anymore, we should not be careless. We should all take the responsibility…. We have an assignment. Our assignment is to continue the good work that has been introduced to us by READ CO. These participating actors who have seen the change in the reading culture of the children, reading improvements, and the support of parents and other stakeholders in the community are highly motivated to keep the change ongoing. As discussed, there was community preparedness and willingness to continue supporting the children. Community members seemed determined to follow the track of the project in mobilizing money, printing/duplicating books, buying more books, and contributing money for volunteers’ incentives. The constructed reading centers, and the changes observed in students’ reading skills created high interest to sustain the reading activities, such as monthly competitions, rewarding outperforming students, and engaging community volunteers and MT teachers in the reading activities. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 58 Clearly, many stakeholders perceive a positive change in parental dispositions, attitudes, and behaviors. However, challenges to sustaining the culture of reading remain. On the one hand, there were many examples of parents accepting new responsibilities and eager to “see the work on.” On the other, there was evidence that many community members may still see the education of the child (including their own) as being squarely in the purview of the school system. Many cannot afford to provide the extra monetary support to sustain an out-of-school component. While parents have READ CO clubs and camps to send their children to, they may be supportive; over time, over, the opportunity-cost calculation may be altered once the READ CO project is no longer directly supporting the school and the community—and no longer a motivating, structured force in the community. Thus, the prospects for long-term sustainability of the reading culture remain moderate. Capacity Development of Stakeholders The project built the capacity of the KETB, PTSA, reading volunteers, MT language teachers, parents, woreda education offices, supervisors, and school principals to support reading activities both in the school and in the community. Most expressed satisfaction with the training events and consistently noted their relevance and effectiveness for the implementation and monitoring of the reading activities. The training built their capacity to implement the reading activities, assess children, and develop and use SRMs. The trainings equipped stakeholders to support children with reading difficulties. Stakeholders learned to use the forms and procedures for lending books from the book banks and reading corners. The evaluation team witnessed the uniformity of project approaches in all the sites, especially for the school-based activities. Apart from facilitating capacity-building trainings, in selected sites, the project provided office equipment for the printing, duplication, and lamination of school￾based SRMs. Such centers were regarded as a center of excellence, not only for targeted schools but also for nontarget schools in their respective areas. Based on the evidence, it appears that the capacity development trainings and material support were relevant and effective, and they seem to have made an impact on participant stakeholders’ support for children with reading difficulties. However, Grades 1 and 2 language teachers received more investment and support than their Grades 3 and 4 MT counterparts. At the same time, as noted in the findings section, there were challenges to capacity building. Figure 5 presents recommendations for overcoming barriers to capacity building. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 59 Figure 5. Capacity Development of Stakeholders: Recommendations for Overcoming Barriers • Consider providing refresher training and ongoing coaching for those involved in locally based and school-based SRM development. • Strengthen the linkages and working relationship between mother-tongue (MT) teachers and reading club volunteers by creating opportunities to continuously share ideas, exchange experiences, and support one another. • Build the capacity of MT language teachers (Grades 1 through 4) in reading activities related to instructional improvement in literacy. • Encourage the trained teachers to share their knowledge and experience with new teachers in the schools to minimize the effect of personnel turnover. • Enhance the ownership of schools’ reading activities by integrating those with the schools’ regular activities. Prospects for Sustainability of This Activity According to the community and government stakeholders, the capacity-building support provided for in- and out-of-school communities will likely sustain some—though not all—of the reading services in the project target areas. The mentoring and coaching support provided by the READ CO team, woreda supervisors, and woreda education office experts already has succeeded in creating a culture that promotes reading activities in many of the project sites. Monitoring and follow-up indicators regarding the reading activities are already included in the monitoring and follow-up checklists of woreda supervisors and woreda education office experts. Woreda supervisors aim to replicate good practices and find solutions for challenges. The strengthened monitoring and reporting activities likely will continue after the end of the project. The evaluation also revealed that there is improved capacity of KETBs and PTSAs to oversee and support the reading activities and to mobilize resources to address funding shortages and other problems. However, the capacity of the KETBs and PTSAs has not been uniformly evident. In many communities, the kebele and other local officials appeared to have some of lowest capacity; they failed to engage with youth volunteers and support their initiatives. The continued capacity-building and follow-up processes of the project facilitated a smooth working relationship between MT language teachers and reading volunteers in many—though not all—communities. These communities cascaded the skill to new areas because of the capacity￾building trainings they received from the project and from their own experience in implementing reading activities. The extensive efforts invested in the community and in the schools are making a difference, but not consistently throughout the target areas. In those areas in which the project was implemented as intended, community members have demonstrated a willingness to support the activities by raising financial and material resources. These successes might continue after the close out of the project if the accountable offices play a leadership role in pushing it forward and strengthening it through their own contribution and those of stakeholders. An important barrier to sustainability noted by many was the constant turnover of teachers and other personnel. Many leave the activities after receiving training, and An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 60 their replacements are not trained adequately. There is a concern that unless the existing team of experts in the community share and expand their knowledge and skill to newcomers, and unless the government office and school leadership continues ensuring its utilization, the community reading activities might not last. Emerging READ CO Activities Youth Reading Volunteers Volunteers were a unique part of the intervention and played a key role in the success of the implementation. In many areas, after building capacity through training and mentoring, READ CO assigned the volunteers to work in coordination with principals, MT teachers, and cluster supervisors. The volunteers believed the training was relevant and equipped them to support children with reading difficulties. These youth volunteers were mainly in charge of community￾based reading activities and sometimes facilitating reading clubs in schools. Teachers provided advice and mentored them on ways to work with the students both inside and outside of school. In return, volunteers added substantial value to the work that reading teachers were doing in schools. This created a smooth relationship between youth volunteers and teachers in many schools. It meaningfully contributed toward a fundamental change of reading culture in the target communities. Many teachers reported the effectiveness of their endeavors when their collaboration with their youth volunteers was excellent. Woreda officials also noted the support of the volunteers and collaboration with them as a crucial factor in explaining the effectiveness. However, the data also revealed that the project has a poor motivation and reward system for these reading volunteers. In many kebeles, officials made little or no effort to procure the necessary facilities to support community reading camps and book banks. Thus, the support provided to volunteers was widely perceived as inadequate and varied widely by community. Further, in some cases, there was little or no public recognition of the volunteers’ important contribution or even a certificate to honor their commitment. Volunteer support for students is thus best perceived as relevant but only effective and impactful in the communities in which kebeles, PTSAs, and schools supported volunteers with strong recognition and with professional and logistical support. In other communities, the youth volunteers essentially disappeared from the horizon. Figure 6 presents recommendations for overcoming barriers to successfully deploying volunteers. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 61 Figure 6. Youth Reading Volunteers: Recommendations for Overcoming Barriers • Support youth volunteers by facilitating different motivational schemes and rewards. • Provide a certificate of service completion for volunteers who served an agreed-upon period in recognition of their effort and commitment in the presence of local government and community groups. • Organize volunteers’ recognition events and celebrations of a volunteers’ day. • Increase pressure on kebeles and other community institutions to support the youth volunteers with safe and convenient rooms (rent free) for reading camps. • Discuss publicly ways in which communities can incentivize and retain youth volunteers and coordinate volunteers’ incentive systems with existing community activities, such as food for work programs in areas in which the project is functional. • Consider developing income-generating initiatives and projects to support youth volunteer services (nonprofit small businesses, such as cafeterias). Prospects for Sustainability of This Activity The community—especially school principals, woreda education office experts, and woreda supervisors—strongly believed that the volunteers need to be incentivized to stay engaged. A school principal in the Tigray region supported youth volunteers’ participation in employment opportunities in their kebele. Another principal from Wolayta planned to support youth volunteers from the school revenues and to include an incentive scheme in their school’s annual budget plan. He was determined to design a motivational strategy for the volunteers. On the same note, a woreda expert from Hadiya was planning to include the youth volunteers in the government employment system. He discussed the issue with the head of the woreda education office and has promised to permanently hire all youth volunteers as adult education teachers. In one woreda of Amhara, volunteers received formal jobs to keep them motivated to continue providing their reading services. The community also contributed money and incentivized volunteers by facilitating different scholarships to advance their education. All these efforts are to keep the involvement of capacitated volunteers in their activities, even after the end of the project. These endeavors are encouraging and provide a modicum of hope for the sustainability of the activities after the READ CO project ends. Community-Based Reading Activities Community reading activities such as reading camps were not as effective as the school-based reading activities. The activities provided opportunities for students to practice their reading in some, but not all, communities. Some teachers and volunteers believed these activities were beneficial because they were free from the domination of active students and strict classroom environments. They argued that children who were afraid to participate in classroom activities could better participate in the community-based literacy activities and successfully learn to read with the help of the youth volunteer facilitators. Others disagreed with this perspective and stressed that the reading camps and sometimes book banks were less successful. The support from the local administrations for locating reading spaces and supporting youth volunteers with incentives was limited. The community work was being done An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 62 in rental houses, in churches, under tree shade, and in vacant rooms with no chairs, mats, or proper roofing or windows. In many cases, community activities were less favored by students due to untidiness and inconvenience in the facilities used. The reasons for poor results could be the full dependency of the approach on youth reading volunteers and committee members in different structures created or strengthened for this purpose, as well as lack of strict procedures on the part of the project to make those involved accountable for their activities. Figure 7 presents recommendations for overcoming barriers to community-based reading activities. Figure 7. Community-Based Reading Activities: Recommendations for Overcoming Barriers • Engage accountable community administrations and structures to oversee and closely support community reading activities. • Sign memoranda of agreement with local administrators, laying out expectations of them to support this endeavor and facilitate their procurement of reading spaces to solidify their support. • Align supervision, space identification, and resource mobilization with strong and dominant community structures. • Create more awareness within the community to promote commitment to supporting reading camps and book banks. Prospects for Sustainability of This Activity These activities were dependent on reading volunteers who had provided their services with the expectation of some form of incentive from the project, the government, or the community. There were some motivating factors in the project, such as trainings and review meetings. As some of the experts mentioned, community reading activities may not continue unless the government or the community offers incentives to volunteers. These factors will affect the sustainability of community-based reading activities. A Tigray woreda expert said: “The school-based activities will be sustained and will not be impacted if the project ends, but the community-based approaches will totally be impacted because volunteers might not continue working after the end of the project. Government/local authorities need to encourage the volunteers to continue their service either by allocating some budget for transportation/phone or encourage them through job creation opportunities.” Mother-tongue language teachers, supervisors, and PTSA members in the Wolayta Zone and in the Amhara and Oromia regions doubted whether woreda experts and READ CO focal persons gave due attention to sustainability of the community work. In some areas, KETBs and woreda education offices have not yet integrated reading activities within the regular activities of the community and administrative structures. They did not even start discussing with the kebele administration when and how to obtain reading spaces in the community. If so, they argued, more effort would have been taken toward the sustainability of community-based activities by this time. As a solution, the experts suggested maintaining the available books in as good a condition as possible and passing them on to the next school generation. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 63 However, there were some principals who planned to use their school grant budget to support community activities. In one community, it was not yet clear whether their plan would be practical after the end of the program. Nothing was mentioned regarding reaching an agreement with the rest of the committee members or the school principals. People were satisfied with what they witnessed, but their determination to sustain these activities seemed vague. The project was challenged to sustain community-based activities in comparison to the prospects of the school￾based activities. There were differences, however, between the reading camps and book banks, which frequently found a home location or were established in schools instead of the community. Locally and School-Based SRM Production Because of the grassroots SRM development skills that MT teachers and youth volunteers received from the project, they prepared additional materials following the project’s material development guidelines. The material production was accomplished by collecting poems, proverbs, stories, tales, and other oral literature. Selected MT teachers, volunteers, students, and other community members participated in the development process. For example, teachers in the Amhara region collected poems from elderly people for collation and use. The community forgot some of the poems, but due to the efforts of capacitated and responsible actors, they are now well documented and ready for students to read. Teachers in the Hadiya Zone also supported children to prepare their own stories and made them available for other children to read in the reading clubs. In the preparation process, they prioritized issues such as agriculture, local food, and the environment. The project provided school cluster resource centers with a computer, laminating machine, and printers. Mother￾tongue teachers and others used these resources for materials preparation. This showed how the community and schools used the technical and materials support they received to realize and retain the expected changes. In some regions, however, stakeholders stopped developing their own stories when they received READ CO books. In one Amhara region, a teachers said, “… during our training, we were told to write short stories that would reflect our community. But once the READ CO books arrived, we got distracted and focused only on the READ CO books.” Government stakeholders also actively participated in supporting the story collection process, by documenting and facilitating the printing of local stories. Where there was a need to scale up to nontarget schools, the woreda education office took the lead role in organizing events and forums for sharing experiences. Students also participated in the collection of stories and tales from the community, according to a teacher from Oromia. However, these materials were not developed in the anticipated quantity. Moreover, in some woredas, it was noted that the many of the handwritten SRMs were not accessible to children because they were written in poor handwriting and therefore difficult for children to read. Based on these and other barriers presented in the findings section, Figure 8 presents recommendations for overcoming barriers to locally based SRM production. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 64 Figure 8. Locally and School-Based SRM Production: Recommendations for Overcoming Barriers and Sustaining Successes • Recognize the efforts of those involved and encourage schools to follow the guidelines for the preparation of school-based SRMs. • Strengthen reading clubs through alumni project beneficiaries with close support of mother￾tongue language teachers. • Create support groups and mobilize material and financial resources to provide paper, pens, markers, and pencils to support locally and school-based SRMs preparation activities. • Facilitate experience sharing events with other schools. Prospects for Sustainability of This Activity The provision of SRMs is critical for sustaining both school- and community-based reading activities. Sustaining these activities becomes more realistic when MT teachers, youth volunteers, students, story writers, and other members of the school community participate in the preparation and production of local SRMs. Although these efforts did not show conspicuously in this evaluation, there were encouraging signs that they need to be sustained. A woreda education office expert from Wolayta indicated that children were accessing and effectively using different SRMs prepared at the school level. An MT teacher in Oromia noted that they received training and prepared SRMs by collecting proverbs, stories, and sayings from elders and other members of the community. Students also participated in the collection of stories and tales. School principals and non-language teachers also participated in the preparation of local SRMs. A principal from Oromia stated: “Mother-tongue teachers prepared about 30 SRMs, while other teachers produced about 50 SRMs. Volunteers also prepared about 18 SRMs.” In an effort to sustain the SRM preparation, as mentioned earlier, the project supplied school cluster resource centers with computers, laminating machines, and printers. These materials were intended to be used by those project schools in that cluster. However, it appeared from the data that some members of cluster schools were not using these resources and instead turning handwritten materials into laminated print materials. Better access to these resources and knowledge about using them may encourage schools to use them effectively and sustain materials preparation activities. Government Ownership and Engagement Government ownership and engagement is one of the pillars of the project’s implementation. From the government schools up to the Regional Education Bureau, there are positive attitudes toward the project activities. Woreda education office experts and cluster supervisors actively participated in the implementation and evaluation processes. The government structure was embedded in the school and the communities in which the reading activities were ongoing. Woreda education office experts, cluster supervisors, and KETB members played a key role in coordinating, supporting, and supervising the implementation process and facilitating cross￾learnings with nontargeted schools. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 65 The project succeeded in different aspects of its activities because of the close support of these government stakeholders, mainly woreda education office experts and supervisors. They were actively engaged in the resource mobilization efforts and supporting principals in budgeting and using available resources for the duplication, purchase, or preparation of school-based SRMs. In most regions, it is expected that these offices will take the lead role in sustaining the project activities. However, it was not clear whether or not woreda experts were always accountable for the success of the reading activities. In some places, they were involved; in other places, they seemed like reluctant participants due to a perception that this was not “their responsibility” or due to a lack of resources. As noted in the findings section, there were barriers to this activity. Recommendations for overcoming these barriers are presented in Figure 9, below. Figure 9. Government Ownership and Engagement: Recommendations for Overcoming Barriers and Sustaining Successes • Provide close supportive supervision and continuous and tailored capacity-building effort, which are essential for stronger engagement of local structures, such as KETBs and PTSAs. • Consider strengthening existing woreda-level systems and structures for accountability in resource mobilization; effective use of mobilized resources; and follow-up of reading activities, both for the in-school and out-of-school activities. • Ensure that local education offices take full responsibility for sustaining these efforts and strengthen their accountable structure toward sustainability of reading activities. Prospects for Sustainability of This Activity Government stakeholders raised community awareness at the school, at both the community and woreda levels, to sustain reading activities. To ensure sustainability, education office experts incorporated indicators within their monitoring and follow-up checklists and in their annual plans. Following up the activities, using task checklists and receiving reports from supervisors that included reading activities, were considered key strategies for ensuring sustainability. In some sites, the office also hired reading volunteers to support adult education, a community library, and early childhood teaching-related activities. Thus, stakeholders were attempting to address concerns regarding volunteers’ incentives. Woreda education offices still seek support from different stakeholders to continually support volunteers, print SRMs, regularly follow up with the monitoring of these reading activities, and scale up these efforts to more schools. In most of the sites visited by the evaluation team, members observed commitment and involvement from education office experts. In some areas, however, the community was uncertain about the future of the reading activities: Amhara Region, North Wollo Zone, Lasta and Woldiya woredas, South Gonder Zone, Dera woreda, Oromia Region, Bale Zone, Gore woreda, Tigray Region, and Tselemti woreda. In these areas, the government offices were not sure of their capacity to sustain reading activities. They said the schools did not have adequate income to follow up or to sustain the reading activities. Activities would not be as effective as they had been before, due to the turnover of cluster supervisors, teachers, and youth volunteers. Therefore, unless new programs continued to build on efforts already started, the community might forget its commitments, and the books might run out. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 66 Conclusion The evidence from this evaluation indicates that key READ CO activities have high prospects for sustainability. Many communities have already started to invest resources and demonstrate commitment to sustaining school-based reading activities and SRM development. Other endeavors, such as community-based activities that require local governments to share in the costs, may need more time to evolve and become institutionalized in a broader spectrum of communities. READ CO achieved several outcomes of particular merit in the schools sampled. First, the production and distribution of high-quality SRMS was a success (IR 1). The use of high-quality SRMs has been credited with increasing both primary student motivation and learning outcomes across the regions. Stakeholders of all types noted the high quality of the SRMs, and communities are investing their limited financial resources to ensure the sustainability of their use and further distribution. Despite barriers to reaching all students through SRM distribution in some communities, the relevance, alignment, and effectiveness of these materials should be considered among the greatest successes of the READ CO project. Also of particular merit, teachers, principals, parents, volunteers, and woreda specialists were able to cite specific evidence of the impact of the READ CO program on student reading outcomes. Stakeholders embraced new tools for diagnostic assessments, teaching, and monitoring student progress in school-based and out-of-school activities. Teachers and volunteers incorporated new ideas about the benefits of mixed reading groups and the inclusion of girls and the weakest readers within targeted activities. Credible evidence indicated that school-based reading activities—in particular, reading corners and clubs— motivated students and parents to participate actively in reading endeavors, which promoted the development of a reading culture in many communities. Book banks and other book-lending systems were effective and impactful across the communities. Despite the merit of the project activities and successful attainment of outcomes for several IRs, there are lessons for improvement to be learned from the challenges that emerged during the implementation of READ CO. USAID’s Ethiopia Education Office, local governments, and practitioners in community development need to invest more thinking into how to incentivize any volunteer work that is planned as a key dimension of a community development activity. Volunteers were highly effective in some communities but not uniformly so, as discussed earlier in the report. Although teachers and volunteers in many communities collaborated closely— which had a multiplier effect on student learning—there was considerable variability in successfully keeping volunteers from leaving their positions. Many stakeholders clearly felt that schools were the most culturally appropriate place for education initiatives and did not necessarily believe that institutions outside of schools should be playing a strong role in education. Whether or not investment should continue to be made to reorient this culture of “school centered” thinking is a question worthy of further discussion. Finally, regarding government ownership of READ CO initiatives, woreda education office experts and cluster supervisors actively participated in the implementation and evaluation An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 67 processes. They took part in student assessment activities, observed and monitored community reading activities, and occasionally led instruction and even read to children. They were engaged in the resource mobilization efforts and supported principals in budgeting and using available resources for the duplication, purchase, or preparation of school-based SRMs. However, government support was mixed, ranging from very strong in some communities to somewhat weak (in the kebele committees), especially for activities that required government contributions, such as spaces for reading and the provision of incentives for youth volunteers. As participants of the evaluation indicated, one of READ CO’s biggest shortcomings was the failure to work with local government to find appropriate facilities for conducting community activities, such as the reading camps and book banks. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 68 References Abeberese, A. B., Kumler, T. J. & Linden, L. L. (2011). Improving reading skills by encouraging children to read: A randomized evaluation of the Sa Aklat Sisikat reading program in the Philippines. National Bureau of Economic Research. Alexander, K., Entwisle, D., & Olson, L. (2001). Schools, achievement, and inequality: A seasonal perspective. Educational Evaluation and Policy Analysis, 23(2). American Institutes for Research. (2019). Early Grade Reading Assessment (EGRA) 2018 endline report. USAID Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation. Washington, DC: Author. Borkum, E., He, F., & Linden, L. (2012). The effects of school libraries on language skills: Evidence from a randomized controlled trial in India. National Bureau of Economic Research. Bronfenbrenner, U. (1992). Ecological systems theory. In R. Vasta (Ed.), Six theories of child development: Revised formulations and current issues (pp. 187–249). London, England: Jessica Kingsley Publishers. Bus, A., van IJzendoorn, M., & Pellegrini, A. (1995). Joint book reading makes for success in learning to read: A meta-analysis on intergenerational transmission of literacy. Review of Educational Research, 65, 1–21. Coleman, James S. Equality of Educational Opportunity (COLEMAN) Study (EEOS), 1966. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2007-04-27. https://doi.org/10.3886/ICPSR06389.v3. Chianca, T (2008). The OECD/DAC criteria for international development evaluations: An assessment and ideas for improvement. Evaluation in International Development. retrieved from: http://evaluation.wmich.edu/jmde Clark, R. M. (1990, Spring). Why disadvantaged children succeed. Public Welfare, 17–23. Glewwe, P., Kremer, M., & Moulin, S. (2009). Many children left behind? Textbooks and test scores in Kenya. American Economic Journal: Applied Economics, 1, 112–135. Glewwe, P., Kremer, M., Moulin, S., & Zitzewitz, E. (2004). Retrospective vs. prospective analyses of school inputs: The case of flip charts in Kenya. Journal of Development Economics, 7(4), 251–268. Henderson, A., & Berla, N. (1994). A new generation of Evidence: The family is critical to student achievement. National Committee for Citizens in Education. Miles, M. B., Huberman, M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (Third Edition). Los Angeles, CA: Sage. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 69 National Opinion Research Center (NORC) at the University of Chicago. (2014). Out-of-school parental and community involvement interventions: Literature review. USAID Reading and Access. Nye, C. T., & Herb, Schwartz, J. (2006). Approaches to parent involvement for improving the academic performance of elementary school age children. Campbell Systematic Reviews, 2(4). Patton, M. Q. (2008). Utilization-focused evaluation (4th Edition). Thousand Oaks, CA: Sage. Scarborough, H., & Dobrich, W. (1994). On the efficacy of reading to preschoolers. Developmental Review, 14(3), 245–302. Senechal, M., & Young, L. (2008). The effect of family literacy interventions on children’s acquisition of reading from kindergarten to grade 3: A meta-analytic review. Review of Educational Research, 78(4), 880–907. Sheldon, S. B., & Epstein, J. L. (2005). Involvement counts: Family and community partnerships and mathematics achievement. Journal of Educational Research, 98(4), 196–206. Toomey, D. (1993). Parents hearing their children read: A review. Rethinking the lessons of the Haringey Project. Educational Research, 35(3), 223–236. Trines, S. (2018, November 15). Education in Ethiopia. World Education News and Reviews. Retrieved from https://wenr.wes.org/2018/11/education-in-ethiopia United States Agency for International Development. (2018). Improving reading in Ethiopia [Fact sheet]. Retrieved from https://www.usaid.gov/sites/default/files/documents/1860/Ethiopia-Fact￾Sheet_Improving-Reading_Oct-2018.pdf Weiss, H. B., Little, P. M. D., Bouffard, S. M., Deschenes, S. N., & Malone, H. J. (2009). The federal role in out-of-school learning: After-school, summer learning and family involvement as critical learning supports. Washington, DC: Center on Education Policy. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 70 Appendix A. The READ M&E Project The Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation (READ M&E) project is one of four READ projects supported by the U.S. Agency for International Development (USAID) mission in Ethiopia.10 The core tasks of this five-year project are to conduct assessments of early grade reading outcomes and provide monitoring and evaluation technical support for the portfolio of USAID READ projects. READ M&E, carried out by The American Institutes for Research, works in partnership with the Ethiopian Ministry of Education (MoE) and the National Educational Assessment and Examinations Agency (NEAEA) in close collaboration with the regional state education bureaus (RSEBs) and education stakeholders throughout seven regions in Ethiopia. READ M&E supports the USAID READ projects through (a) conducting the EGRA at Grades 2 and 3 for monitoring purposes, (b) conducting evaluations of READ projects at key intervals to determine whether implementation is on track and outputs are leading to results, and (c) conducting summative evaluations of READ projects to determine overall project results. These endeavors provide information to stakeholders about how well USAID investments across the READ projects are producing the desired changes and how implementation can be improved. 10 The American Institutes for Research (AIR) implements READ M&E in Ethiopia. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 71 Appendix B. Data Collection Tools Focus Group Discussion Protocol for Students Facilitator’s name ____________________________________________________________ Note taker’s name ___________________________________________________________ FGD’s Code _____________________________________ Cell phone of the facilitator _________________________ Background and Informed Consent My name is (name of the facilitator) and I’ll be discussing with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s Focus Group Discussion (FGD) is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording the discussion to help capture what transpires in the discussion. All information collected will be treated confidentially. The names of the participants will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary. If you have questions about the FGD, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as participants of the discussion, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The FGD will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the FGD? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 72 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Name of the school _______________________________________________________ 6. Name, Sex, Age, and Grade: No. Name Sex Age Grade 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Guiding Questions 1. How many of you have read any of these books? Please raise your hands. (Facilitator: Please show them the different SRMs prepared for the language and count the number of children who have read the SRMs) 1.1. From where do you get these SRMs? 1.2. Where do you read? (Facilitator: Please probe for the different types of literacy activities like reading corner, reading club, book bank, and reading camp). 1.3. How often do you get these materials? 1.4. Can you take them home? 2. Please, identify three books you like most? Facilitator: Please, let them identify three of their favorite SRMs, what their reason is, and ask for the characters of at least one of the SRMs) 3. Are there books that you do not like? Why? (Facilitator: Please, let them identify the SRM which is least favored with the reason) 4. Tell us about the literacy activities and what they do for you. 5. How do you know that you have improved your reading skill? An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 73 6. In which of the literacy activities do you participate the most? (Facilitator: Please probe for the different types of literacy activities like reading corner, reading club, book bank, and reading camp). 7. Which one is more useful? (Facilitator: Please name each of the learning activities at a time and ask how many of them participated in it and count and identify the literacy activity which is favored by most children and literacy activity favored by small number of FGD participants and ask children WHY they favored most and least.) 8. What do you think should be done for you to continue participating in these activities? 9. Those of you who get help from members of the family in terms of accessing and reading these SRMs, please raise your hands. (Facilitator: Please count the number of children who get help from members of the family and probe for which member of the family provided them with support and the kinds of support they received if any). 10. Who helps you from members of the community while reading the SRM? Please raise your hands (Facilitator: Please count the number of children who get help from members of the community and probe for which member of the community provided them with the support) Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 74 Focus Group Discussion Protocol for Parents Facilitator’s name ____________________________________________________________ Note taker’s name ___________________________________________________________ FGD’s Code _____________________________________ Cell phone of the facilitator _________________________ Background and Informed Consent My name is (name of the facilitator) and I’ll be discussing with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s Focus Group Discussion (FGD) is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording the discussion to help capture what transpires in the discussion. All information collected will be treated confidentially. The names of the participants will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary. If you have questions about the FGD, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as participants of the discussion, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The FGD will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the FGD? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 75 Demographic Data 1. Region__________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Participant’s Name, Sex, Age, and Educational Level: No. Name Sex Age Educational Level 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Guiding Questions 1. What community-based reading activities have you noticed in your community? (Facilitator: Please probe for the availability of book bank and reading camp 2. Who coordinates them? 3. Those of you who attended READ CO community awareness activities on children’s improved reading, please raise your hands. Facilitator: Rate: Above average, Few, None and ask why, if the majority of them didn’t attend community awareness activities. 4. In what way were you involved in the community-based literacy activities? 5. Facilitator: Please show parents all SRMs developed for the specific MT language and ask them turn by turn to name how many of the SRMs they are familiar with and ask their opinion about accessibility of these SRMs. 6. Those of you who have ever read the SRMs with your children, please raise your hands. OR those of you who have let your children to read the SRMs at home, please raise your hands. OR those of you who have let your children read the SRMs to you, please raise your hands. Facilitator: Count and Rate as Above average, Few, None and ask why, if the majority of them didn’t? Probe also for whether boys and girls get the same kind of support. 7. How have you brought positive changes in supporting your child’s reading? Can you describe these changes? What causes these changes? An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 76 8. How do the school or community-based reading activities support your child’s reading skill improvement? Facilitator: Please probe for whether the book bank lends SRMs for children, if reading camps/community reading spaces are arranged, how reading clubs involve students etc. 9. Describe any changes you observed in your children’s reading habit. (Facilitator: Please probe for changes in reading skill and time they spend on reading, interest to participate in reading activities, and how do they know that there are changes). 10. How does the school and parents cooperate in terms of supporting children’s learning to read? How can such collaboration be better organized in the future to enhance children’s learning to read? 11. How can parents continue supporting children’s learning to read? (Facilitator: Please probe for how parents would help children to read in the future) Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 77 Focus Group Discussion Protocol for Mother-Tongue Teachers Facilitator’s name ____________________________________________________________ Note taker’s name ___________________________________________________________ FGD’s Code _____________________________________ Cell phone of the facilitator _________________________ Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s interview is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the interview, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the interview? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 78 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Name of the school _______________________________________________________ No. Name Sex Age Qualification 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Guiding Questions 1. Would you please describe the types of literacy activities available in the school? 2. What roles do you play in these activities? Facilitator: Please probe for their roles. 3. From where do you get SRMs? Who provides the materials? 3.1. Which of these SRMs have you read to your students or let your students read? Facilitator: Please give them the folder that contains the title of the SRMs and ask the participants to show the ones they used. Count and Rate as Above average, Few, None and ask why, if the majority of them didn’t? 3.2. What can you say about the appropriateness of the SRMs in terms of age/sex/culture/grade level? 3.3. Which of these SRMs are most favored by students? Why? 4. How do students participate in the school-based literacy activities? 5. How do the students use the available reading resources? 6. How are the literacy activities run by the school effective in helping children learn to read? Facilitator: Please probe for the reasons and also ask whether there are changes in children’s reading skill, independent reading time, interest to participate in the literacy activities, and children’s understanding of the importance of reading. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 79 7. Do you have information about the reading spaces and other community-based activities run in the community? 7.1. Who coordinates and runs these activities? 7.2. How does the community-based reading activities complement the classroom reading instruction? Facilitator: Ask participants about the effectiveness of such approaches in helping children learn to read. 8. How do you work with youth volunteers in the school? Facilitator: Please ask for an example of successful collaboration between mother-tongue teachers and youth volunteers. 9. How do parents, other than members of PTSA, collaborate with the school to support children’s learning to read? Have there been changes in parents’ support towards their children’s reading? How do you know? 10. What changes have you observed as a result of READ CO’s school and community-based reading activities? Facilitator: Please probe for the positive impacts of theirs and the involvement of other community members and the impact such approaches have at the school, family, and child levels. 11. How can any positive changes as a result of READ CO interventions, be strengthened and sustained? What will happen to the activities if the project ends? 11.1. What would you suggest to make both in school and out of school reading activities be more successful? 11.2. What support is needed to ensure and sustain the impacts of these reading activities? Facilitator: Please probe for the strategies that can be utilized to reach more children and to avail more reading materials in the future. 12. What are the challenges in implementing reading corner and reading clubs? What measures have you taken to deal with the challenges? Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 80 Interview Protocol for Youth Volunteers Interviewer’s name ___________________________________________________________ Interviewee’s Code _______________________________ Cell phone of the interviewer ________________________ Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s interview is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the interview, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the interview? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 81 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Name of the school _______________________________________________________ Facilitator’s Name Sex Age Educational Level Facilitator Service Year Reading Camp Reading Club Book Bank Guiding Questions 1. In what reading activities have you involved in to enhance children’s reading? Interviewer: Please probe for how they deliver their services, what motivates the volunteer to engage in the activities, and the number of hours they volunteer per week. 2. Have you received training that would help you to facilitate reading activities? Interviewer: If yes, please probe how effective the training was and how it changed his/her way of thinking. 3. Where do you get SRMs and other inputs (if any)? Interviewer: Please probe types/titles /appropriateness of SRMs to the needs and reading levels of children. 4. With whom do you collaborate to facilitate reading activities? Interviewer: Please probe for whether youth volunteers work with teachers, parents, or other members of the community or not. 5. How do students participate in school and out of school reading activities? 6. Are there children who are successful in improving their reading skill as a result of various reading interventions? Interviewer: Please ask the interviewee to mention a best example of a successful child 7. What changes have you observed on parents with regard to supporting their children’s reading activities? Interviewer: Please probe for what brings those changes. 8. How can changes in children’s reading, community participation, etc., be strengthened and sustained? 9. What are the challenges encountered while delivering activities that support reading? What measures have you taken to deal with the challenges encountered if any? An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 82 10. Would you please rate your degree of satisfaction (1-5) on: 1 2 3 4 5 10.1 Children’s involvement in reading activities 10.2 Improvement in reading habit 10.3 Parental involvement in children’s learning to read Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 83 Interview Protocol for Principals and Supervisors Interviewer’s name ___________________________________________________________ Interviewee’s Code _______________________________ Cell phone of the interviewer ________________________ Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s interview is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this interview is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the interview, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the interview? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 84 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Name of the school _______________________________________________________ 6. Name of the interviewee ____________________________________________________ 7. Position: a. Principal b. Supervisor Guiding Questions 1. Please describe the kind of literacy activities implemented by READ CO? 2. What roles do you play in these activities? 3. Who leads and coordinates these activities? 4. What types of SRMs are available in the school and community to facilitate the literacy activities? 5. If available, where do you get these SRMs? Are the SRMs sufficiently available? 6. Do children enjoy using these SRMs? 7. Are the SRMs age appropriate and aligned to the needs of children? 7.1. How do students participate in the school and community-based literacy activities? 7.2. Which of these (reading corners, reading clubs, book bank, and reading camp) is more attractive and favorable for children? 7.3. Are the literacy activities effective in helping children learn to read? Please explain if you have any example which support this? 8. How do you describe your working relationship with youth volunteers and MT teachers? 9. What kind of support are you giving for mother-tongue teachers and for youth volunteer? Interviewer: Please probe for examples of successful collaboration between MT teachers/principals/supervisors, and youth volunteers. 10. How do parents or other than members of PTSA collaborate with the school to support children’s reading activities? 11. Have you noticed any improvement in the way parents approach their children’s learning to read? How do you know these changes? 12. What changes have these reading activities brought on children’s learning to read? How do you evaluate the changes? Interviewer: Please probe whether these changes are related to children’s reading skill, independent reading time, interest to participate in the literacy activities, and children’s understanding of the importance of reading. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 85 13. How do you evaluate the overall effectiveness of READ CO’s school and community-based approaches in mobilizing the community to support children learning to read? Interviewer: Please allow respondents to justify their responses 14. If the school and community-based approaches are effective in enhancing children’s reading skill, how can these be strengthened and sustained? 14.1. How will the school sustain the literacy activities? 14.2. What additional support is needed to sustain the impacts of these reading activities? 14.3. What would you suggest to make both in and out of school reading activities more successful? Interviewer: Please probe for the strategies that can be utilized to reach more children and to avail more reading materials in the future. 15. Would you please describe challenges encountered while implementing the literacy activities and the measures you have taken? Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 86 Interview Protocol for PTSA/KETB Members Interviewer’s name ___________________________________________________________ Interviewee’s Code _______________________________ Cell phone of the interviewer ________________________ Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s interview is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this interview is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the interview, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the interview? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 87 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Kebele (if any) ___________________________________________________________ 5. Name of the school _______________________________________________________ Name of Participant Sex Age Head/Member of PTSA KETB Service Year Guiding Questions 1. Did you attend any training provided by READ CO? When and for how long? 2. What types of literacy activities have READ CO implemented in your school or community? 2.1. What is your major role in the implementation of these literacy activities? 2.2. Which components of these outreach activities are working effectively? Why? Interviewer: Please probe for the reasons for an activity or activities which is/are not working very well if any. 2.3. Are there community or school-based reading spaces accessible for children to practice reading? If yes, explain how it is implemented? 3. Would you please describe the working relationship between Youth Volunteers, PTSA, and KETB? 4. Is the working relationship effective in enhancing children’s reading? How? 5. How has PTSA/KETB engaged parents in the literacy activities implemented in the school or community to support children’s reading? 6. Are the SRMs sufficiently available? How are they accessible for children? If not, why? 6.1. How would you describe the quality of the SRMs? 6.2. How do children use SRMs both in the school and community reading places? 6.3. Would you please rate (high, medium, and low) the level of children’s participation in both school and community-based literacy activities? Interviewer: Please rate for in school and community-based activity responses separately. 6.4. What is your role in raising children’s level of participation in literacy activities? 7. Would you please state positive changes observed on children’s reading as a result of READ CO interventions or approaches? Interviewer: Please probe for the kinds of change observed An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 88 like reading skill improved, students’ level of participation in literacy activities improved, increased community awareness about the importance of reading, increased the community mobilization effort towards improved student reading, etc. Also probe for the impact such approaches have at the school, family, and child levels and evidences of changes. 8. What will happen to these activities when the project ends? 8.1. Can you explain if there is readiness from the part of the school and community to sustain the literacy activities? How is it demonstrated? If no, what is the reason? 8.2. How are these school and community-based literacy activities integrated into other initiatives of the community or school? Please describe by citing examples. 8.3. What would you suggest to make both in school and out of school reading activities to be more successful? 8.4. What additional support is needed to ensure and sustain the impacts of these reading activities? Interviewer: Please probe for the strategies that can be utilized to reach more children and to avail more reading materials in the future. 9. What were the challenges in implementing school and community-based literacy activities? Do you think the challenges were successfully managed? How? Interviewer: Please ask the interviewee the measures taken by the school and the community to overcome the challenges. 10. Would you please score your overall satisfaction on READ CO literacy activities: 1 2 3 4 5 Children’s involvement in reading activities Children’s reading habit, reading skill Parental involvement in children’s learning to read Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 89 Interview Protocol for Experts from Woreda Education Office Interviewer’s name ___________________________________________________________ Interviewee’s Code _______________________________ Cell phone of the interviewer ________________________ Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting a final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Community Outreach (READ CO). The purpose of today’s interview is to learn about the various activities carried out by READ CO in collaboration with the Ministry of Education (MoE) and Regional State Education Bureaus (RSEBs). We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this interview is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the interview, please contact (Dr. Daniel Tefera, the focal person for this study) at +251911663402. You can also email him at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) through the Chief of Party for the Field Office at sareaya@air-read-me.org. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate in the interview? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 90 Demographic Data 1. Region _________________________________________________________________ 2. Zone ___________________________________________________________________ 3. Woreda _________________________________________________________________ 4. Your higher level of education ______________________________________________ 5. Age ____________________________________________________________________ 6. Name of the interviewee ___________________________________________________ 7. Sex a. Female b. Male 8. Your current position ______________________________________________________ 9. How long have you been in this position? ______________________________________ Guiding Questions 1. Did you attend any training provided by READ CO? 2. When and for how long? 3. In READ CO implementation, tell me about the activities which you know and you are involved in? 4. What is your role in this community outreach project? 5. What kinds of supports and resources are available for children to participate in reading activities? 6. Who is providing such supports if available? Interviewer: Please probe for the school and community-based activities separately. 6.1. Who is responsible for coordinating and carrying out these activities? 7. Which school and community-based literacy activities are more effective and which ones are less effective? Why? Interviewer: Please probe for the school and community-based activities individually. 8. Would you please describe about SRMs which are in use in your woreda? 9. Which party is responsible for developing and/or distributing these SRMs to schools in the woreda? 9.1. Are the SRMs sufficiently available? 9.2. How are they accessible for children? If not accessible, why? 9.3. How effectively do children use the SRMs? 10. How do you rate children’s level of participation (as high, medium, and low) in both school and community-based literacy activities? Interviewer: Please probe for the role of the interviewee in raising children’s level of participation in literacy activities separately for the school and community-based activities. An Evaluation of Reading for Ethiopia’s Achievement Developed Community Outreach Project 91 11. What changes have you observed as a result of school and out of school reading activities? Interviewer: Please probe for the value the community outreach project added to efforts directed toward improving the reading and writing skills of children, the kinds of change observed like reading skill improved, children’s interest for reading raised, time for independent reading increased, level of participation in literacy activities improved, etc. Also probe for evidences of changes. 12. Were the approaches (engaging parents and the community, the use of SRMs, engaging youth volunteers) effective? Can you describe why or why not? 12.1. How has READ CO’s school and community-based approaches raised the collaboration and support between the schools and the community? Interviewer: Please probe for any change in this regard and change in community awareness about the importance of reading. 13. As an education expert, do you feel a sense of ownership of the activities? Why or why not? 14. What initiatives have been taken by the community and/or the woreda to sustain these school and community-based literacy activities? Interviewer: Please ask the interviewee what will happen to the activities if the project ends. Probe for school, community, and the woreda education office’s readiness to sustain the literacy activities. 14.1. How are these activities integrated into other activities of the woreda administration? Please describe by citing examples. 14.2. How can the school and community-based activities be strengthened? 14.3. What additional support is needed to ensure and sustain the impacts of these reading activities? Who should provide the support? Interviewer: Please probe for the strategies that can be utilized to reach more children and to avail more reading materials in the future. 15. Describe the challenges or barriers to implementing these school and community-based literacy activities, if any? Interviewer: Please ask the interviewee the measures taken by the WEO to overcome the barriers. 16. What are the differences between schools supported by READ CO and those who are not? Closing Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you very much for participating in this discussion today.