January 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Timothy P. Williams, Nidhi Kapur, and Alphonse Nshimyiryo on behalf of Proteknôn Consulting Group. and fill this text box. Mureke Dusome Performance Evaluation: Documenting successful approaches and lessons learned in promoting early grade reading through sustainable school￾community partnerships in Rwanda 1 MUREKE DUSOME PERFORMANCE EVALUATION DOCUMENTING SUCCESSFUL APPROACHES AND LESSONS LEARNED IN PROMOTING EARLY-GRADE READING THROUGH SUSTAINABLE SCHOOL-COMMUNITY PARTNERSHIPS IN RWANDA January 2020 USAID Cooperative Agreement Number AID-696-A-16-00002-School-Community Partnerships for Education Activity Evaluation Mechanism Number: 720-696-19-P-00007 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 2 CONTENTS ABSTRACT................................................................................................................................................... 4 ACRONYMS & ABBREVIATIONS ................................................................................................. 5 EXECUTIVE SUMMARY....................................................................................................................... 6 PROJECT BACKGROUND ................................................................................................................13 EVALUATION PURPOSE & QUESTIONS ..............................................................................16 EVALUATION METHODS................................................................................................................17 FINDINGS....................................................................................................................................................25 EVALUATION QUESTION #1..........................................................................................25 EVALUATION QUESTION #2..........................................................................................47 EVALUATION QUESTION #3..........................................................................................67 CONCLUSIONS........................................................................................................................................78 RECOMMENDATIONS .......................................................................................................................79 ANNEX 1 - EVALUATION STATEMENT OF WORK......................................................83 ANNEX 2 – APPROVED EVALUATION DESIGN...............................................................92 ANNEX 3 – DATA COLLECTION TOOLS.............................................................................114 3 ANNEX 4 – SOURCES OF INFORMATION..........................................................................121 ANNEX 5 – DISSEMINATION PLAN.......................................................................................122 ANNEX 6 – CONFLICT OF INTEREST DISCLOSURE...................................................123 ANNEX 7 – QUANTITATIVE ANALYSIS SUPPLEMENTARY TABLES...........131 4 ABSTRACT Mureke Dusome was a four-year project that aimed to improve early-grade reading skills in Rwanda through school-community partnerships. It recently received a cost extension to further embed interventions within local systems. This performance evaluation provides an opportunity to learn from the project's first four years prior to this extension. The evaluation was guided by three questions: (1) How has Mureke Dusome improved literacy-related knowledge, attitudes, and practices?; (2) Which knowledge, attitudes, and practices are correlated with higher reading skills?; and (3) What elements of Mureke Dusome program are likely to be sustained? Data sources include 91 interviews and focus groups at the national and sub-national levels as well as secondary data analysis of the Early Grade Reading Assessment (EGRA), which had a nationally representative sample of 4,635 lower primary school students. The study finds that the communities who successfully improved literacy-related knowledge, attitudes, and practices did so by monitoring reading clubs, linking program activities to performance contracts indicators (e.g., dropout), and utilizing community structures to promote reading. Analysis of EGRA data indicates positive associations between reading fluency and participating in reading activities after school, reading at home, having a place to borrow books, having a favorite book, and being read to. Mureke Dusome also transformed Rwanda's book industry by building capacity and creating supply and demand. Findings suggest that there is sufficient demand for Mureke Dusome interventions by key stakeholders, and that Rwanda’s new National Literacy Policy is key to sustaining and institutionalizing Mureke Dusome activities within communities and Rwanda's education system. 5 ACRONYMS & ABBREVIATIONS CFWPM CSPM CWPM DDE DEC DEO ECD ECE EGRA EQ FBO FGD GDP INGO JADF KAP KI KPL L3 Literacy Champion MIGEPROF MINALOC MINECOFIN MINEDUC MINISPOC NGO NISR ORF PTA RALSA RCBO REB RENCP RNEC SBCC SCOPE SD SEI SGAC SLU SOW UNICEF USAID Correct Familiar Words Per Minute Correct Syllables Per Minute Correct Words Per Minute District Director of Education Development Experience Clearinghouse District Education Officer Early Childhood Development Early Childhood Education Early Grade Reading Assessment Evaluation Question Faith-Based Organization Focus Group Discussion Gross Domestic Product International Non-governmental Organization Joint Action Development Forum Knowledge, Attitudes and Practices Key Informant Kigali Public Library Literacy, Language, and Learning Literacy Champion Ministry of Gender and Family Promotion Ministry of Local Government Ministry of Finance and Economic Planning Ministry of Education Ministry of Sports and Culture Non-governmental Organization National Institute of Statistics Rwanda Oral Reading Fluency Parent Teacher Association Rwanda Archive and Library Services Authority Rwanda Children’s Book Organization Rwanda Education Board Rwanda Education NGO Coordination Platform Rwanda National Ethics Committee Social and Behavioral Change Communication School-Community Partnerships for Education Standard Deviation School Education Inspector School General Assembly Committee School Leadership Unit Statement of Work United Nations Children’s Fund United States Agency for International Development 6 EXECUTIVE SUMMARY Mureke Dusome (‘Let’s Read’ in English) was designed as a four-year program funded by USAID. It was implemented by Save the Children Rwanda and its partner organizations, Umuhuza and Urunana DC. The program aims to improve early-grade reading skills for children in lower primary school (P1 - P3) through the promotion of school-community partnerships. To further embed the sustainability of Mureke Dusome interventions within local systems, USAID extended the performance period until July 2021. The purpose of this performance evaluation of Mureke Dusome is two-fold: 1. To document successful approaches and lessons learned to inform the design of similar community reading and family engagement activities; and 2. To develop recommendations to promote the sustainability of Mureke Dusome activities. Findings and recommendations from this evaluation will feed into the sustainability and implementation plan for Mureke Dusome’s extension period. USAID, the Government of Rwanda, and Save the Children and its partners may also use this study to inform support of community-based literacy initiatives in Rwanda and elsewhere. PROJECT BACKGROUND Through national and community-focused activities, Mureke Dusome seeks to strengthen family and community support for reading, and increase reading practice for children outside of school, through three objectives: 1. Strengthening the capacity of school leadership to improve student literacy through school-community partnerships; 2. Increasing effective community and parental involvement to improve literacy skills; 3. Fostering a culture of reading. Save the Children, in collaboration with the Government of Rwanda, developed modules and trained Head Teachers and School General Assembly Committees (SGACs) to increase community support for reading and school. They implemented social and behavioral change communication (SBCC) activities to increase literacy-supportive practices within families. They also established reading clubs in communities across the country. Clubs received books and were run by Literacy Champions, who were identified by school and village leaders and trained by a local partner, Umuhuza. Mureke Dusome also worked with the Rwandan publishing industry to build capacity and stimulate supply and demand for high quality children’s books in Kinyarwanda. During the extension period, Mureke Dusome has a specific mandate to deepen its focus on the sustainability of its activities through system-level change and community engagement. It was within the context of this transition that Proteknôn Consulting Group was commissioned by USAID to undertake this performance evaluation. METHODOLOGY Data sources for the evaluation include: • Document review of program descriptions, implementation plans, existing evaluations and assessments, sustainability plans and the National Literacy Policy. • National-level key informant interviews (n=22) with individuals from the Government of 7 Rwanda, INGOs, civil society, the private sector book industry, and other key stakeholders. • Sub-national level key informant interviews (n=49) with education officials and representatives from other groups entities, such as faith-based organizations and village leaders. • Focus group discussions (FGDs) with SGAC members (n=10) with a total of 99 participants. • Focus group discussions with children (n=10) FGDs with a total of 96 participants, including students in P1 to P3 who regularly attended Mureke Dusome reading clubs for at least one year. • Secondary quantitative data analysis (n=4,635) of the 2018 Early Grade Reading Assessment (EGRA). Qualitative data collection was carried out in five districts, ten sectors and ten schools, including one district per province, two sectors per district, and one school per sector. District site selection was done purposefully in consultation with Save the Children Rwanda and USAID. The five chosen districts are ‘model districts’ that will play a central role during the extension phase, and include Gasabo, Kirehe, Ruhango, Burera, and Ngororero. MAIN FINDINGS EQ1. How has Mureke Dusome improved literacy-related knowledge, attitudes, and practices? Mureke Dusome’s efforts to improve literacy-related knowledge, attitudes, and practices took a holistic approach. The program attempted to shift perspectives and behaviors around literacy by altering practices and policies. And conversely, it attempted to shift practices and policies by altering perspectives and behaviors. Collectively, Mureke Dusome interventions sought to achieve this goal by aligning systems, activities, and stakeholders at multiple levels with the ultimate objective of promoting children’s literacy. Almost all (99%) children reported that they enjoy reading in a group with other children. The majority of children reported having opportunities to practice reading at home, through participation in community-based reading activities, reading on their own, having someone reading stories to them, and/or reading to someone at home. Mureke Dusome has improved literacy-related knowledge, attitudes and practices in the following ways: • Social and behavioral change activities combined with policy advocacy, improved access to books, and local capacity building effectively improved literacy-related knowledge, attitudes and practices. • Successful communities transformed their community culture through: the active monitoring of reading clubs by school leaders; linking program activities to imihigo1 contracts (e.g. contracts to reduce school dropout); follow-up by teachers and parents at reading clubs; and utilizing local structures, like umuganda2 and Faith-Based Organizations (FBOs) to promote reading. Communities that were less successful lacked these characteristics, had high turnover of Literacy Champions, held reading clubs infrequently, and received limited engagement or oversight from local officials. • Efforts to build capacity and strengthen the Rwandan publishing industry included: enhancing the capacity of authors, illustrators and publishers; encouraging new authors through Abana Writers Café; stimulating supply and demand of books; and establishing the Rwanda Children’s Book Organization. 1 Imihigo are performance contracts between civil servants and the Government of Rwanda. 2 Umuganda is a monthly community volunteering program. 8 • Mureke Dusome contributed to capacity building and the system strengthening of schools and local government by providing training manuals to facilitate self-study and offering settings of communities of practice for SGACs and head teachers. Several factors limited Mureke Dusome's success. These include: the possible de facto exclusion of some children with disabilities due to a lack of training for Literacy Champions; a mismatch between the expectations of stakeholders and the Mureke Dusome workplan; miscommunication regarding the extension of the program; and an insufficient number of new books to continue to stimulate children. EQ2: Which knowledge, attitudes and practices (KAP) are correlated with higher or improved student reading skills? Almost all children reported that they enjoy reading in a group with other children (Figure 1). Figure 1: Percentage of students with reading knowledge, attitudes and practices (KAP) characteristics The majority of children reported having opportunities to practice reading at home, through participation in community-based reading activities, reading on their own, having someone reading stories to them, and/or reading to someone at home. Reading KAP factors that were significantly correlated with reading fluency For P1 students, three KAP factors were associated with higher/improved scores on reading comprehension and oral reading fluency (ORF). These included (Figure 2): • Participating in reading activities after school, • Taking Kinyarwanda books home from the classroom for reading practice after school, and • Reading independently at home 97.9% 78.1% 69.5% 77.1% 56.2% 62.0% 77.2% 53.1% 37.9% 98.7% 84.5% 82.6% 84.3% 75.9% 71.3% 74.1% 65.2% 49.7% 99.4% 88.5% 86.9% 85.3% 76.3% 75.4% 74.7% 56.7% 49.5% Enjoy reading in a group with other children Participating in any reading activity after school Having a favorite book Reading independently at home Children who ever lend or borrow a book or other learning materials to/from other students Reading to someone out loud at home Having someone who reads a story to the child at home Taking Kinyarwanda books home from the classroom and use them to practice reading Having a place in the community where the child can go to read/borrow Kinyarwanda book P1 P2 P3 9 Figure 2: Percent distribution of P1 students meeting the grade-level benchmark by knowledge, attitude and practices (KAP) characteristic For P2 students, KAP factors associated with higher/improved scores on reading comprehension and ORF, were (Figure 3): • Participating in reading activities after school, • Having a favorite book • Reading independently at home • Reading to someone out loud at home • Having a place in the community where children can go to read or borrow books Two KAP factors were associated with higher/improved scores on reading comprehension and ORF for P3 students (Figure 4): • Reading independently at home • Having a place in the community where children can go to read or borrow books Reading KAP factors that were significantly correlated with higher/improved lower order reading skills Consistently, the three top-ranked predictors of higher/improved scores on low order reading skills (letter name identification, syllable sound identification and familiar word reading) for all grades, were: • reading to someone out loud at home, • reading independently at home, and • participating in reading activities after school. 3% 8% 5% 5% 22% 26% 25% 25% Participating in reading activities after school Taking books home for reading practice Participating in reading activities after school Reading independently at home Reading Comprehension Oral Reading Fluency (ORF) No Yes 10 Figure 3: Percent distribution of P2 students meeting the grade-level benchmark by knowledge, attitude and practices (KAP) characteristic Figure 4: Percent distribution of P3 students meeting the grade-level benchmark by knowledge, attitude and practices (KAP) characteristic EQ3. What elements of Mureke Dusome program are likely to be sustained? How could the sustainability of the Mureke Dusome program be further promoted? There is demonstrable ownership and demand for Mureke Dusome interventions. Energy should be 11% 16% 17% 26% 2% 5% 6% 41% 40% 44% 46% 18% 18% 19% Participating in reading activities after school Having a favorite book Reading to someone out loud at home Having a place in the community where children can go to read/borrow books Reading independently at home Having a favorite book Reading to someone out loud at home Reading Comprehension Oral Reading Fluency (ORF) No Yes 9% 23% 2% 33% 37% 6% Reading independently at home Having a place in the community where children can go to read/borrow books Having a place in the community where children can go to read/borrow books Reading Comprehension Oral Reading Fluency No Yes 11 invested in promoting literacy activities as a community-driven effort rather than one that is directed by an INGO. Ownership could be improved by including literacy-focused indicators in imihigo contracts. Sustainability could be also enhanced by articulating the link between early literacy activities and other government priorities. This could include sector-focused activities, like reducing drop out, as well as other national aims, such as improving human capital. The National Literacy Policy is the key to sustaining key elements and activities of Mureke Dusome. Sustainability is further promoted by working within existing systems, using REB standards, and working through schools for community outreach. The evaluation identifies several challenges to sustainability: • It is a challenge to retain Literacy Champions because they are serving as volunteers. • Stakeholders' skill levels and capacity to sustain Mureke Dusome interventions is constrained by limited skills and finite resources. • The recurrent financial costs of Mureke Dusome interventions include a continued challenge to improve access to storybooks and other reading material for children, particularly outside of Kigali. CONCLUSIONS Stakeholders at multiple levels have begun to understand, appreciate and value reading. The National Literacy Policy offers a promising policy architecture to increase lifelong literacy skills starting from an early age, which will also contribute to a wider culture of reading. The Rwandan book sector has stronger capacity to produce higher quality, age-appropriate Kinyarwanda titles for P1 to P3 children. Without clear incentives and accountability measures, however, local government engagement on early literacy issues will likely lack consistency or be less focused than hoped. Gaps also remain in terms of the accessibility of books outside of urban centers in Rwanda. Left unchecked, this could reinforce existing inequalities. Any reduction in Mureke Dusome’s continue support to Rwanda’s nascent book industry is a point for careful consideration as it will require the industry to find new ways to sustain and to grow. RECOMMENDATIONS 1. Foster greater cross-sectoral communication and coordination • Engage in advocacy with MINEDUC to increase recognition of the importance of community engagement to support children’s learning. • Consider further research to better understand the interface between MINALOC, MINEDUC and other line ministries related to literacy. • Advocate with MINEDUC, MINISPOC and MINALOC to channel more resources into expanding community libraries. 2. Explore different options to further strengthen the work of Literacy Champions • Continue working with MINEDUC to implement the National Literacy Policy. • Examine whether and how the Literacy Champion model may emulate existing successful volunteer models in Rwanda, such as the Community Health Workers program. Identify a package of material and non-material incentives that can bolster sustainability. • Determine how best to integrate literacy activities into existing community structures at the 12 village level, for example through umuganda, inteko y’abaturage and itorero, while establishing clear channels for accountability and reporting. 3. Harmonize services to avoid duplication of effort • Given the multiple demands placed on local officials, development and implementing partners must work with government to develop a strategy to present one basic education plan for each district to clarify key objectives for the year and assign responsibilities with corresponding budgets. • Consider possible entry points to better harmonize service delivery to fully maximize the positive impact of programming. One example is reading clubs, which could service P1 to P3 students and also act as a gateway to other target groups and issue areas. • Consider whether existing positions at the school or community level could take on additional responsibilities in terms of literacy promotion, thereby investing in state-mandated structures such as school-based mentors or classroom teachers, for example, rather than creating parallel systems. • As community libraries grow in number, their link to reading clubs should be strengthened. Functioning community libraries should serve as future sites for reading clubs. • District level coordination of literacy activities should be improved to increase the efficiency, impact and performance of Mureke Dusome. • Consider the potential advantages and feasibility of linking with adult literacy programs. • Examine ways to scale up activities through partnerships with faith-based organizations and churches, which have longstanding relationships with community members. 4. Improve access to books and reading opportunities for all children • Interventions to improve lower primary students’ reading skills should continue to focus on the following reading practices: participation in reading activities after school; having a place in the community where children can go to read or borrow books; taking Kinyarwanda books home from the classroom to use for reading practice; reading to someone out loud at home; and reading independently at home. • Development partners should continue to advocate to donors, private sector actors, and MINEDUC/REB for increased accessibility to reading materials in homes and communities. • Development partners should work together with private sector actors to create a strategy to ensure books and other relevant reading material reach local communities. Specific ideas to improve book distribution could include establishing community savings groups for books. • Development partners should enlist more involvement from the Private Sector Federation and MINECOFIN to improve access to books for poor households through, for example, providing grants and subsidies to the book sector, reducing taxes on books, and engaging sectors in corporate social responsibility aimed at literacy. • Consider recruiting different actors in government and the private sector to sponsor the development and distribution of books that include key messages and content which aligns with their sector. Actors could represent areas such as finance, culture, taxes, infrastructure, investment, gender, health, or youth. • Development partners and donors should consider digital library solutions by Kigali Public Library in collaboration with REB. There may be unexplored synergies between the digitization of books and future literacy-related programming. 13 PROJECT BACKGROUND Mureke Dusome (‘Let’s Read’ in English) was designed as a four-year program funded by USAID and implemented by Save the Children Rwanda and its partner organizations, Umuhuza and Urunana DC. The program was introduced with the goal of improving early-grade reading skills for children in lower primary school (primary grades 1 to 3, P1-P3) through the promotion of school-community partnerships. It was founded on the evidence that children learn to read better if they have family support and opportunities for reading practice outside of school, compared to if they receive only classroom-based instruction. Implementation of Mureke Dusome was predicated upon the following three core objectives: 1. Strengthening the capacity of school leadership to improve student literacy through school￾community partnerships; 2. Increasing effective community and parental involvement to improve literacy skills; and 3. Fostering a culture of reading3 While Mureke Dusome’s remit was nationwide, its community-based interventions were rolled out incrementally across all 30 districts of Rwanda in three distinct phases: ● Phase 1: [January - September 2016] in Burera and Gicumbi districts;4 ● Phase 2: [October 2016 - September 2017] expansion to 10 more districts; ● Phase 3: [October 2017 – September 2019] expansion to the remaining 18 districts. Key components of Mureke Dusome’s work at the community level included establishing reading clubs. Clubs were supplied with books, run by a cadre of Literacy Champions identified by school and village leaders, and trained and monitored by Umuhuza. At the national level, Save the Children in collaboration with the Government of Rwanda developed modules and trained Head Teachers and School General Assembly Committees (SGACs) to increase community support for reading and school; implemented a social and behavioral change communication (SBCC) strategy to increase literacy-supportive practices within families; and built the capacity of key actors in the Rwandan publishing industry to stimulate supply and demand for high quality children’s books in Kinyarwanda. Through these interventions – both top￾down and bottom-up - Mureke Dusome sought to strengthen family and community support for reading and increase reading practice for children outside of school.5 Over the course of implementation, Mureke Dusome identified several challenges related to achieving national scale with limited funding. For example, at inception, it was anticipated that because Mureke Dusome would engage with every public and government-aided primary school in the country, all P1-P3 students in the public schooling system would have an increased opportunity to participate in community reading activities. However in practice, children who lived in villages that did not have a school had less access to community reading activities because of the distance from their homes. 6 Also at inception, it was also anticipated that an initial supply of 100 books per reading club would be sufficient, whereas the 3 Jonason, C. et al. (2018). Mureke Dusome impact evaluation endline report: program impact on literacy knowledge, attitudes, and practices. Save the Children. USAID. Rwanda Education Board. 4 When it was launched, Mureke Dusome was first known as School-Community Partnerships for Education (SCOPE). The two districts were selected for the first year in order to build on the existing operational and program activities and capacity that had been set up by Save the Children and Umuhuza through other projects. This meant that Mureke Dusome’s activities could begin more quickly by building on existing experience, partnership, and local relationships with schools, communities, local government, and other actors. 5 Jonason, C. et al. (2018). Mureke Dusome impact evaluation endline report: program impact on literacy knowledge, attitudes, and practices. Save the Children. USAID. Rwanda Education Board. 6 Jonason, C. et al. (2018). Mureke Dusome impact evaluation endline report: program impact on literacy knowledge, attitudes, and practices. Save the Children. USAID. Rwanda Education Board. 14 project found that children quickly read through all books and demanded more. Finally, at inception the team envisioned a light-touch to monitoring as interventions were to be community-led, but on the ground were met with local expectations for more intensive monitoring and also identified the benefit of monitoring visits to motivate local stakeholder’s activities. Mureke Dusome was designed to work within local systems, with leadership and ownership at the community level. In order to further embed the sustainability of the initial interventions of Mureke Dusome and to strengthen how interventions address equity and inclusion, USAID extended the period of performance by 18 months to July 2021. The four objectives for the extension period are: • To sustain capacity strengthening for school leadership to promote school-community partnerships and improve student literacy; • To sustain increased effective community and parental involvement to improve literacy skills; • To sustain improvements to the culture of reading; • To promote equity in literacy work, with a focus on gender empowerment and the inclusion of children with disabilities. Mureke Dusome’s approach to strengthening prospects for sustainability is generally to focus on institutionalizing activities within existing structures. For example, school leadership discussions to improve school-community partnerships will be embedded in quarterly Education Council meetings, parent education and literacy messaging will be embedded in faith-based organizations and the national urugerero program, and access to reading materials will be addressed through strengthening MINEDUC’s community library system. In addition, Mureke Dusome is supporting the Ministry of Education to validate and implement its National Literacy Policy, which would become another mechanism to institutionalize community reading interventions. The Sustainability Plan developed by Mureke Dusome in October 2018 identified that sustainability does not necessarily mean that the activities of the program continue in the same form as originally conceived, funded or implemented. Programs often evolve over time to adjust to the changing levels of support and needs of the community. Sustainability of program goals (i.e., strengthened school-community partnership leading to improved reading skills among children) will emerge from a culture that values reading and is committed to serve children more effectively. Proteknôn Consulting Group was commissioned to undertake this performance evaluation during the transition between Mureke Dusome’s initial 4-year intervention and the additional 18-month extension. CONTEXT ANALYSIS: EARLY GRADE LITERACY IN RWANDA Over the last two decades, Rwanda’s education sector has experienced notable success. Fee-free basic education has significantly improved access to schooling for children at both the primary and secondary levels, while gender parity has been achieved - making Rwanda one of the few countries to meet its education-related commitments to the UN Millennium Development Goals.7 While accessibility has substantively improved, learning outcomes for children remain low. Most children in primary school do not acquire age-appropriate literacy or numeracy skills, and many leave primary school unable to read in any language. A recent study of reading comprehension using REB benchmarks for students in Primary One found that just 1 in 3 students met the reading fluency benchmark and 1 in 10 students achieved the reading comprehension benchmark.8 7 Abbott, P. et al. (2015). Rwanda's potential to achieve the Millennium Development Goals for education. International Journal of Educational Development 40: 117-25. 8 Sinclair, B. et al. (2018). Rwanda Early Grade Reading Assessment baseline report – draft for review. Soma Umenye Project. Chemonics. USAID: Kigali. 15 Low learning in early primary school is concerning for many reasons. It contributes to student dropout, repetition, and low transition rates to secondary school. Low literacy rates can also limit the potential of children to more fully contribute to the social and economic development of Rwanda.9 Improving early grade literacy is an important component of improving the quality of education more broadly. To date, education interventions often center on addressing technical aspects of children’s classroom-based educational experience, such as improving pedagogy through teacher training, providing resources and materials to enhance the curriculum, and improving infrastructure. USAID has directly contributed to these classroom-focused interventions through its Literacy, Language, and Learning (L3) and Soma Umenye activities, among others. Mureke Dusome’s community-based activities complement this classroom-based work. Its activities were borne from a recognition that low literacy rates also correspond to factors beyond the classroom that shape learning and early grade literacy, including the interaction between schools and communities to improve learning. During the planning phase of the program, for example, Save the Children staff found that community participation in children’s schooling was low.10 Aside from paying school-related expenses, parents did not see it as their job to be involved in their child’s education. This belief led to high turnover and low engagement on General Assembly Committees (SGAC) (formally known as Parent Teacher Associations, or PTAs) because there was no shared understanding that teachers and parents should hold one another accountable to improve the quality of their children’s education. It was also found that children had little access to high-quality, age-appropriate Kinyarwanda language reading materials, and children had limited opportunities to practice reading outside of school. Thus, if communities wanted to improve children’s literacy, there was limited opportunity to do so. 9 MINEDUC. (2013). Education Sector Strategic Plan 2018 - 2024. Ministry of Education, Government of Rwanda. World Bank. (2018). Future drivers of growth in Rwanda: innovation, integration, agglomeration, and competition. Washington, D.C.: World Bank Group. 10 Nzabonimpa, J.P. et al. (2016). Rwanda: baseline survey tracking literacy knowledge, attitudes, and practices at the school and community level. Save the Children. USAID. 16 EVALUATION PURPOSE & QUESTIONS EVALUATION PURPOSE The overarching purpose of this performance evaluation of Mureke Dusome is two-fold: 1. To document successful approaches and lessons learned to inform the design of similar community reading and family engagement activities in Rwanda and elsewhere; and, 2. To develop recommendations to promote the sustainability of Mureke Dusome activities. The findings and recommendations of this report will help finalize the sustainability plan for Mureke Dusome’s extension period. USAID, the Government of Rwanda, Save the Children and its partners may also use this study to inform current and future community-based literacy initiatives in the country. Finally, key findings may be used to inform similar interventions in other contexts. KEY EVALUATION QUESTIONS This study is based upon three key evaluation questions (EQs) and guided by several sub-questions. The key evaluation questions served to inform the conceptual framing and organization of this study, while responses to the sub-questions were embedded through the themes that emerged in the qualitative components of the fieldwork. EQ1. How has Mureke Dusome improved literacy-related knowledge, attitudes, and practices? • How have the most successful communities targeted by Mureke Dusome transformed their community culture to be more supportive of children's literacy? • What were the biggest perceived contributors to changes in literacy behavior among head teachers, parents and children? • What has been Mureke Dusome’s contribution to the capacity building and system strengthening of the Rwandan publishing industry? • What has been Mureke Dusome’s contribution to the capacity building and system strengthening of schools and local government? • Which aspects of Mureke Dusome were not successful and why? EQ2: Which knowledge, attitudes and practices are correlated with higher [or improved] student reading skills? EQ3. What elements of Mureke Dusome program are likely to be sustained? How could the sustainability of the Mureke Dusome program be further promoted? • Is there demonstrable ownership and demand for Mureke Dusome interventions? How could Mureke Dusome further promote ownership and demand? • What is the level of skills and capacity among stakeholders to sustain Mureke Dusome interventions? How could Mureke Dusome further build up skills and capacity among stakeholders? • What are the recurrent financial costs of Mureke Dusome interventions, and will future revenue streams be sufficient to sustain them? What could Mureke Dusome do to ensure sustained financing for their interventions? 17 EVALUATION METHODS To answer the three key evaluation questions and related sub-questions, a mixed-methods approach was utilized. This included a review of available literature, collection and analysis of qualitative data gathered through key informant interviews and focus group discussions conducted at national and sub-national levels, as well as analysis of existing quantitative data sets. SOURCES OF DATA ● Document review: Proteknôn undertook an initial desk-based review of existing literature on Mureke Dusome provided by USAID and Save the Children. This included project documents, such as program descriptions, implementation plans, monitoring, evaluation and learning plans, sustainability plans, as well as, existing reports and assessments. This review helped to inform study design, the development of qualitative question guides, the analysis and interpretation of qualitative and quantitative data. ● Key Informant Interviews: Proteknôn worked with Save the Children and USAID to jointly identify national, district and local level key informants representing a range of different program stakeholders, including from government, non-government, local civil society and the private sector. Question guides were developed to conduct semi-structured interviews of around 30 to 60 minutes in length that centered on the perceptions and insights based on each informant’s particular area of expertise. ● Focus Group Discussions: Proteknôn developed tools to conduct disaggregated focus group discussions with parents and children who are familiar with the Mureke Dusome intervention through their respective participation in School General Assembly Committees (SGACs) and reading clubs. Discussions typically lasted between 30 and 50 minutes. The tools were designed to draw on the experiences of participants, producing data relevant to understanding the effectiveness and long-term sustainability of Mureke Dusome. ● Secondary analysis of quantitative data: In order to answer Evaluation Question #2, Proteknôn analyzed data from the Early Grade Reading Assessment (EGRA) data collected by Soma Umenye between September and October 2018 on a nationally-representative random sample of 4,635 primary school students in grades 1, 2 and 3. EGRA is a tool which is used to measure reading skills among early primary grade students. The 2018 EGRA sample drew from all public and government-aided schools across Rwanda, and the sample of students in each grade who participated in EGRA was selected in two stages. First, schools were randomly selected in rural/urban strata, and secondly students were also randomly selected from the selected schools. In total, 1544 students were selected in Primary 1, 1540 in Primary 2 and 1551 in Primary 3. The 2018 EGRA assessed students’ reading skills using six EGRA sub-tasks: listening comprehension, letter sound identification, syllable sound identification, familiar word reading, oral reading fluency and reading comprehension. The reading passage used for the oral reading fluency and reading comprehension sub-tasks differs for each grade. A detailed description of each EGRA sub-task and its rationale for assessing reading skills can be reviewed elsewhere.11 In addition, the 2018 EGRA included a context questionnaire that collected data on children’s reading knowledge, attitudes and practices (KAP), access to reading materials, school attendance (i.e. absence/delay to school), home environment and family socioeconomic status (i.e. having electricity, a phone, radio, bicycle, moto, car, access to food and drinks before going to school. 11 Dubeck, M. M. et al. (2015). The early grade reading assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development, 40, 315-322. 18 SITE SELECTION Sub-national qualitative data collection was carried out in five districts, ten sectors and ten schools, including one district per province, two sectors per district and one school per sector. The districts to be included within the evaluation were purposively selected. They comprise the five ‘model districts’ in the country that will become the focal districts for further collaboration during the upcoming cost extension phase of the program. These districts included: Gasabo, Kirehe, Ruhango, Burera, and Ngororero. Amongst the two sectors selected per district, there was an intentional decision to include one sector which was deemed as particularly successful and a second sector that was characterized by more modest achievements and/or specific operational or programmatic challenges from which lessons learned could be drawn to inform future programming. While there was a degree of subjective discretion involved in sector selection, Proteknôn and Save the Children shared the view that a “successful sector” was in which examples of strong leadership, functional reading clubs, and/or consistent parental involvement could be found. On the other hand, the “other sector” was selected with a view to exploring the range of challenges faced by Mureke Dusome during program implementation. For these reasons, site selection aimed to account for various factors such as the length of the Mureke Dusome intervention in the targeted area, key personnel changes, as well as urban, peri-urban and rural settings, among others. It is important to note that the goal of site selection was not to develop case studies on different sectors and schools, nor to achieve any degree of statistically valid representation. Rather, sampling was undertaken to sufficiently explore the variability – in terms of both successes and challenges – in how Mureke Dusome was experienced at local level. The breakdown of specific sites and criteria for their selection are outlined in greater detail below: Table 1 - District, sector and school site selection considerations and characteristics District “Successful” sector (School) “Other” sector (School) Gasabo District ● Has a successful school community supporting children’s literacy and has benefitted from more exposure to Mureke Dusome activities compared with other districts in Kigali Province Nduba Sector ● Functional reading club ● Literacy Champions are active ● School collaborates with Literacy Champions in promoting children’s literacy activities ● Community supports the reading club Ndera Sector ● Motivated Literacy Champions with moderate community support ● Low attendance of children in reading club ● Urban environment Kirehe District ● Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Eastern Province Nyarubuye Sector ● Reading club meets regularly ● Community engagement with good collaboration between the school and Literacy Champions Gahara Sector ● One Literacy Champion is committed, while another has dropped out 19 Ruhango District ● Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Southern Province Mbuye Sector ● SGAC is engaged, visits reading club, and collaborates with Literacy Champions ● Good community engagement with parents supporting Literacy Champions ● Literacy Champions are active ● School is located in rural area Byimana Sector ● Literacy Champions are committed and collaborate with the school ● Children are enjoying the reading club, but parents are not as engaged ● School is located in urban area Burera District ● One of two districts of Rwanda where Mureke Dusome first commenced interventions ● Shares border with Uganda (cross-border trade and linguistic specificities compared to other areas of Rwanda) ● Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Northern Province Rugarama Sector Strong collaboration between local leaders, parents and Literacy Champions in promoting children’s literacy activities ● Local leaders are motivated and work with parents to support Literacy Champions to develop materials ● Parents attend the reading club Kivuye Sector ● Community speaks Urukiga dialect, which presents difficulties to children when adapting to reading materials in Kinyarwanda ● Reading club does not take place regularly Ngororero District ● Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Western Province Nyange sector ● Literacy Champions are committed ● Children’s attendance is high ● Community members are supportive ● Rural environment Ngororero sector ● Literacy Champions are committed ● Children’s attendance is moderate ● Community engagement is limited ● Urban environment SAMPLE AND SAMPLING STRATEGY As with the site selection, our sampling strategy for the selection of participants was purposive. It was designed to strike a good balance between the breadth and depth that this evaluation requires. The study design permitted for a vertical investigation into the evaluation questions, because we were able to speak with stakeholders in various roles at different levels. We found the sample size to be sufficient to achieve theoretical saturation to address the questions and sub-questions of this study. And while our methods were systematic, they were not rigid. In the event that a prospective participant was unavailable or non￾responsive, our research team was able to flexibly replace individuals at the sub-national level with another alternative, knowing that the primary criteria for participant selection was stakeholders’ familiarity with Mureke Dusome. This was to ensure that the sample population could speak knowledgeably about their engagement with the program with respect to the priority evaluation questions. 20 In consultation with Save the Children, Proteknôn identified participants to obtain national and local-level insights from children, parents, head teachers, Literacy Champions, education officials, civil society and other development partners, and private sector representatives from the Rwandan book industry with strong knowledge of and/or experience with Mureke Dusome interventions. National level: A total of 22 key informants were identified for interviews - either individually or in small group sessions - from the following institutions: Government of Rwanda: ● Ministry of Education (MINEDUC): ● Minister of State for Primary and Secondary Education ● Public and Community Libraries Officer ● Rwanda Education Board (REB): ● Director of School Leadership & Management Unit ● School-Community Partnerships Coordinator (embedded Save the Children staff member funded by USAID Mureke Dusome) ● Advisor to the Director General of REB (funded by USAID Soma Umenye) ● Ministry of Local Government (MINALOC): ● Director of Mentoring & Volunteerism, National Itorero Commission ● Ministry of Sports and Culture (MINISPOC): ● Acting Director of the National Library Key INGO, civil society, private sector and other stakeholders: ● Save the Children/Rwanda: ● Chief of Party Mureke Dusome ● Deputy Chief of Party Mureke Dusome ● USAID/Rwanda: ● Senior Education Specialist ● Umuhuza: ● Executive Director ● Community Engagement Coordinator ● Urunana DC: ● Director ● Chemonics: ● Chief of Party Soma Umenye ● Deputy Chief of Party for Field Operations Soma Umenye ● Additional members of Soma Rwanda Steering Committee: ● Education Specialist, UNICEF 21 ● Director, Rwanda Bookmobile ● Education Programme Manager, World Vision ● Children’s book sector representatives: ● Chairperson, Rwanda Children’s Book Organization (RCBO) ● Director, Arise Education ● Director, Mudacumura Publishing House ● Director, Sankofa Creatives Sub-National level: At the local level, education officials, including District Directors of Education (DDE), District Education Officers (DEO) and Sector Education Inspectors (SEI), were targeted by the evaluation for key informant interviews along with Joint Action Development Forum (JADF) officers. In addition, Literacy Champion Representatives at sector level, and Head Teachers or Deputy Head Teachers at school level, were interviewed. In order to gain further insights from those most knowledgeable about Mureke Dusome at the local level, further interviews with other individuals were also conducted, including representatives of Faith-Based Organizations (FBOs), cell or village leaders, as relevant. Focus group discussions were held with School General Assembly Committee (SGAC) members, including the President, Vice-President, and other parents with children in reading clubs. Sometimes it was also possible to include school-level Literacy Champions within these focus group discussions. There were additional focus group discussions with Primary 1 to Primary 3 children who had been regularly attending Mureke Dusome reading clubs for at least one year. In total, we conducted a total of 69 key informant interviews and focus group discussions at the sub￾national level with a total of 240 participants. A breakdown of participants disaggregated by both district and participation type is summarized in Table 2 below: Table 2. Sub-national evaluation participants Selected Participant/s Gasabo District Kirehe District Ruhango District Burera District Ngororero District TOTAL Key Informant Interviews Joint Action Development Forum (JADF) officer 1 0 1 1 1 4 District Director of Education (DDE) 1 1 0 0 1 3 District Education Officer (DEO) 0 0 1 1 0 2 Sector Education Inspector (SEI) 2 2 2 2 2 10 Head Teacher/Deputy Head Teacher 2 2 2 2 2 10 Literacy Champion Representative 2 2 2 2 2 10 Others (cell/village leaders, FBO, etc.) 2 2 2 2 2 10 22 Focus Group Discussions Parents (including SGAC President, Vice-President) 20 19 20 19 21 99 Children (reading club members) 20 18 19 20 19 96 DATA ANALYSIS For the qualitative analysis (EQ#1 and EQ #3), interview and focus group transcripts were uploaded to a qualitative software program called NVivo. All transcripts were first grouped thematically by interview questions. Then themes were identified by key evaluation questions and sub-questions. Particular attention was paid to the significance of gender, disability, and the individual profile of each research participant (e.g. Literacy Champion, local official, etc.), as appropriate. In addition, on 14 November 2019, Proteknôn convened a meeting of key stakeholders to present emerging findings from the qualitative fieldwork. The meeting was attended by approximately 20 individuals from Save the Children, Umuhuza, and USAID. The purpose of the meeting was to facilitate the co-creation of recommendations for the evaluation. It also added a validation step to confirm the validity of some of our key findings. For the quantitative analysis (EQ#2), we categorized the outcome variables for reading comprehension and oral reading fluency (ORF) into binary variables based on Kinyarwanda reading fluency benchmarks validated by the Rwanda Education Board (REB) in June 2019: ● “Higher/improved reading comprehension” was defined as an EGRA score on reading comprehension ≥60% for primary 1&2 students and ≥80% for primary 3 students; ● “Higher/improved ORF”, was defined as an EGRA score on ORF ≥10 correct words per minute (cwpm) for primary 1 students, ≥25 cwpm for primary 2 students, and ≥40 cwpm for primary 3 students. For the analysis of KAP factors associated with higher performance on lower order reading skills, we used continuous data on: ● “Letter identification” – the number of correct letters per minute (clpm), only applicable to primary 1& 2 students; ● “Syllable sound identification” – the number of correct syllables per minute (cspm); ● “Familiar word reading” – the number of correct familiar words per minute (cfwpm). Then, we described KAP and outcome variables using frequencies and percentages for categorical data, as well as mean and standard deviations (SD) for continuous data. We used logistic regression analysis to investigate KAP factors associated with higher/improved EGRA scores for ORF and reading comprehension binary outcomes. For continuous outcome variables, multiple linear regression (MLR) was used to assess KAP factors correlated with higher EGRA scores on letter identification, syllable sound identification, familiar word reading and ORF sub-tasks. Both logistic and multiple linear regression models were adjusted for gender and age of students, and other possible confounders around home environment, socioeconomic status and school location (rural/urban as a proxy for child’s residence) and attendance. We excluded the listening comprehension subtask from the MLR analysis, because it might be inappropriate for a bound and discrete outcome variable with only 6 values (i.e. 0-5 possible listening questions correct). We tested our hypothesis using the α=0.05 significance level. All analyses were adjusted for sampling weights to account for unequal probability sampling of schools and students in different strata and data were analyzed separately for each grade. We also conducted a dominance analysis to identify KAP factors that were most important in 23 predicting higher/improved scores on letter name, syllable sound and familiar word reading skills. We used Stata v.15.1 to analyze the data. ETHICAL CONSIDERATIONS Proteknôn’s work is strongly rooted in the overriding principles of ‘do no harm’ and the ‘best interests of the child.’ The research team responsible for data collection received two-days of orientation prior to the start of fieldwork, including child safeguarding training provided by Save the Children. Additional training in ethical research protocols (confidentiality, data handling, use of informed consent and assent) as well as age-appropriate and inclusive fieldwork methods was jointly provided by the Proteknôn Team Lead and Field Coordinator. Specific emphasis was placed on conducting an inclusive evaluation to the furthest extent possible. Both parent and child focus group discussions were mixed-gender groups with roughly equal representation of males and females. Data collection tools were designed to examine gender and/or inclusion-related dynamics in relation to the proposed evaluation. Research Assistants who led the discussions were also trained to focus on ensuring that the voices of all participants will be heard. In addition, Proteknôn worked in close collaboration with Save the Children to ensure that participants (adult or child) with disabilities were actively identified for participation and that any individual accommodations were catered to. Proteknôn sought to ensure full compliance with any applicable requirements in relation ethical clearances.12 In order to facilitate data collection at the national and sub-national levels, relevant representatives of MINEDUC, REB and MINISPOC were formally informed of the evaluation purpose and planning by USAID prior to any contact with local authorities. In addition, Proteknôn was provided with a copy of these introductory letters. STUDY LIMITATIONS The strength of qualitative-focused work is that it enables us to examine questions such as ‘how’ and ‘why’ a program like Mureke Dusome operates as it does because the findings draw from the expertise and insights of individuals and groups who have intimate knowledge and experience with the program. One challenge is that people experience the program differently based on factors such as their position of authority (head teacher, government minister, education officer) and factors such as gender, age, geographic location, and so on. While this can be read as a limitation, we also saw this diversity of views as a strength in that it will allow us to unpack and report on the complexities of Mureke Dusome to offer targeted findings and recommendations for sustainability. Another challenge was the possibility of a positive response bias. In other words, there may have been little incentive for our respondents to speak critically of a program they may have benefitted from in some capacity (e.g., services, employment, partnership, etc.) - or hoped to benefit from in the future. This may be even more an issue since the five districts targeted for the evaluation are the same that have been selected as ‘model’ districts in which to undertake the cost-extension phase. To anticipate this potential threat to validity, we stressed to participants that the overall aim of this evaluation was simply to understand the factors that shaped the program’s effectiveness, with the goal of long-term sustainability. Our study team also stressed the importance of informed consent, assent and confidentiality. Fieldwork facilitation skills were practiced and draft question guides for both key informant interviews and focus 12 After consulting with Save the Children and USAID, formal ethical clearance from the Rwanda National Ethics Committee (RNEC) was deemed to be not required. This was because the purpose of the proposed evaluation was to assess ongoing activities, with consultation only with direct stakeholders and beneficiaries of the Mureke Dusome program. Similarly, clearance from the National Institute of Statistics Rwanda (NISR) was deemed unnecessary as existing quantitative data sets were to be used for analysis, without any further collection of primary quantitative data as part of this evaluation. 24 group discussions were piloted at an additional field site prior to the start of formal data collection. This allowed an additional opportunity to debrief on any practical, linguistic or conceptual challenges faced at field level before finalizing the data collection tools. The limited timeframe and parameters around budget shaped our fieldwork strategy. For example, there were several instances when key informants at the sub-national level were unavailable during the research team’s scheduled presence at their locality. In order to facilitate participation wherever possible, some interviews were conducted by phone, potentially limiting the level of the depth and rapport with study participants. At the national level, two of the identified key informants did not respond to multiple attempts to schedule interviews. Finally, another limitation is that it is not possible to generalize findings from a qualitative study across the country. However, by conducting in-depth work in 10 sectors across 5 geographically dispersed districts, and by developing a data collection plan that aimed to arrive at conceptual saturation of key themes, study findings do provide a good level of insight into Mureke Dusome that can be relevant to other parts of the country. Our reliance on the 2018 EGRA also came with limitations. We note that extraneous factors could influence how children responded during interviews.13 However questions included in the 2018 EGRA context questionnaire were formulated to ensure that they could easily be understood and answered by children in the target age range. The 2018 EGRA context survey did not collect data on the availability and type of available reading materials at home, frequency and time of participating in community-based reading activities, or data on the frequency and amount of time spent on practicing reading at home. The survey also did not capture information on the literacy of primary caretakers. Each of these factors posed limits to the depth of our analysis and qualifies the types of conclusions we are able to form. 13 Dubeck, M. M., et al. (2015). The early grade reading assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development, 40, 315-322. 25 FINDINGS Principal findings from the data collection and analysis are presented according to the three core evaluation questions (EQs) that guided this study. Where relevant, findings have been grouped thematically, with due consideration of the sub-questions. EQ1. HOW HAS MUREKE DUSOME IMPROVED LITERACY-RELATED KNOWLEDGE, ATTITUDES, AND PRACTICES? Evaluation Question #1 speaks to the effectiveness of Mureke Dusome’s efforts to improve literacy-related knowledge, attitudes and practices (KAP) to ultimately improve early grade reading amongst Rwandan P1, P2 and P3 students. These findings draw principally from qualitative data gathered at the national and sub-national levels, and have been organized around the following four key thematic areas: ● fostering a culture of reading; ● building school-community partnerships; ● supporting the Rwandan children’s book industry; and, ● promoting equity and social inclusion. In each sub-section, we aimed to consider evaluation sub-questions, including perceived contributors to change, contributions to capacity building and systems strengthening, as well as challenges experienced along the way. Summary: key findings related to effectiveness • Social and behavior change activities, combined with policy advocacy, improved access to books, and local capacity building was a potent recipe to improve literacy-related knowledge, attitudes and practices. • Successful communities transformed their community culture through active monitoring of reading clubs by school leaders; linking program activities to imihigo (e.g., reducing school dropout); follow-up by teachers and parents at reading clubs; and utilizing local structures like umuganda and FBOs. Communities were less successful if they lacked these characteristics, had high turnover of Literacy Champions, held reading clubs infrequently, and received limited engagement from local officials. • Capacity building and system strengthening of the Rwandan publishing industry included: enhancing capacity for authors, illustrators and publishers; encouraging new authors through Abana Writers Café; stimulating supply and demand of books; and establishing the Rwanda Children’s Book Organization. • Mureke Dusome’s contribution to capacity building and system strengthening of schools and local government included providing training manuals to facilitate self-study and offering settings of communities of practice for SGACs and head teachers. • Less successful aspects of Mureke Dusome included the possible de facto exclusion of some children with disabilities due to lack of training for Literacy Champions; a mismatch between expectation of stakeholders and the Mureke Dusome workplan; miscommunication about extension of the program, and insufficient number of new books to continue to stimulate children. 26 • Mureke Dusome was perceived as having national coverage, but the design of one reading club per public government school meant most of Rwanda’s 14,000 villages did not have a club. If there was one overarching finding, it was that Mureke Dusome’s efforts to improve literacy-related knowledge, attitudes, and practices sought to be comprehensive. The program attempted to shift perspectives and behaviors around literacy by altering practices and policies. And conversely, it attempted to shift practices and policies by altering perspectives and behaviors. Collectively, Mureke Dusome interventions sought to achieve this goal by aligning systems, activities, and stakeholders at multiple levels with the ultimate objective of promoting children’s literacy in Rwanda. FOSTERING A CULTURE OF READING At the outset, Mureke Dusome successfully identified the lack of a reading culture in Rwanda as one of the major barriers to early grade literacy amongst Rwandan children. The program therefore ambitiously sought to instigate a nationwide attitudinal and behavioral transformation at multiple levels to foster a culture of reading for the first time. Interventions ranged from the grassroots community level up to the national level. It was acknowledged that for a culture of reading to take root, children would need to want to read more often than they currently do. For that to happen, they would need access to high-quality, age-appropriate books as well as the opportunity and encouragement to read those books. To bring about these changes amongst children and their communities would require systemic strengthening of key actors and institutions at national level. Building a culture of reading amongst children and communities: “Mureke Dusome has incited the envy of reading among children. Even the little ones, you can see that they are curious to know. We have a wall where our values and prohibitions are written and you can find that a P1 child is trying to read!”14 A key starting point to fostering a culture of reading is to first incite a love of reading amongst children. Mureke Dusome struck a good balance between supporting children to develop literacy skills to improve their school performance, but also – importantly – to showcase how reading could be used to have fun, play games, and embark on new forms of creativity, exploration and imagination. Emphasis was placed on the ability of literacy skills to open up possibilities for children to understand and engage with their world in different ways. In short, literacy promotion activities under Mureke Dusome were not only about the act of reading. Rather, the broader purpose was to highlight new ways to learn, to think and to question. “Currently you find children exchanging story books and even the younger ones like those kinds of books with pictures of planes and vehicles. And then they start to develop the curiosity that ‘I will be a pilot, I will make a car, I will not be a street wanderer, I will be an artist if I study well.’ I have a child who goes in the reading club. She is 6 years old. She often tells us that she will also write a book 14 KI, Head Teacher 27 with more pictures of intore with imigara [Rwandan traditional dancers who wear a crown] and guitars for children to become artists.”15 One of the primary ways that Mureke Dusome’s intended objectives were operationalized was through the creation and facilitation of reading clubs. These were led by community volunteers called Literacy Champions and monitored by Save the Children’s implementing partner, Umuhuza. Over 2,500 reading clubs were established in every village with a state-sponsored primary school across the country. They convened at designated times each week, and included interactive activities such as reading stories, letter identification and songs. While reading clubs were intended for P1 to P3 students, they were frequented by children of all ages – often the younger or older siblings of those formally targeted by the reading club. Literacy Champions worked closely with Head Teachers, SGACs and local leaders to stimulate and maintain children’s attendance at reading clubs, encouraging parents to allow their children to routinely participate. Reading clubs helped to introduce a culture of reading to communities across Rwanda. They provided more children with increased access to reading materials, particularly for children from poorer households who could not otherwise afford them. Through the conduit of reading clubs, Mureke Dusome created spaces for engagement with books. As a result, children had not only more access to reading materials, but also more opportunities and motivation to read. Reading clubs replaced otherwise idle time with books, learning, and creativity. While practicing reading skills outside of the classroom likely had corresponding knock-on benefits for children’s academic performance at school, the reading clubs themselves also became part of a quasi-formal monitoring system to help address school dropout. Literacy Champions were responsible for monitoring attendance at reading clubs so that they could follow up on any absentee children. Moreover, some children opted to attend reading clubs - even when they were no longer in formal schooling – because they wanted to learn how to read. In some cases, children who had previously dropped out of school were encouraged to attend and/or parents of dropouts were sensitized on the importance of schooling, thanks to Literacy Champions and the reading clubs they ran. This was also some evidence in the case of several children with disabilities. They were reportedly motivated to re-start school after having positive experiences at the reading clubs. Through the reading clubs, children also learned to value books themselves. They were taught how to handle and treat books with care. One mother shared her testimony that: “Before Mureke Dusome, children did not know the value of a book. They would damage books. When you gave them books, they would not even remember where they put them. But now my child knows where to keep a book. He remembers well where he put it. He keeps it in his bag or gives it to me to keep it for him and comes back to request it to me when he needs it.”16 Parents themselves also learned the importance of caring for books through sensitization activities conducted by Save the Children’s partner organization, Urunana DC: 15 KI, Village Leader 16 FGD Parents, Burera District 28 “The most important thing is that Urunana encourages parents not to tear children books as they used to do before when searching for paper to smoke cigarettes.”17 Many local respondents, including school and local leaders as well as parents, explained that part of the benefit of the reading clubs was that it kept children occupied with something productive to do. It prevented children wandering in the community and on the streets. This fed into the perception of local stakeholders that reading clubs were also an avenue through which they could collectively reduce repetition and curb dropout, while keeping children engaged in activities that could stimulate their cognitive development and contribute to their overall academic performance. One father said: “Reading clubs make children focused. They would be wandering the streets or in the community but they got extended time to learn more and it has improved their skills in different aspects.”18 If a child was absent from reading club, Literacy Champions were in a position to follow up with parents and schools in a timely manner. As one Literacy Champion put it: “The reading club in our community helped 10 school drop-outs return to school because the activities of reading club encouraged them to learning and reading. For example, there was a school drop-out who was eight years old. He used to wander around in the community. But he liked to come to our reading club that was held under a tree. Then I called him and told him “Would you join us and read books?” “I do not know to read,” he told me. “Did you study?” I asked him. “I went there once and left,” he said. Then I encouraged him to join our reading club, but it looked like he was not interested with reading. I used to cross with him on the street and kept encouraging him and told him, “Would you please keep coming to the reading club! Haven’t you seen or heard that we have interesting stories and even return to school!” Then he laughed at me. Later I met with his mother and encouraged her to send the child back to school. The mother did it, but the child would go to school rarely, but later he kept going to school regularly. Now he is in P2 and currently he comes to the reading club.”19 Some key informants said that the best reading clubs were those that were located in the vicinity of the school. This proximity helped facilitate the involvement of school leadership, particularly in terms of follow up on children’s attendance at reading clubs, management of books and other club materials, as well as motivating Literacy Champions in their work. Where school leadership was positively engaged with these community-based activities, it was seen as a vehicle for effective implementation and governance. In some cases, for example, reading clubs could use the nearby school to safely store books and other materials. Proximity to school was also a key factor for children. With the double-shift format of primary schools in Rwanda, children could attend reading club in the morning, then class in the afternoon, or vice versa. Another way in which Mureke Dusome contributed to a culture of reading in Rwanda was through the 17 KI, Sector Official 18 FGD Parents, Ruhango District 19 KI, Literacy Champion Representative 29 focus on learning Kinyarwanda. Parents, especially, felt that the emphasis on Kinyarwanda language was important to them for diverse reasons. Many saw Kinyarwanda literacy as a basic life skill to which they could relate easily. Some also felt that their mother tongue was no longer sufficiently privileged or prioritized in present-day Rwanda, where children in later years learn in English at school. In this way, Mureke Dusome’s focus on Kinyarwanda language books with locally relevant storylines and characters was an important part of teaching children to discover, value and appreciate Rwandan culture. One mother explained: “To me before Mureke Dusome came, Kinyarwanda has started to disappear totally. So Mureke Dusome has brought a good culture of making alive Kinyarwanda language. So we have to protect our language and teach it to our children.”20 Another mother said: “My child used to dislike Kinyarwanda because it is a difficult language, but now she likes it thanks to the songs, games and reading books they learn in Mureke Dusome. They made her like Kinyarwanda.”21 In areas near the Ugandan border, where children do not routinely speak Kinyarwanda at home, the focus on Kinyarwanda language through reading clubs was especially helpful. One father explained: “Reading clubs help our children to know Kinyarwanda. Our children speak another language [Urukiga] and so with Mureke Dusome books those who attend become more fluent in Kinyarwanda than those who don’t.”22 Literacy Champions were valued by local respondents because they came from their own communities and were selected by their peers. Literacy Champions were seen as a linchpin for community￾based literacy efforts. They were critical to forming linkages between Head Teachers and SGACs at school and reading clubs in the community. They supported the day-to-day functioning of reading clubs, facilitating activities for children to engage positively with reading materials. Children also valued the role Literacy Champions played in improving their literacy skills. One child explained: “They give us books and then they read for us. Learning Champions read for us and we follow in our books. After that they ask us to read one by one. Last week we read the story of “Isazi n’Akanyoni” (Fly and Bird).”23 20 FGD Parents, Ngororero District 21 FGD Parents, Gasabo District 22 FGD Parents, Burera District 23 FGD Children, Ngororero District 30 Beyond routine weekly activities such as reading clubs, Mureke Dusome also explored other methodologies to promote literacy more widely. One particularly successful method was the promotion – and later institutionalization – of Umuganda Literacy (or literacy events held during monthly community works). While adult community members were busy with routine Umuganda tasks, children were productively occupied by Literacy Champions in reading practice sessions. As one key informant put it: “Mureke Dusome successfully identified community structures which could be leveraged for literacy like Umuganda – traditionally used for physically laborious tasks. But Mureke Dusome showed that the time could be used to productively engage children in reading too. This changed the mindset of how to capitalize on this time for children too, not just adults.”24 These public events offered children an opportunity to receive recognition and praise for their newly developed reading skills. They were able to build self-confidence and self-esteem while reading aloud in front of a public audience. At the same time, parents witness and share pride in their children’s achievements. Local leaders were also able to capitalize on such events to share key messages concerning literacy, fostering greater buy-in and collective ownership of literacy promotion activities. One Sector Education Inspector confirmed that: “Such organized events have been the effective channel to spread the message of how reading culture is important to all people in general and many parents currently know and value the importance of reading.”25 In addition to Umuganda Literacy events, Mureke Dusome sought to further the development of a reading culture by organizing reading competitions. During competitions children were assessed on their ability to read a given passage of text within a limited timeframe. Reading competitions were seen as a way to celebrate the importance of literacy, highlighting particularly successful children publicly in a way that could motivate and encourage themselves and others. These competitions were reportedly taken seriously by communities, with parents and local leaders in attendance. Reading competitions were an opportunity for children to demonstrate their skills and gain recognition in front of their peers, their parents, and even local leaders. Winners were sometimes allowed to continue on and compete at sector level. Top performers could win prizes like notebooks and pens. One father explained: “In partnership with school management, we organized reading competitions. The school provides awards to encourage children to keep reading. The awards are like pens and notebooks. At previous reading competitions, Mureke Dusome intervened and provided books as awards to the best readers and the school provided notebooks.”26 Institutionalizing a national reading culture: Mureke Dusome’s strategy at the national level was to work with individuals and institutions to effect 24 KI, National 25 KI, Sector Official 26 FGD Parents, Ruhango District 31 and embed change. National engagement required gaining buy-in from multiple key stakeholders, and this necessitated a reliance on relationships. National level key informants emphasized that the working relationship with key actors in the education sector was important on an institutional level. Moreover, these institutions were proactive in building strong relationships with key individuals within the education sector that could effect change from the inside. For example, Mureke Dusome had a Save the Children staff member embedded within REB’s School Leadership Unit (SLU) with dual line management between REB’s SLU Director and Mureke Dusome’s Deputy Chief of Party. This type of relationship was indicative of Mureke Dusome’s focus on the importance of capitalizing on relationships to bring about systemic change. One key informant captured it well, explaining that “Mureke Dusome has been able to harness the cases of individuals who have been passionate about literacy.”27 While enthusiasm itself is not something that can always be relied upon, it is one of the factors of success of the program. Mureke Dusome sought to capitalize on the power, capacity and enthusiasm of individuals within the education sector, even at the highest levels of government such as the Minister of State in Charge of Primary and Secondary Education: “The persona of the Minister of State and his personal investment in bringing together a diverse group of actors who now understand their individual and collective role in literacy promotion and its prioritization in Rwanda was absolutely critical.”28 According to many respondents, one of the most promising systems-based results of Mureke Dusome was the project’s efforts to co-develop Rwanda’s first National Literacy Policy. The policy is an important milestone for promoting a culture of reading, because some of the key priorities and components of the Mureke Dusome program are embedded in the policy itself. In other words, the policy has the potential to institutionally embed previous programmatic interventions within the policy architecture of the Rwandan education system. Once approved, the National Literacy Policy ensures that community-based literacy initiatives would no longer be the purview of a donor-dependent program alone. Instead, they would be formally recognized as a government priority. The policy was characterized as “a great milestone” and “a huge achievement” by various national key informants who contributed to its development. It is evidence of the success of Mureke Dusome in convincing MINEDUC and REB of the importance of literacy promotion, and community-based initiatives within that. While it is not yet officially approved, the policy can be considered both a product and a process in the sense that it highlights Mureke Dusome’s efforts at engaging national level stakeholders while also laying the groundwork for future literacy-focused initiatives and programs through policy change. While the policy development itself may have seemed off to a slow start, several key informants noted that this was inevitable and even necessary to ensure the policy had the right buy-in to be successful. This was done through wide consultation and stakeholder meetings, under the leadership of Mureke Dusome and in close collaboration with government. Ownership and future stewardship of the policy by the current Minister of State is clear. Similar to the National Literacy Policy, Mureke Dusome played a pivotal role in establishing the Soma Rwanda Steering Committee (previously known as Rwanda Reads). It is the only platform in Rwanda that regularly brings together different actors within the literacy space, straddling local civil society organizations, national and international NGOs, donors, private sector companies and government representatives. Periodic meetings are now co-chaired by the Minister of State on behalf of MINEDUC. This was seen as a necessary and positive shift from the previous set up, in which REB was the co-chair, 27 KI, National 28 KI, National 32 because of the potential for the Minister of State to engage directly in a cross-ministerial approach. Whereas REB is an implementing agency, MINEDUC can interface more effectively with other ministries such as MINISPOC and MINALOC. This recent change has not only raised the profile of Soma Rwanda but has significantly increased its strategic potential to advocate and systematize change within the education sector. This was already evident in the Minister of State’s personal role in ensuring that Umuganda Literacy events become a national phenomenon by taking place in all 30 districts in September 2019. While Umuganda Literacy activities were first initiated by Save the Children, it has now been taken over for future leadership by the government itself. Soma Rwanda has been facilitating the celebration of National Literacy Month since 2016 (and in 2014 and 2015 it helped celebrate National Literacy Day). Soma Rwanda members organized events throughout the month. In 2019, the scope of celebration was extended beyond Kigali. Some key informants felt that Soma Rwanda has even greater potential, for example, as a conduit for building an evidence-base around the value of early grade literacy. It could be used as a platform to develop more information and data on the impact of a lack of literacy on national development in Rwanda. A more fact-based account of the current shortcomings of literacy in Rwanda could provide more authority to advocate for investments in order to improve Rwanda’s gross domestic product (GDP), economic growth, Human Development Index, and so on. Informants noted that this has been a successful campaign in the Early Childhood Development (ECD) space that could potentially transfer over to early grade literacy too. In the minds of education sector partners, Soma Rwanda is synonymous with Mureke Dusome. While newer and bigger actors and agencies are now populating the education space in Rwanda, Save the Children has retained its visibility as a leader in literacy promotion, largely thanks to the National Literacy Policy development process and Soma Rwanda. Mureke Dusome also invested in improving public and community libraries by providing Kinyarwanda children’s storybooks. Mureke Dusome was one of the first to see the potential of community libraries and book banks in helping to advance a culture of reading. By garnering media attention via television coverage and online messaging, these community libraries gained more visibility amongst local populations. Following a Joint Assessment with MINEDUC and (formerly) MINISPOC in January 2019, Mureke Dusome allocated 500 Kinyarwanda books to each of the 40 community libraries which were found to be functional. Given the future strategic planning by the government to upgrade 20 libraries and build 22 new ones,29 Mureke Dusome has paved the way in highlighting the need for designated reading materials for children, as well as, the need for these materials to be available in Kinyarwanda language. There are also future plans to provide training to librarians. As such, Mureke Dusome has been a recognized partner to RALSA which has a formal duty to promote reading and writing culture in Rwanda. As one key informant put it, “The interest of young children in these community libraries is just beginning.”30 Although community libraries serve a larger population of all ages, there are – or should be – clear links between reading clubs for younger children and community libraries, with parallel links between Literacy Champions and librarians. However, where the 40 community libraries may be located at district/sector level – and still not in every district at present – the over 2,500 reading clubs at school/village level were physically closer to children and therefore much more accessible. As community libraries continue to multiply as part of government strategic planning, there could be more opportunities for Mureke Dusome to exploit the inroads it has already made in supporting them. While in 2015 a decision was taken to move school and community libraries under MINEDUC, policy oversight over all kinds of libraries in Rwanda remains with MINISPOC. Given Mureke Dusome’s existing strong collaboration with both line ministries, it is uniquely positioned to help move this agenda forward in a way that serves young readers. 29 As per KI, National 30 KI, National 33 BUILDING SCHOOL-COMMUNITY PARTNERSHIPS Although Mureke Dusome is primarily focused on community-based interventions, the program was intentional in viewing schools as a hub through which community outreach could be extended. Mureke Dusome attempted to connect the dots between school leadership, parents/community and children. The creation of reading clubs is one example of school-community partnerships promoted by Mureke Dusome. Rwanda Education Board has defined School Leadership Standards, one of which is dedicated to school￾community partnerships. Mureke Dusome played a critical role in helping to operationalize this standard through the development of relevant training modules. Working alongside its partners, Mureke Dusome articulated how schools and communities can partner to align interests and inputs to improve children’s literacy. Through these efforts, they introduced new forms of implicit and mutual accountability between schools, communities, and parents. The program’s effectiveness in promoting literacy-related knowledge, attitudes and practices along the school-community nexus therefore relied upon both Head Teachers (or their deputies) and School General Assembly Committees (SGACs) as two key entry points. Both received training through Mureke Dusome’s interventions, which helped to solidify their understanding of the importance of community and parental engagement to promote literacy (with clearly positive implications for student learning outcomes). Indeed, Mureke Dusome helped make the case that literacy requires more than just classroom intervention. As one high-ranking government official put it: “Before the mindset was that reading was ‘just the business of the classroom’ but there has been a change in understanding. Now people understand that literacy is a foundational skill for later learning and must be supported in the early years.”31 For Head Teachers, it was important to recognize and internalize the message that community-based actions outside of the school perimeter also fell within their purview. Greater awareness of how reading clubs led by community volunteers, for example, could have positive knock-on effects for classroom performance in terms of early grade literacy was an essential component of this. Where Mureke Dusome was more successful, Head Teachers went on to personally support these initiatives by providing informal support and supervision to Literacy Champions and reading clubs. In some instances, Head Teachers valued the contribution of these activities so much that they opted to use part of their school’s capitation grant to hire Literacy Champions at the nearby reading club for temporary positions at school such as cleaners or security guards. This was means through which Head Teachers could offer financial incentives to Literacy Champions in order to maintain their levels of motivation. For SGAC members, training provided by Mureke Dusome was critical to better understanding – sometimes for the first time – their roles and responsibilities as committee members. Often members were unsure of their remit and thought that it was limited to oversight of school leadership – limiting further opportunities to contribute to improved student performance. Respondents indicated that prior to Mureke Dusome’s interventions, the role of parents at SGAC meetings was principally logistical and administrative, focusing on questions of good governance. Mureke Dusome helped to redefine this relationship such that parents, teachers and school leadership (Head Teachers and Deputy Head Teachers) could also encompass mutual accountability in promoting literacy, both in and outside the classroom. The SGAC meetings themselves became a forum through which teachers and parents could meet to discuss issues salient to their children’s educational experience, helping different actors to view literacy as a collective effort, not limited to the purview of the classroom alone. 31 KI, National 34 SGACs went on to play a pivotal role in the mobilization of parents, largely based around awareness￾raising and sensitization efforts. Many respondents agreed that parental engagement has subsequently improved. For example: “The Mureke Dusome intervention has been effective in this district in terms of early literacy promotion through reading clubs and partnership between parents and school leadership. And this partnership has been impactful in these areas - parents are now involved in their children’s learning. At school, children are supported by teachers and at home, parents take the lead. A learning process is no longer the school’s responsibility only but also parents'. This partnership has also been promoted through training that Mureke Dusome provided to teachers and committee members who went to sensitize others in the community to be more involved in their children’s learning and reading.”32 Another said: “Parents’ involvement has increased. They have understood that the success of their children at school is not only for school’s responsibility, but they also have a role to play. We could not have been able to strengthen SGACs without the contribution of Mureke Dusome.”33 Concretely, SGACs and Head Teachers were able to work together to encourage parents to send their children to reading clubs. This happened through different means, including – for example – by school leaders during general assembly meetings where they reminded parents to ensure their child’s attendance at reading clubs. Others would initiate additional awareness-raising activities: “We did sensitization sessions aimed at calling upon parents to encourage children to attend reading clubs. In addition, we organized a literacy party at which children play games and drama. Therefore, their parents realized the value of reading clubs, and became conscious of what their children do, that they do not come to reading club in vain.”34 Not only did SGACs and school leaders instruct parents to allow their children to attend reading clubs, they also sensitized parents on why this was important. Study respondents referenced different ways in which they would convince parents, including citing links to the healthy cognitive development of children. Improving parents' knowledge related to the value of literacy helped shift parental attitudes more broadly: 32 KI, District Official 33 KI, District Official 34 KI, Head Teacher 35 “Parents’ attitudes and knowledge were changed by the program intervention. They currently give value to literacy and learnt the importance of the reading club and books. Actually, parents’ understanding on value of reading was improved.”35 There were also instances where school leadership mobilized parents to directly support reading clubs. Indeed, some of the best practices identified during the course of this evaluation included instances in which SGACs were able to transform parental attitudes and practices convincingly. In one school, for example, the Head Teacher was able to enlist the support of two parent volunteers to support each reading club session, in addition to the Literacy Champions themselves. With time, SGAC meetings became a space for improved dialogue and communication about literacy promotion. Communities in which Mureke Dusome was relatively more successful included those that offered Literacy Champions the opportunity to speak directly with parents during assemblies about Mureke Dusome’s interventions. Furthermore, it was an opportunity for teachers to take the initiative and find ways to contribute to the collective effort by mobilizing children to attending reading clubs: “During parents’ meetings, we always have a time where the parents’ committee gives the floor to Literacy Champions to talk about Mureke Dusome program and interest parents to send their children in Mureke Dusome. Teachers also are contributing for example when LCs need children teachers are cooperative to send them and they help LCs to develop teaching materials. We do whatever we can to assist them with our means.”36 Mobilization was not limited to school leaders and parents. The support of faith-based organizations was also sometimes enlisted, including through the use of churches to host reading clubs, or religious leaders ‘preaching’ about the importance of literacy to their congregations, thereby increasing the attendance of children at reading clubs and encouraging parents to read to their children: “The high priest of Gahara Parish mobilized parents in announcements in the masses he gave; he has really played a good role in the Mureke Dusome reading club. Parents also nowadays come to borrow books for their children.”37 In places where Mureke Dusome had truly taken root, an iterative and mutually reinforcing relationship was ultimately established. This allowed individual stakeholders – from both the school and community levels - to rely on each others' support as they worked towards a common goal. One Literacy Champion representative, for example, explained: “The school also contributes to the parents’ mobilization by visiting our reading clubs. During every parents meeting, we are given a time to talk to parents about Mureke Dusome and interest them in sending their children. With the school also we help each other. For example, at the school where I 35 KI, Head Teacher 36 KI, Head Teacher 37 KI, Head Teacher 36 am based I may have a book damaged and if I go to borrow a stapler to repair it, they lend it to me without any problem. They can lend to me a marker if I need to name new books I have received. So we have a good collaboration with the school.”38 These efforts to embed school-community partnerships did not go unnoticed. District officials in particular credited Mureke Dusome with offering training and support which helped align key stakeholders to bridge the school-community divide. One District Director of Education (DDE) stated: “Mobilizing and sensitizing parents to be SGAC members was the key of children learning and literacy promotion. Through SGAC meetings at school, parents are sensitized to encourage their children to read often and they even encouraged to read for their children, inciting the children to have a culture of reading out of school. Mureke Dusome has helped a lot on this. They have strengthened those committees by working closely with school leaders and also their volunteers called Literacy Champions. This has been very successful. Now parents are more involved not only in their children ‘s learning, but also with their reading.”39 Mureke Dusome adopted complementary and parallel approaches with local officials, Head Teachers and SGACs. Indeed, local officials played a key role in community outreach, awareness-raising, as well as, activity implementation and monitoring of Mureke Dusome activities. Although interventions were ostensibly community-based, Mureke Dusome sensitized Sector Education Inspectors and other education officials at district level that these activities warranted follow up given their overall contribution to literacy promotion and a wider culture of reading. Local leaders occupied a unique position in uniting how parents and other community members like religious leaders, as well as Head Teachers and other teachers viewed their roles and responsibilities in terms of children’s literacy. Local officials offered practical support on the ground, for example by selecting and supporting Literacy Champions, visiting reading clubs, and training SGACs. When appropriately engaged, local officials were also able to use their political and social capital to influence and direct changes in knowledge, attitudes and practices across a broad spectrum of stakeholders. This was largely done through sensitization efforts aimed at increasing attendance at reading clubs by sharing key messages during Umuganda or other community gatherings: “The role of local government authorities was to sensitize parents through community general assemblies and different meetings which gather community members and to encourage parents to allow their children to attend the reading club. Local government has no financial means to support the program, but what they do is to mobilize parents encouraging them to support their children's literacy as it was identified that children even would finish primary school without being able to read using Kinyarwanda.”40 BUILDING A BOOK INDUSTRY: CAPACITY BUILDING AND SYSTEM STRENGTHENING As a result of Mureke Dusome’s efforts, the landscape of the Rwandan children’s book industry looks, as 38 KI, Head Teacher 39 KI, District Official 40 KI, Sector Official 37 one key informant put it, “completely different now.”41 The programmatic model of effecting change from within the private sector was seen as innovative and pioneering. It has gained global visibility as an example of best practice for other countries that are looking to improve the availability and accessibility of high￾quality children’s reading materials. This model was characterized by several key components: ● Pioneering private sector collaboration; ● System strengthening through the capacity-building of local publishers, authors and illustrators; and ● Stimulating supply and demand by both direct book purchases and by creating a wider appetite for reading. Private Sector Collaboration: Unlike typical approaches by other development partners in Rwanda and elsewhere, Mureke Dusome took the bold step of working directly with private sector actors to transform the book industry from the inside. Where other actors might have resorted to import containers of books or develop materials in￾house to resolve supply issues for their literacy-based interventions, Mureke Dusome opted to collaborate with book publishers in Rwanda, including start-ups. This was largely done through the organization of technical trainings, including some delivered by international experts. By fostering a close working relationship with these publishing houses – identifying and responding to their capacity needs - Mureke Dusome was able to invest in their technical and commercial development with durable results. There are now over 30 Rwandan publishers who are active in the country.42 Even as the program winds down, these publishers continue their work. Today, they are producing more Kinyarwanda children’s titles and with greater quality than ever before. Nearly all respondents we spoke with recognized and appreciated the improvement in book quality compared to before Mureke Dusome. Systems Strengthening: The transformative power of Mureke Dusome’s work with the Rwandan book sector could be felt at multiple levels. The collaborative approach adopted within the publishing industry was reflected on a wider scale through Mureke Dusome’s efforts to bring together a diverse set of actors with a common goal of promoting literacy. The program was also able to capitalize on its visibility and standing to influence other key actors for systemic change, helping to harmonize approaches. For example, it encouraged other organizations and agencies to also procure books locally wherever possible, further stimulating the Rwandan book sector. The Soma Rwanda Steering Committee –a platform through which different literacy partners work together - was an important entry point for the systems strengthening aspect of Mureke Dusome’s work with the book industry because it is chaired by the Minister of State in Charge of Primary and Secondary Education. Abana Writers Café was another Mureke Dusome-led initiative to promote networking amongst various stakeholders interested in books, including publishers, writers and illustrators. It was an occasion for writers to share their children’s stories and receive considered inputs from others. For young and upcoming writers, this was especially encouraging as a safe space to grow their talents. The Abana Writers Café remains a key starting point for content development as the production of new titles depend on the authors who write them. Previously organized by Mureke Dusome staff, it has now been handed 41 NI, National 42 Save the Children. 2019. Country profile: book publishing in Rwanda. Save the Children. Rwanda Children's Book Organization 38 over to Sankofa Creatives – a local creative start-up – for the purposes of continuity. Mureke Dusome helped to strengthen systems by encouraging organizations to form and then helped them to transition to autonomous entities that no longer needed external support. One example of this is the program-initiated discussions that resulted in the formation of the Rwanda Publishers and Booksellers Union. Similarly, the Rwanda Children’s Book Organization (RCBO) began as a Mureke Dusome initiative. Now the RCBO functions independently of Mureke Dusome and is an active member of the Soma Rwanda platform. In addition, Mureke Dusome encouraged publishers to advocate for a draft book policy. Stimulating supply and demand: Mureke Dusome also helped to stimulate supply by increasing the demand for high-quality Kinyarwanda children’s books. Through its reading clubs and Umuganda Literacy events, facilitated by dedicated Literacy Champions, the program offered children the opportunity to access and engage with books outside of the classroom, exposing children to the joy of reading and thereby increasing their demand for fun and relevant books. Continual sensitization of local officials, school leaders and parents generated a wider understanding of the value of early grade literacy that also cultivated an appetite for reading. “My peers also encourage me to read and we go together in reading clubs on Tuesdays and Wednesdays. When we borrow books we read stories at home and people at home are happy.”43 Mureke Dusome itself purchased many books. Indeed, after REB, Mureke Dusome was the biggest purchaser of books in Rwanda and was a key commercial client of the book industry at the time. The ability of Mureke Dusome to work on both the supply and demand sides was a key factor in its success. Through its interventions, Mureke Dusome worked on both the content development side – ensuring books were high￾quality, age-appropriate and engaging to children - and the marketing, sales and distribution side. Because of its work to improve the book industry, the client base for publishers has expanded significantly beyond REB and Mureke Dusome. Concretely, Mureke Dusome was able to create channels and spaces for book accessibility where none existed before, particularly outside of Kigali and other urban centers in Rwanda. By distributing books to reading clubs and community libraries, children originating from poorer and/or rural households were able to access reading materials their families could not otherwise afford. “Mureke Dusome lends us books and we enjoy stories at home with my younger sister”44 In addition, these materials were specifically in Kinyarwanda language. Where books may have been previously available in urban areas, they were often in foreign languages such as English or French. Mureke Dusome’s emphasis on stimulating the production of titles in Kinyarwanda dovetailed well with the 2015 revised curriculum for lower primary students in Rwanda, which had a strong focus on Kinyarwanda literacy. Similarly, there have been efforts to produce books which promote inclusive messaging through their content as well as to publish braille versions of books – both initiatives that advance Rwanda’s policy on inclusive education. 43 FGD Children, Gasabo District 44 FGD Children, Gasabo District 39 “Rwanda Reads brings together different literacy partners, but Mureke Dusome is on the frontlines – it has supported the production and distribution of Kinyarwanda storybooks, where almost none existed before. Some of these have even been in braille, and contain gender and inclusive messaging.”45 Past and present challenges for the Rwandan book sector: While there is clear and collective agreement on the powerful impact Mureke Dusome has had on the Rwandan book industry, some problems persist. One of the most challenging is the perennial problem of book procurement, especially in a context of low purchasing power such as Rwanda. Where Kinyarwanda titles may be available and accessible to wealthier families living in urban areas at supermarkets or other retail points, many respondents are concerned that book procurement challenges remain particularly acute for poor and rural communities who cannot afford to buy books themselves at current market prices. Yet pricing is directly related to the cost of production, which remains difficult to regulate given the prevailing economic dynamics in Rwanda at present. As a result, book sellers do not have clear commercial incentives to penetrate the market beyond urban and peri-urban centers. Sustaining the accessibility of books in these communities therefore remains problematic, particularly given the poor book distribution infrastructure. While community libraries exist, they are few and far between with only 40 across the country. They are also unevenly distributed: at present, some districts are without even one community library. Thus, community-based literacy remains institutionalized within a limited number of community libraries and within school settings, without further diffusion into communities. This is a frustrating situation for all involved – publishers are producing more relevant content than ever before, while parents have been sensitized through Mureke Dusome on the value of reading, yet they remain incapable of providing these books for their own children. This challenge underscores the importance of reading clubs as a mechanism for children to access these books. “Mureke Dusome and others have not been able to answer the hard question of how to make books available to communities that lack purchasing power and financial resources. The reality is that it is expensive to make books in Rwanda.”46 Another challenge is that the book industry is still in its infancy and subject to economic shocks. With Mureke Dusome phasing out and REB’s recent decision to print school textbooks in-house, publishers risk financial struggles. In the long run, however, both schools and community libraries have potential to be a reliable market. Schools, for example, could use their capitation grants towards the purchase of books if school leadership chooses to recognize and prioritize the importance of reading for their students. Related to economic viability are concerns around the availability of financial means to continue monthly Abana Writers Café events. Previously paid for and hosted by Mureke Dusome, the initiative has recently been transferred to Sankofa Creatives which is a small start-up without a dedicated budget for the events. Concerned respondents strongly valued Abana Writers Café for its ability to foster young writers and encourage authorship of new Kinyarwanda children’s titles well into the future. One concern amongst respondents was in relation to the levelling of Kinyarwanda storybooks in line with recent REB guidelines. Some felt that a levelled approach to storybooks – even those designed for use 45 KI, National 46 KI, National 40 outside of the classroom – would enhance student learning outcomes by creating clear linkages between their in-classroom and at-home experiences. This would allow Literacy Champions or parents, for example, to guide children’s choice of book according to what they were concurrently learning at school, potentially providing a more coherent approach to the question of early grade literacy. However, others felt strongly that storybooks should not be made to directly correspond to grade levels at school. Rather storybooks and reading at home should be founded on creative and cultural learning and experience, prioritizing the pleasure and enjoyment of reading over its instructional value. It was felt by some that children should not feel limited by levelling or categorization of books, but rather should feel at ease to explore different titles freely. Kinyarwanda language books for young learners may also be particularly difficult to level as word length can be an issue, as well as the lack of alternative or simplified vocabulary. EQUITY AND INCLUSION IN LITERACY: OPPORTUNITIES AND CHALLENGES One theme we sought to explore throughout the evaluation process was whether and how issues of inclusion were acknowledged and accounted for through the planning and implementation of Mureke Dusome. Given that gender and disability can be, at times, separate, intersecting or overlapping factors that contribute to being marginalized in programming, the research team was intentional in probing these two aspects more specifically. This marginalization is not always immediately visible – sometimes de facto forms of exclusion can manifest in the ways in which interventions are planned, targeted, implemented, monitored and evaluated. “All children who have disability who can reach school or who study in formal schooling are welcomed into the reading club. There is no exclusion of children who have any kind of disability.”47 As the quote above suggests, participants were clear about the importance of inclusivity. At every level, study participants explained that discrimination or exclusion on the basis on gender, disability, or any other factor directly contradicted Rwanda’s approach to development efforts, which is explicitly as inclusive as possible. In terms of Mureke Dusome interventions themselves, study participants strongly endorsed the idea that no child or adult should be excluded on the basis of their disability. However, they were less concrete about how to operationalize this concept. Broadly speaking, there was often less of a focus on the specific provisions or individual accommodations that would be required in order to better guarantee the meaningful participation of children with disabilities. As a result, even though children with disabilities were not explicitly excluded, many noted that few – if any – attended reading clubs. Similarly, when it comes to gender there may not have been sufficient consideration for the differential ways in which boys and girls could be appropriately targeted and continually engaged in literacy-related activities such as reading clubs. It was noted that girls attended reading clubs more frequently and with greater regularity than boys. While the high levels of attendance of girls could be viewed as evidence of a changing mindset around the importance of girls’ education, the relatively lower attendance of boys may also be an indication that reading clubs did not sufficiently cater to their specific needs. Some study participants felt that the content of reading materials were not always interesting to boys, while others said that boys would have been better engaged through more dynamic or sports-oriented activities. Children themselves indicated that they liked to join reading clubs, but that they also had competing responsibilities such as household chores. These domestic tasks often had gendered dimensions that could limit their ability to attend a reading club. 47 KI, Sector Official 41 “Boys and girls participate equally. No one is excluded in the reading club. They are equally taken care of. Regarding children's performance, in general they perform well and boys require much more attention than girls because there are some who choose not to go to the reading club and instead go to work for money, because in this area children go find jobs peeling cassava. But currently there is a decrease because of the pressure of the school and local government authorities in eradicating that child labor. Serious punishments have been set for the people who employ and who use children in such activities.”48 In focus group discussions with parents, many felt that parental assistance operated roughly along the lines of gender, with mothers more likely to help children read than fathers. They suggested that this was because women are more likely to be in the house, while males were characterized as either being away from the home at a bar or working, and when they were around, they tended to play more of a disciplinarian role. “Most men are not available at the time children leave the schools. Some men are still at their jobs at that time, while others are in bars getting together with their friends.”49 “I think women contribute approximately 75% [of parental support] because they are the ones that are with children more than men. I am in charge of security in this village and I walk around in this community. From my observation women are the ones that help children practice what they learnt at home more than men, because men for most of the time are not at home - they are gone in bars or somewhere else.”50 Despite acknowledging the importance of equity and inclusion, it was not clearly prioritized in practical terms. Some of the key challenges are outlined here: ● Disability awareness was often insufficient. Knowledge was often lacking regarding the potential diversity of disabilities, particularly in terms of less visible impairments. Where physical disabilities were referenced, accommodations to improve accessibility or mobility were possible. This was often not the case for other sensorial, intellectual and mental impairments or developmental delays. “We still have the challenge of those specific disabilities - visual and sensorial - because we don’t have materials and skilled staff to take care of them.”51 ● These limitations were also evident in the ways in which many study participants spoke about disability. Despite collective agreement on the importance of inclusion and non-discrimination, there was a lack of sensitivity in the use of terms to describe issues around disability. This could be seen in the use of derogatory terms in the local language of Kinyarwanda which are inherently 48 KI, Head Teacher 49 FGD Parents, Ruhango District 50 FGD Parents, Ruhango District 51 KI, Sector Official 42 pejorative and can lead to increased stigmatization.52 “Children who have disabilities are not excluded in this program. They are also taken care of … for example, there is a child who has a problem with their eyes. They can’t see well. In such a case, the teacher and literacy champions put him in front to help him read easily and to make follow-up to ensure if he has seen what they wrote … We are lucky that we have a very low number of children who have disability and for many of them, it's a physical disability. The children concerned in relation to the Mureke Dusome program are only two. One has a skin disability (albinism) and the other has a physical disability. Both attend the reading club, so there is not any discrimination.”53 ● Literacy Champions had limited resources for working with children with disabilities. Although they had received the training toolkit which stated that they should make an effort to welcome every child, it was difficult to know how to practically support individual children with diverse impairments without appropriate training or materials. “In terms of disability, the place where children meet is very inclusive. It is a friendly and safe environment and accessible for children with physical disability. But I think maybe Mureke Dusome didn’t think beyond physical disability … they should have thought how those children [deaf and blind] should be included, especially in reading clubs, like training on basic skills literacy with disabled children. They should have trained Literacy Champions about sign language for deaf children and how to treat those children with special needs among the other children who are physically able.”54 Finally, it is worth noting Rwanda’s challenging topography can also limit easy movement of children with physical disabilities. The limited government support system to cater to the needs of all children with disabilities (physical, sensorial, intellectual) across the country limits effective interventions by Mureke Dusome. ADDITIONAL CHALLENGES FACED BY MUREKE DUSOME There were other cross-cutting challenges faced by the Mureke Dusome program as a whole which limited its effectiveness. On balance, one of Mureke Dusome’s weakest areas was the ongoing monitoring of activities in the field. While the majority of key stakeholders at the local level were largely enthusiastic about the program, many were also frustrated with what they felt was insufficient follow up. Site visits and reporting mechanisms were cited as lacking. In other words, there was a tendency to view Mureke Dusome as a solid concept, but one which could have benefited from a greater imprint by the NGO implementer in the communities with which it worked. These frustrations were felt on multiple levels, ranging from district/sector officials to school leaders to Literacy Champions. As one Head Teacher put it: 52 Recommended terms in Kinyarwanda have been published by the National Council for Disabled Persons (NCDP) in Rwanda and can be found at: https://en-gb.facebook.com/ncpdrwanda/posts/2918709458156357. m 53 KI, Head Teacher 54 KI, District Official 43 “There was lack of communication and follow-up by Mureke Dusome. They introduced an interesting program, but they did not conduct visits. There was limited follow up on reading clubs and at school. They should be visiting the clubs more often.”55 Similarly, Literacy Champions and their representatives at the sector level felt that there was a lack of ongoing communication about the program: “No follow up is done on activities assigned to Literacy Champion Representatives to ensure the sustainable implementation of the program. It is so difficult that three or four months can pass without receiving any phone call or feedback request from the program leaders.”56 There were some reading clubs that reportedly never received a site visit during the duration of the program, and others that received very few over a long period of time. This lack of physical presence at field level was in sharp contrast to other education development partners who had staff permanently based at district, and sometimes even sector, level. Some of these agencies were also funded by USAID and therefore were perceived by the external environment as being somehow linked – and therefore comparable - to Mureke Dusome. However, Mureke Dusome’s financial envelope was substantively smaller – a fact that was not always well known to external actors. This may, at times, have led to some negative misperceptions and misunderstandings about Mureke Dusome’s staff, partners and interventions. One JADF officer stated, for example: “We don’t how Mureke Dusome’s budget is managed, which is a barrier for us to monitor their activities. It is not easy to evaluate or monitor the project only in terms of time duration and ignore money value. This is not possible at all. I know that Mureke Dusome has been working with the District Director of Education closely but, as JADF, they have not been very cooperative with me. There is a lack of communication. But it should be more effective with JADF. There was a lack of coordination. They never attend the open day meetings that I have organized.”57 Mureke Dusome’s light operational footprint was directly tied to its budgetary restrictions. The scale of the program was not matched by the resources available. Staffing, in particular, was very limited. Umuhuza, for example, had just four community engagement officers to cover all 30 districts. While it could be argued that the program was perhaps too ambitious given its tight budget, it could also be said that what Mureke Dusome was able to achieve at national scale with such limited resources was quite remarkable. These findings point to the question of ownership and incentive for follow-up. For instance, in the context of Rwanda, if the motivation for reading clubs originated at the local level as a home-grown solution, would the community have needed external support for implementation? Mureke Dusome and reading clubs were not home grown, per se, but they were introduced in consultation with local communities and in collaboration with local partners. And while community members claimed ownership of the program, perhaps there were also limitations in the degree to which ownership could genuinely be claimed since 55 KI, Head Teacher 56 KI, Head Teacher 57 KI, District Official 44 the program was not home grown or government initiated – and whether this will change with the infusion of reading clubs into the National Literacy Policy. To be sure, this question is not limited to Mureke Dusome, NGOs, or even Rwanda. But it is worth thinking about when it comes to expectations concerning program effectiveness, adoption, and sustainability. Findings also raise a question about whether there may have been a specific need or gap in knowledge that the community was not able to fill and thus needed external support from Mureke Dusome. External perceptions of Mureke Dusome have not be helped by the fact that the program was scaled up incrementally to cover all 30 districts of Rwanda, only to be significantly scaled back to only 5 districts during the cost extension phase. While stakeholders from the remaining 25 districts were not targeted as part of this evaluation, it was clear from interviews held at the national level that representatives of line ministries and partner agencies were not all previously consulted, fully informed, or made aware of this decision in a timely manner. This may have led to some additional frustrations. It could also be argued that the fact that stakeholders have communicated their disappointment about Mureke Dusome’s scale-down is evidence that the program’s presence and value had been recognized and appreciated. Exiting from 25 districts has been made more problematic by the limited interface between Mureke Dusome and local government. The lack of clear and consistent coordination at both the national level with the Ministry of Local Government (MINALOC) and local level with JADF officers, DDEs, DEOs and SEIs was seen as another limitation to Mureke Dusome, particularly in terms of the long-term sustainability of its interventions. While the program did a good job of raising the profile of literacy and cultivating an appetite for related activities, this did not always translate into concrete measures by local government officials to increase ownership. (This will be discussed more in Evaluation Question #3.) This is primarily done through avenues such as performance contracts known as imihigo and district development plans as well as school improvement plans (that are monitored by SEIs as part of their role). When probed during the course of this evaluation, it was clear that community-based literacy promotion activities or monitoring of reading skills was not consistently featured in any of these documents. The limited visibility of such activities in formal state and school documentation is indicative of a limitation to embed these interventions and ensure adequate capacitation, handover and ownership for the purposes of continuity. It is unlikely that such activities will be prioritized without more concrete systems strengthening. At the same time, there could have also been other ways in which Mureke Dusome was embedded in different programs that did not emerge in the study. For example, if Communities of Practice and peer learning circles are in school plans, improved SGAC functioning can potentially be sustained. The National Literacy Policy is also a clear indication of ownership at the national level. In practical terms, reading clubs also had a number of challenges that limited their effectiveness. While the reading club was established where the primary school was located (typically at cell level), the related catchment area for students covered multiple villages. This meant that reading clubs were sometimes too far away for children from neighboring villages to attend. In order to mitigate distance as a factor and not discourage children’s attendance, reading clubs would need to have been significantly multiplied to provide full coverage. This challenge can partially be explained by examining the origins of the Mureke Dusome program. Reading clubs were the product of the previous ‘Literacy Boost’ project implemented in Gicumbi District in 2016. Under this project, reading clubs were established in each and every village. In order to achieve national scale up on available funds, Mureke Dusome recast the one club/village as one club/school, significantly affecting the number of children who could feasibly attend. Another challenge was a lack of infrastructure in place for reading clubs. While some clubs were fortunate to be hosted by faith-based organizations or sector/cell offices, many reportedly took place under a tree or in an empty classroom when one was available. However, this meant that the club had no permanent space to convene, nor a consistent place to store materials. A shared concern across many respondents was that where reading clubs were held outdoors, they were subject to cancellation due to rain or 45 extreme heat. The lack of a permanent meeting place, along with the unpredictability of weather, led to inconsistent programming for reading clubs. “The reading place is not an appropriate environment where they sit in shadow of trees while in this area we have few trees. The sunlight disturbs them and they are required to move with the shadow of trees and of the church building because they often gather at the Catholic Church compound. So if they could have a reading room with chairs to sit and read comfortably it should be very helpful. They have mats they sit on but they are not sufficient for them all, so many of them sit on the ground.”58 Without guaranteed infrastructure, some reading clubs also struggled with questions of book storage. While community libraries were viewed as an ideal option, with only 40 across the country compared to over 2,500 reading clubs, this was not a practical possibility for the vast majority. Sometimes Literacy Champions were able to store books at nearby schools, but other times they were obliged to carry the books home with them in order to ensure their security. Partly in consequence to issues of book storage and partly due to normal wear and tear, book maintenance and replenishment was also a major concern, particularly in the later years of Mureke Dusome and during its exit phase. Of the books that were previously purchased and distributed by the program, some have fallen into disrepair while others have gone missing. Some respondents also indicated that children have worked their way through all the titles available at their local reading club. Without a fresh supply of new books, children risk becoming disinterested and may stop attending altogether. This challenge has become increasingly apparent as the program evolved: “There was a good and enthusiastic forum that used to be conducted on Saturday and Sunday, during which were conducted reading sessions and dramas, but it was terminated. It is no longer operational. I personally have been very sad it no longer works. Children used to come and join the forum, but the issue has been insufficient books, such that stories that were read one day were repeated the following day, and this was a discouragement to children.”59 One father explained the issue: “Children are no longer at reading clubs in number as before because the books have not changed, there are no new books, and you find children have memorized all the stories in the books. Even the child who doesn’t come to the reading clubs knew the stories because siblings shared the stories. New books are needed.”60 For others, the number of books has always been limited compared to the larger number of children attending clubs: 58 KI, Head Teacher 59 KI, Head Teacher 60 FGD Parents, Burera District 46 “The books we have are limited. We started with 100 storybooks, and the schools also supported the reading clubs borrowing books for us from the school libraries, but we still need more. What we have is not sufficient because children who attend the reading club and who want books to read are many and that is the biggest challenge we face.”61 While Literacy Champions themselves were valued by children and communities, the initiative suffered some setbacks that prevented it from being more effective. Many of these challenges stemmed from the program’s reliance on a community volunteer model. Specifically, motivation and incentives for Literacy Champions were widely seen by respondents as lacking. Many noted that while Rwanda values homegrown solutions that often rely on volunteers, the program was not adequately designed nor sufficiently resourced to provide consistent support to Literacy Champions in their work. This lack of motivation reportedly led to high turnover and disengagement by some Literacy Champions, with knock on effects to reading clubs and other community-based literacy activities such as Umuganda Literacy. One Head Teacher explained that motivation could come in different forms, aside from financial incentives: “Literacy champions should have been given some incentives or something as kind of motivation as a sign of appreciation for their tremendous work. And the motivation is not always the money but can be like certificates or something else.”62 Unlike other community volunteering schemes that have been nationally institutionalized (and often have better funding), Literacy Champions are not yet fully recognized to the same degree. At the local level, however, comparisons between different community volunteer structures are inevitably drawn, leading to feelings of frustration: “Literacy Champions don’t receive any kind of incentive, like stipends, a bike, airtime, or health insurance. They don’t benefit from any kind of follow up. For example, Community Health Workers and Village Leaders are not salaried but they benefit from some kind of incentive to motivate them in their work. It should also be applied to Literacy Champions, too.”63 While some Literacy Champions were organized into cooperatives to help start income generating activities – and were appreciative of the effort – many found the 14,000 RWF start-up capital to be insufficient. Part of their dissatisfaction may have had to do with the natural tendency to compare the Literacy Champion program with other longer-standing national volunteering schemes that offered larger grants or loans available to beneficiaries. Other challenges that were mentioned less frequently included teachers’ lack of understanding or respect for the role of Literacy Champions as well as a lack of clarity about who has oversight and accountability for Literacy Champions. These cumulative challenges sometimes meant that individual Literacy Champions worked inconsistently or stopped altogether. Without incentivization, Literacy Champions may not have been inclined to 61 KI, Literacy Champion Representative 62 KI, Head Teacher 63 KI, Literacy Champion Representative 47 prioritize Mureke Dusome activities over paid employment or personal responsibilities.64 One Sector Education Inspector explained: “To me, the lack of motivation of the Literacy Champions was the major issue. Literacy Champions may have other projects they volunteer for. And when they see that those other organizations call them often, invite them to seminars and give them incentives and Mureke Dusome does not… then that is why you see reading clubs are no longer active like at the start.”65 This underscores the fact that Mureke Dusome had big ambitions, but a light footprint. Limited resources meant that the model and depth of intervention that was initially piloted in the first two districts was different than what was eventually rolled out nationally. Thus, it is important to note that a discussion about effectiveness must be predicated on an understanding that the program did not achieve full population saturation, despite its national ambitions. Mureke Dusome also faced additional challenges over the course of the program that impacted its ability to be effective. For example, when some of Save the Children’s other programs were closed and not replaced, it meant that Mureke Dusome lost some of its own core capacity because programs housed under the same organization often share resources, both in terms of materials and infrastructure as well as knowledge and expertise. Some felt that Mureke Dusome suffered from a misalignment between objectives and resources, which also limited its ability to translate its activities at scale. To be sure, the challenges facing Mureke Dusome were similar to many of those often faced by INGOs. For example, one informant explained that Save the Children did not fully cost out the real financial implications of its approach. The underestimated real cost paired with an over-commitment resulted in Save the Children operating, in what one respondent characterized as a “survival approach” to the program. Another challenge the program faced was the comparisons between Mureke Dusome and its better-funded counterpart, Soma Umenye. In addition to resources, Some Umenye generally drew more attention and visibility than Mureke Dusome. EQ2. WHICH KNOWLEDGE, ATTITUDES AND PRACTICES ARE CORRELATED WITH HIGHER [OR IMPROVED] STUDENT READING SKILLS? Our evaluation showed that positive changes in literacy-related knowledge, attitudes and practices at home and in the community did, for the most part, translate into better student performance in the classroom. QUANTITATIVE DATA SOURCE AND SAMPLE We analyzed the 2018 Early Grade Reading Assessment (EGRA) data that was collected by Soma Umenye between September and October 2018 from a nationally representative random sample of 4,635 primary school students in grades 1, 2 and 3 in public and government-aided schools. Of the 4,635 students, 1544 (33.3%) were in Primary 1, 1540 (33.2%) in Primary 2 and 1551 (33.5%) in Primary 3. The 2018 EGRA 64 While it didn’t come up in interviews, it should be noted that Literacy Champions were asked to contribute about 3 to 5 hours per week to the project, with flexible timing. 65 KI, Sector Official 48 assessed students’ reading skills using six EGRA sub-tasks, namely listening comprehension, letter sound identification, syllable sound identification, familiar word reading, oral reading fluency and reading comprehension. The reading passage differed for each grade. A detailed description of each EGRA sub￾task and its rationale for assessing reading skills can be reviewed in Dubeck and Gove (2015).66 In addition to assessing reading skills, the 2018 EGRA included a context questionnaire that collected data on the following children’s knowledge, attitudes and practices (KAP) about reading that included: • Taking Kinyarwanda books home from the classroom and using them to practice reading; • Having a place in the community where the child can go to read/borrow Kinyarwanda book; • Lending or borrowing a book or other learning materials to/from other students; • Participating in any reading activity after school; • Enjoying reading in a group with other children; • Having someone who reads a story to the child at home; • Reading to someone out loud at home; • Reading independently at home; and • Having a favorite book. The 2018 EGRA context survey also collected data on factors related to school attendance (i.e. absence/delay to school), home environment, and family socioeconomic status (i.e. having electricity, a phone, radio, bicycle, moto, car, access to food and drinks before going to school) through interviews with children who participated in EGRA. DEMOGRAPHIC AND READING KNOWLEDGE, ATTITUDES AND PRACTICES (KAP) CHARACTERISTICS OF STUDENTS STUDENT DEMOGRAPHICS As indicated in Table 1, the sample for the 2018 EGRA was gender-balanced, and the mean age ranged from 7.4 years old for P1 students to 10.5 years old for P3 students. Table 1: Demographic characteristics of students Student demographics P1 P2 P3 Gender Boys 51.3% 52.8% 51.1% Girls 48.7% 47.2% 48.9% Age (in years), mean (Standard error) 7.4 (0.08) 9.3 (0.09) 10.5 (0.07) READING KAP CHARACTERISTICS OF STUDENTS The results in Figure 1 indicate that nearly all children surveyed enjoy reading in a group with their peers, and the majority have various opportunities for practicing reading after school. However, fewer children reported to have a place in the community where they can go to read or borrow Kinyarwanda books and only around half take Kinyarwanda books home after school for reading practice. 66 Dubeck, M. M., & Gove, A. (2015). The early grade reading assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development, 40, 315-322. 49 Figure 1: Percent of students with the reading knowledge, attitudes and practices (KAP) characteristics The results in Table 2 indicate that boys and girls have similar characteristics on all reading KAP. There are two categories where there is a slight gender difference: P2 boys are more likely to have a favorite book (boys: 88.6% vs. girls: 78.1%) and P3 boys are more likely to participate in reading activities after school than P3 girls (boys: 90.9% vs. girls: 86.0%). The latter is particularly interesting because it contradicts the qualitative findings when interviewees observed that girls were more likely to participate in after school reading activities than boys. Table 2: Percent of students with the reading knowledge, attitudes and practices (KAP) characteristic by gender Reading KAP characteristic P1 Students P2 Students P3 Students Boys, N=784 Girls, N=760 Boys, N=773 Girls, N=767 Boys, N=777 Girls, N=774 Enjoy reading in a group with other children 97.8% 98.1% 98.5% 99.0% 99.7% 99.1% Participating in any reading activity after school 74.2% 82.1% 83.5% 85.5% 90.9% 86.0%** Having a favorite book 67.5% 71.7% 86.6% 78.1%** 87.5% 86.2% Reading independently at home 74.0% 80.5% 83.7% 85.1% 85.8% 84.7% Children who ever lend or borrow a book or other learning materials to/from other students 58.2% 54.1% 76.0% 75.8% 75.8% 76.8% 97.9% 78.1% 69.5% 77.1% 56.2% 62.0% 77.2% 53.1% 37.9% 98.7% 84.5% 82.6% 84.3% 75.9% 71.3% 74.1% 65.2% 49.7% 99.4% 88.5% 86.9% 85.3% 76.3% 75.4% 74.7% 56.7% 49.5% Enjoy reading in a group with other children Participating in any reading activity after school Having a favorite book Reading independently at home Children who ever lend or borrow a book or other learning materials to/from other students Reading to someone out loud at home Having someone who reads a story to the child at home Taking Kinyarwanda books home from the classroom and use them to practice reading Having a place in the community where the child can go to read/borrow Kinyarwanda book P1 P2 P3 50 Reading to someone out loud at home 58.9% 65.2% 71.1% 71.6% 74.8% 76.1% Having someone who reads a story to the child at home 75.4% 79.1% 70.3% 78.4% 76.4% 73.0% Taking Kinyarwanda books home from the classroom and use them to practice reading 51.2% 55.1% 65.0% 65.5% 56.5% 56.9% Having a place in the community where children can go to read/borrow Kinyarwanda books 35.3% 40.5% 51.2% 48.1% 47.7% 51.4% **Significantly different at α=0.05 significance level STUDENTS’ PERFORMANCE ON EGRA SUB-TASKS Table 3 shows the overall performance of students on EGRA sub-tasks. Table 3: Mean scores by EGRA sub-task EGRA subtask Grade Mean SE* [95% CI**] Letter Name Identification (clpm) P1 24.2 0.93 [22.3, 26.0] P2 38.3 1.84 [34.6, 41.9] Syllable Sound Identification (cspm) P1 14.1 0.81 [12.5, 15.7] P2 19.8 1.46 [16.9, 22.7] P3 21.6 1.31 [19.0, 24.2] Familiar Word Reading (cwpm) P1 5.7 0.39 [4.9, 6.5] P2 10.4 0.74 [9.0, 11.9] P3 14.8 1.09 [12.6, 16.9] Oral Reading Fluency (ORF - cwpm) P1 4.8 0.35 [4.1, 5.5] P2 10.6 0.82 [9.0, 12.2] P3 16.3 1.10 [14.1, 18.4] *Standard error; **Confidence interval The Rwanda Education Board, with support from Soma Umenye, has established grade-level benchmarks that define expectations for children’s oral reading fluency and reading comprehension levels. For oral reading fluency, the benchmark for P1 is 10 correct word per minute (cwpm); for P2 is 25 cwpm, and for P3 is 40 cwpm). For the reading comprehension EGRA sub-task, 1 the benchmark is a score of ≥60% of questions answered correctly for P1 and P2, and ≥80% correct for P3 students. As shown in Table 4, a small percentage of children met these benchmarks in 2018. 51 Table 4: Percentage of students meeting reading fluency benchmarks by grade level P1 P2 P3 Oral Reading Fluency (ORF) 20.0% 15.5% 3.9% Reading Comprehension 17.7% 36.2% 29.9% READING KNOWLEDGE, ATTITUDES AND PRACTICES (KAP) THAT ARE CORRELATED WITH HIGHER/IMPROVED STUDENT’S READING SKILLS Reading Passage tasks: KAP factors correlated with meeting the grade-level benchmark on the Reading Comprehension sub-task The results of this report indicate six KAP factors which were significantly correlated with meeting the grade-level benchmark for reading comprehension across grades: 1) participation in reading activities after school, 2) having a place in the community where children can go to read or borrow books, 3) taking Kinyarwanda books home from the classroom and using them for reading practice, 4) reading to someone out loud at home, and 5) having a favorite book and 6) reading independently at home (see Figure 2 and Table 5). Figure 2: Percent distribution of students achieving the grade-level benchmark for reading comprehension, by grade and knowledge, attitudes and practices about reading Note: p-value<0.001 for all comparisons (i) Participating in reading activities after school The participation in reading activities after school was an important factor predicting a score at or above the grade-level benchmark on the reading comprehension sub-task for P1 and P2 students. 3% 8% 11% 16% 17% 26% 9% 22% 23% 26% 41% 40% 44% 46% 33% 37% Participating in reading activities after school Taking books home for reading practice Participating in reading activities after school Having a favorite book Reading to someone out loud at home Having a place in the community where children can go to read/borrow books Reading independently at home Having a place in the community where children can go to read/borrow books P1 Students P2 Students P3 3tudents No Yes 52 ● There was a significantly greater percentage of P1 students meeting the reading comprehension benchmark in children participating in reading activities after school compared to their peers who reported not participating in any reading activity after school (22% vs. 3%). ● For P2 students, 41% of children participating in reading activities after school achieved the reading comprehension benchmark compared to 11% among children who reported not participating in any reading activity after school. ● The multivariable analysis results indicate that P1 and P2 students who reported participating in reading activities after school were 5.12 times and 2.40 times as likely to meet the grade-level benchmark on reading comprehension compared with their peers who do not attend, respectively. (ii) Having a place in the community where children can go to read or borrow books The availability of a place in the community where children could go to read or borrow books after school was an important predictor associated with achieving the grade-level benchmark on the reading comprehension sub-task for students in P2 and P3. ● Forty-six percent of the P2 students who reported having a place in the community where they could go to read or borrow books achieved the reading comprehension benchmark compared to 26% among their peers who do not. ● For P3 students, 37% of children with a place in the community where they could go to read or borrow books achieved the reading comprehension benchmark compared to 23% in their peers who reported not having this place. ● In the multivariable analysis, P3 and P2 students who reported to have a reading place or source of reading materials in their community were 1.62 times and 1.61 times as likely to achieve a score at or above the reading comprehension benchmark compared with their peers who reported not having this place in their communities, respectively. Table 5: Knowledge, attitudes and practices (KAP) about reading which are significantly associated with higher scores on Reading Comprehension sub-task, by grade Odds ratio [95% CI] p-value P1 Students: Student participates in reading activities after school 5.12* [2.22, 11.78] <0.001 Take Kinyarwanda books home from the classroom and use them to practice reading 2.34* [1.09, 5.05] 0.030 P2 Students: Student participates in reading activities after school 2.40* [1.35, 4.28] 0.003 Student has a favorite book 2.16* [1.31, 3.57] 0.003 Student reads to someone aloud at home 2.14* [1.36, 3.36] 0.001 Student reported a place in his/her community where children can go to read/borrow Kinyarwanda books 1.61* [1.09, 2.36] 0.016 P3 Students: Student reads independently at home 2.72* [1.42, 5.22] 0.003 53 A place in the community where children can go to read/borrow Kinyarwanda books 1.62* [1.11, 2.36] 0.012 Notes: - Odds ratios in each model (P1, P2 and P3) was adjusted for possible confounding factors about student’s gender and age, being late or absent to school, school location and household socioeconomic factors (see Table 1 in Annex VII). - CI, Confidence Interval - *Statistically significant, p<0.05 (iii) Taking Kinyarwanda books home from the classroom and using them to practice reading This practice was significantly associated with increased odds of achieving the reading comprehension benchmark for P1 students. ● Twenty-six percent of children achieved the reading comprehension benchmark in the P1 students who reported both taking Kinyarwanda books home from the classroom and using them for reading practice compared to only 8% in their peers who do not. ● From the multivariable logistic model, P1 students who reported taking books home from the classroom and using them for reading practice were 2.34 times as likely to meet the reading comprehension standard compared with their peers who do not. (iv) Reading to someone out loud at home This reading practice was significantly associated with increased odds of meeting the reading comprehension benchmark for P2 students. ● Forty-four percent of the P2 students who reported to practice reading to someone at home achieved the benchmark for reading comprehension compared to 17% among their peers who do not. ● In the multivariable analysis, P2 students reading to someone out loud at home were 2.14 times as likely to meet the reading comprehension standard than their peers who do not. (v) Having a favorite book P2 students who reported to have a favorite book had increased odds of scoring at or above the benchmark for the reading comprehension sub-task. ● Forty percent of the P2 students who reported to have a favorite book achieved the benchmark for reading comprehension compared to 16% among their peers who do not. ● In the multivariable analysis, P2 students who reported to have a favorite book were 2.16 times as likely to meet the reading comprehension standard than their peers who do not. (vi) Reading independently at home The time for reading independently at home was particularly associated with increased odds of meeting the reading comprehension benchmark for P3 students. ● There was a significantly greater percentage of P3 students meeting the grade-level benchmark among children who reported to take time and read on their own at home compared to their peers who do not (33% vs. 9%). ● In the multivariable analysis, P3 students who reported reading on their own at home were 2.72 times as likely to meet the reading comprehension standard than their peers who do not. 54 KAP factors correlated with meeting the grade-level benchmark on the Oral Reading Fluency (ORF) sub-task In the multivariable analysis, five KAP factors that were identified to be significantly associated with meeting the grade-level benchmark on the ORF across the grades, were: 1) reading independently at home, 2) participating in reading activities after school, 3) reading to someone out loud at home, 4) having a favorite book, and 5) having a place in the community where children can go to read or borrow books (see Figure 3 and Table 6). (i) Reading independently at home P1 and P2 children who reported taking some time to read on their own at home were more likely to achieve the grade-level benchmark for ORF compared with children who did not report reading independently at home. ● The proportion of P1 students who achieved the grade-level benchmark for ORF among children who reported reading independently at home (25%) was 5 times greater than that of their peers who do not (5%). ● For P2 students, 18% of the students who reported reading independently at home achieved the ORF benchmark compared to only 2% among their peers who do not. ● From the multivariable analysis, P2 and P1 students who reported to take time and read independently at home were 4.46 times and 2.81 times as likely to meet the ORF benchmark than their peers who do not, respectively. Figure 3: Percent distribution of students achieving the grade-level benchmark for oral reading fluency (ORF) by knowledge, attitudes and practices about reading 5% 5% 2% 5% 6% 2% 25% 25% 18% 18% 19% 6% Participating in reading activities after school Reading independently at home Reading independently at home Having a favorite book Reading to someone out loud at home Having a place in the community where children can go to read/borrow books P1 Students P2 Students P3 Students No Yes 55 Table 6: Knowledge, attitudes and practices (KAP) about reading which are significantly associated with higher scores on Oral Reading Fluency (ORF) sub-task, by grade Odds ratio [95% CI] p-value P1 Students: Student participates in reading activities after school 3.93* [1.94, 7.99] <0.001 Student reads independently at home 2.81* [1.33, 5.94] 0.007 P2 Students: Student reads independently at home 4.46* [1.47, 13.5] 0.008 Student has a favorite book 3.26* [1.45, 7.34] 0.005 Student reads to someone aloud at home 2.37* [1.35, 4.14] 0.003 P3 Students: A place in the community where children can go to read/borrow Kinyarwanda books 2.49* [1.36, 4.57] 0.003 Notes: - Odds ratios in each model (P1, P2 and P3) was adjusted for possible confounding factors about student’s gender and age, being late or absent to school, school location and household socioeconomic factors (see Table 2 in Annex VII). - CI, Confidence Interval - *Statistically significant, p<0.05 (ii) Participating in reading activities after school P1 students who reported participating in reading activities after school were more likely to achieve the ORF benchmark than their peers who do not attend. ● A quarter (25%) of the P1 students who reported participating in reading activities after school achieved the ORF benchmark, compared to only 5% among their peers who do not participate in any reading activity after school. ● In the multivariable analysis, P1 students who participated in reading activities after school were 3.93 times as likely to achieve the ORF benchmark compared with their peers who do not. (iii) Reading to someone out loud at home The P2 students who reported reading to someone out loud at home were more likely to achieve the ORF benchmark than their peers who do not. ● Nineteen percent of the P2 students who reported to have the opportunity to practice reading to someone out loud at home achieved the ORF benchmark compared to only 6% among their peers who do not. ● In the multivariable analysis, P2 students who reported reading to someone out loud at home were 2.37 times as likely to achieve the grade-level benchmark on ORF compared with children in the same grade who do not. (iv) Having a favorite book P2 students who reported having a favorite book were more likely to achieve the ORF benchmark than their peers who do not have. ● Eighteen percent of the P2 students who reported to have a favorite book achieved the ORF 56 benchmark, whereas this proportion was 5% among their peers who reported not having a favorite book. ● In the multivariable analysis, P2 students who reported having a favorite book were 3.26 times as likely to achieve the ORF benchmark compared to their peers who do not have. (v) Having a place in the community where children can go to read or borrow books P3 students who reported to have a place in the community for reading practice or borrowing books were more likely to achieve the ORF benchmark than their peers without this place in their community. ● Six percent of the P3 students who reported to have a place in the community where they could go to read or borrow books achieved the ORF benchmark compared to only 2% among their peers who reported not having this place in their communities. ● In the multivariable analysis, P3 students who reported to have a place in their communities where they could go to read or borrow books after school were 2.49 times as likely to meet the ORF benchmark compared with their peers who reported no to have this place. Lower Order Reading Skills: Knowledge, attitudes and practices (KAP) about reading, which are correlated with higher scores on Letter Name Identification sub-task From the multiple regression (MLR) analysis, four KAP factors that were identified to be significantly correlated with higher scores on letter name identification EGRA subtask for P1 and P2 students, were: 1) participation in reading activities after school, 2) reading to someone out loud at home, 3) reading independently at home, and 4) taking Kinyarwanda books home from the classroom for reading practice (see Figure 4 and Table 7). (i) Participating in reading activities after school The participation in reading activities after school was an important factor associated with higher scores on the letter name identification sub-task for both P1 and P2 students. • P1 students who reported participating in reading activities after school identified an average of 27 clpm compared to 14 clpm for their peers who do not. • For P2 students, the mean score on letter name identification for children participating in reading activities after school was 41 clpm compared to only 24 clpm for their peers who do not. • From the MLR, the mean score on the letter name identification was 8.11 clpm and 6.38 clpm higher for P2 and P1 students who reported participating in reading activities after school, respectively. 57 Figure 4: Mean score (number of correct letters per minute (clpm)) on letter name identification sub-task, by knowledge, attitudes and practices about reading (ii) Reading to someone out loud at home The practice of children reading to someone out loud at home was associated with higher scores on the letter name identification sub-task for both P1 and P2 students. • On average, P1 students who reported reading to someone out loud at home identified 28 clpm compared to 17 clpm for their peers who do not. • P2 students who read to someone out loud at home identified an average of 42 clpm compared to 29 clpm for their peers who do not. • In the MLR analysis, reading to someone out loud at home was associated with an average of more 6.36 clpm and 5.61 clpm for P2 and P1 students than their peers who reported not reading to someone aloud at home, respectively. 14 17 14 19 24 29 27 28 27 29 41 42 Participating in reading activities after school Reading to someone out loud at home Reading independently at home Taking books home for reading practice Participating in reading activities after school Reading to someone out loud at home P1 Students P2 Students No Yes 58 Table 7: Knowledge, attitudes and practices (KAP) about reading which are significantly associated with higher scores on Letter Name Identification sub-task, by grade Coefficient (SE) [95% CI] p-value P1 Students: Student participates in reading activities after school 6.38 (1.50)* [3.42, 9.33] <0.001 Student reads to someone aloud at home 5.61 (1.57)* [2.51, 8.71] <0.001 Student reads independently at home 4.55 (1.63)* [1.32, 7.78] 0.006 Student takes Kinyarwanda books home from the classroom and use them for reading practice 3.36 (1.57)* [0.26, 6.47] 0.034 P2 Students: Student participates in reading activities after school 8.11 (1.96)* [4.24, 11.97] 0.000 Student reads to someone aloud at home 6.36 (2.97)* [0.50, 12.22] 0.034 Notes: - Coefficients in each model (P1 and P2) were adjusted for possible confounding factors about student’s gender and age, being late or absent to school, school location and household socioeconomic factors (see Table 3 in Annex VII). - SE, Standard Error - CI, Confidence Interval - *Statistically significant, p<0.05 (iii) Reading independently at home P1 students who reported taking time to read on their own at home scored higher on the letter name identification sub-task than their peers who do not. • P1 students who reported reading independently at home identified an average of 27 clpm compared to nearly half (14 clpm) for their peers who do not. • In the MLR analysis, reading independently at home was associated with an increase in the mean score on the letter name identification sub-task by 4.55 clpm for P1 students. (iv) Taking Kinyarwanda books home from the classroom and using them to practice reading There was a significantly higher performance on the letter name identification sub-task among P1 students who reported taking Kinyarwanda books home from the classroom for reading practice than their peers who do not. • P1 students who reported taking Kinyarwanda books home and using them for reading practice identified an average of 29 clpm compared to 19 clpm for their peers who do not. • In the MLR analysis, taking Kinyarwanda books home for reading practice after school for P1 students, was associated with an increase in the mean score on the letter name identification sub￾task by 3.36 clpm. From the analysis of the relative importance of predictors of higher/improved reading skills, participation in reading activities after school and reading to someone out loud at home which were consistently associated with higher scores on the letter name identification sub-task for both P1 and P2 students, were also ranked among the top three important predictors and accounted for 31% and 30% of the model 59 variation, respectively (see Table 4 in Annex VII). KAP factors correlated with higher scores on Syllable Sound Identification sub-task From the MLR analysis, seven KAP factors significantly associated with higher scores on the syllable sound identification sub-task across the grades, were (see Figure 5 and Table 8): 1) Reading independently at home 2) Participating in reading activities after school 3) Reading to someone out loud at home 4) Having a favorite book 5) Taking Kinyarwanda books home for reading practice after school 6) Having a place in the community where children can go to read/borrow books 7) Enjoy reading in a group with other children Figure 5: Mean score (number of correct syllables per minute (cspm)) on Syllable Sound Identification sub-task, by knowledge, attitudes and practices about reading (i) Reading independently at home Reading independently at home was consistently associated with higher scores on the syllable sound identification sub-task for P1, P2 and P3 students. • The mean score on the syllable sound identification sub-task for P1 students who reported reading independently at home was slightly more than three times (17 cspm) higher than 5 cspm for their peers who do not. 5 8 5 9 7 11 12 9 16 10 9 13 17 18 17 19 22 22 23 22 23 24 22 23 Participating in reading activities after school Reading to someone out loud at home Reading independently at home Taking books home for reading practice Reading independently at home Having a favorite book Reading to someone out loud at home Participating in reading activities after school Having a place in the community where children can go to read/borrow books Reading independently at home Enjoys reading in a group with other children Having a favorite book P1 Students P2 Students P3 Students No Yes 60 • P2 students who reported reading independently at home scored an average of 22 cspm compared to only 7 cspm for their peers who do not. • P3 students who reported to read on their own at home scored on average more than twice (24 cspm) than their peers who do not. • In the MLR analysis, P3, P2 and P1 students who reported reading independently at home identified an average of more 7.33 cspm, 5.85 cspm and 4.45 cspm, respectively than their peers who do not. Table 8: Knowledge, attitudes and practices (KAP) about reading which are significantly associated with higher scores on Syllable Sound Identification sub-task, by grade Coefficient (SE) [95% CI] p-value P1 Students: Student participates in reading activities after school 5.48 (1.55)* [2.43, 8.54] 0.001 Student reads to someone aloud at home 5.11 (1.34)* [2.45, 7.77] <0.001 Student reads independently at home 4.45 (1.16)* [2.17, 6.74] <0.001 Student takes Kinyarwanda books home from the classroom and use them for reading practice 4.02 (1.90)* [0.28, 7.77] 0.036 P2 Students: Student reads independently at home 5.85 (1.84)* [2.21, 9.5] 0.002 Student has a favorite book 5.71 (2.03)* [1.7, 9.71] 0.006 Student reads to someone aloud at home 5.53 (2.23)* [1.13, 9.93] 0.014 Student participates in reading activities after school 4.13 (1.32)* [1.51, 6.74] 0.002 Student reported a place in his/her community where children can go to read/borrow Kinyarwanda books 2.88 (1.22)* [0.48, 5.29] 0.019 P3 Students: Student reads independently at home 7.33 (2.04)* [3.3, 11.35] 0.000 Student enjoys reading in a group with other children 7.00 (2.75)* [1.56, 12.44] 0.012 Student has a favorite book 3.64 (1.41)* [0.86, 6.42] 0.011 Notes: - Coefficients in each model (P1, P2 and P3) were adjusted for possible confounding factors about student’s gender and age, being late or absent to school, school location and household socioeconomic factors (see Table 5 in Annex VII). - SE, Standard Error - CI, Confidence Interval - *Statistically significant, p<0.05 (ii) Participating in reading activities after school There was a significantly higher performance on the syllable sound identification sub-task among P1 and P2 students who reported participating in reading activities after school. • The mean score for P1 students who reported participating in reading activities after school was more than three times (17 cspm) greater than 5 cspm for their peers who do not. 61 • For P2 students, the mean score was 22 cspm for children participating in reading activities after school compared to 9 cspm for their peers who do not. • In the MLR analysis, participation in reading activities after school was associated with an increase in the mean score on the syllable sound identification sub-task by 5.48 cspm and 4.13 cspm for P1 and P2 students, respectively. (iii) Reading to someone out loud at home The practice of reading to someone out loud at home was significantly associated with higher scores on the syllable sound identification sub-task for P1 and P2 students. • The mean score for P1 students who reported reading to someone out loud at home was more than twice (18 cspm) greater than 8 cspm for their peers who do not. • P2 students reading to someone out loud at home scored an average of 23 cspm compared to 12 cspm for their peers who do not. • In the MLR analysis, the practice of reading to someone out loud at home was associated with a significant increase in the mean score on the syllable sound identification sub-task by 5.53 cspm and 5.11 cspm for P2 and P3 students, respectively. (iv) Having a favorite book P2 and P3 students who reported having a favorite book significantly scored higher on the syllable sound identification sub-task than their respective peers who do not. • The mean score for P2 students who reported having a favorite book was twice higher (22 cspm) than 11 cspm for children in the same grade who reported no having a favorite book. • P3 students who reported having a favorite book scored an average of 23 cspm compared to only 13 cspm for their peers who do not have. • In the MLR analysis, having a favorite book was significantly associated with an increase in the mean score on the syllable sound identification sub-task by 5.71 cspm for P2 students and 3.64 cspm for P3 students. (v) Taking Kinyarwanda books home for reading practice after school P1 students who reported taking Kinyarwanda books home from the classroom and using those books for reading practice significantly scored higher on the syllable sound identification sub-task than their peers who do not. • The mean score for P1 students with this attitude and practice was slightly more than twice (19 cspm) higher than 9 cspm for their peers without. • In the MLR analysis, taking Kinyarwanda books home for reading practice was significantly associated with an increase in the mean score on the syllable sound identification by 4.02 cspm. (vi) Having a place in the community where children can go to read or borrow books P2 students who reported having a place in their community where children could go to read or borrow books significantly outperformed their peers on the syllable sound identification sub-task. • The mean score for P2 students with community-based place for reading or borrowing books after school was 23 cspm compared to 16 cspm for their peers without this place. • In the MLR analysis, having a place in the community for reading or borrowing books was 62 significantly associated with an increase in the mean score by 2.88 cspm for P2 students. (vii) Enjoy reading in a group with other children P3 students who reported to enjoy reading in a group with other children significantly scored higher on the syllable sound identification sub-task than their peers who do not. • The mean score for P3 children who enjoy reading in a group with other children was 22 cspm compared to 9 cspm for their peers who do not. • In the MLR analysis, enjoy reading in a group with other children was significantly associated with an increase in the average score by 7.00 cspm for P3 students. In addition, the top-ranked three important predictors of higher/improved scores on the syllable sound identification sub-task for each grade, were (see Table 6 in Annex VII): • For P1 students: reading to someone out loud at home, reading independently at home and taking Kinyarwanda books home from the classroom and use them for reading practice and accounted for 42% of the model variation. • For P2 students: reading to someone out loud at home, reading independently at home, and participating in reading activities after school, and accounted for 46% of the model variation. • For P3 students: reading to someone out loud at home, reading independently at home, and having a favorite book, and accounted for 49% of the model variation. KAP factors correlated with higher scores on Familiar Word Reading sub-task From the MLR analysis, seven KAP factors were identified to be significantly associated with higher scores on the familiar word reading sub-task across grades (see Figure 6 and Table 9): 1) Reading independently at home 2) Participating in reading activities after school 3) Reading to someone out loud at home 4) Having a favorite book 5) Taking Kinyarwanda books home for reading practice after school 6) Having a place in the community where children can go to read or borrow books 7) Enjoy reading in a group with other children • (i) Reading independently at home There was a significantly higher performance on the familiar word reading sub-task in favor of P1, P2 and P3 students who reported taking time to read on their own at home. • The mean score for P2 students who reported reading independently at home was four times (12 cfwpm) higher than 3 cfwpm for their peers who do not. • P3 students who reported reading on their own at home scored an average of 16 cfwpm compared to 6 cfwpm for their peers who do not. • P1 students who reported reading independently at home also read an average of 7 cfwpm compared to 2 cfwpm for their peers who do not. • In the MLR analysis, children who reported reading independently at home read an average of more 4.98 cfwpm for P3 students, 3.08 cfwpm for P2 students and 1.76 cfwpm for P1 students. 63 Figure 6: Mean score (number of correct familiar words read per minute (cfwpm)) on Familiar Word Reading sub-task, by knowledge, attitudes and practices about reading (ii) Participating in reading activities after school Participation in reading activities after school was significantly associated with higher performance on the familiar word reading sub-task for P1, P2 and P3 students. • P1 students participating in reading activities after school read an average of 7 cfwpm compared to only cfwpm for their peers who do not. • The mean score on the familiar word reading sub-task for P2 students who reported participating in reading activities after school, was three times (12 cfwpm) higher than 4 cfwpm for their peers who do not. • For P3 students, the mean score was 15 cfwpm for those who reported participating in reading activities after school compared to 9 cfwpm for their peers who do not. • In the MLR analysis, participation in reading activities after school was associated with an increased average of reading more 2.81 cfwpm for P1 students, 2.35 cfwpm for P2 students and 2.04 cfwpm for P3 students. 2 3 3 2 6 5 3 4 8 6 6 8 9 7 7 8 7 12 11 12 12 12 16 15 16 15 Participating in reading activities after school Reading to someone out loud at home Taking books home for reading practice Reading independently at home Reading to someone out loud at home Having a favorite book Reading independently at home Participating in reading activities after school Having a place in the community where children can go to read/borrow books Reading independently at home Enjoys reading in a group with other children Having a favorite book Participating in reading activities after school P1 Students P2 Students P3 Students No Yes 64 Table 9: Knowledge, attitudes and practices (KAP) about reading which are significantly associated with higher scores on Familiar Word Reading sub-task, by grade Coefficient (SE) [95% CI] p-value P1 Students: Student participates in reading activities after school 2.81 (0.77)* [1.28, 4.33] <0.001 Student reads to someone aloud at home 2.36 (0.57)* [1.23, 3.49] <0.001 Student takes Kinyarwanda books home from the classroom and use them for reading practice 2.10 (0.86)* [0.4, 3.80] 0.016 Student reads independently at home 1.76 (0.56)* [0.65, 2.88] 0.002 P2 Students: Student reads to someone aloud at home 3.57 (0.98)* [1.63, 5.51] <0.001 Student has a favorite book 3.16 (1.17)* [0.84, 5.48] 0.008 Student reads independently at home 3.08 (1.13)* [0.85, 5.32] 0.007 Student participates in reading activities after school 2.35 (0.73)* [0.91, 3.80] 0.002 Student reported a place in his/her community where children can go to read/borrow Kinyarwanda books 1.49 (0.70)* [0.11, 2.88] 0.035 P3 Students: Student reads independently at home 4.98 (1.43)* [2.16, 7.81] 0.001 Student enjoys reading in a group with other children 4.95 (1.8)* [1.39, 8.5] 0.007 Student has a favorite book 2.81 (1.06)* [0.72, 4.9] 0.009 Student participates in reading activities after school 2.04 (0.89)* [0.29, 3.79] 0.023 Notes: - The coefficients in each model (P1, P2 and P3) were adjusted for possible confounding factors about student’s gender and age, being late or absent to school, school location and household socioeconomic factors (see Table 7 in Annex VII). - SE, Standard Error - CI, Confidence Interval - *Statistically significant, p<0.05 (iii) Reading to someone out loud at home P1 and P2 students who reported reading to someone out loud at home scored higher on the familiar word reading sub-task compared to their respective peers who do not. • P2 students who reporting reading to someone aloud at home read an average of 12 cfwpm compared to 4 cfwpm for their peers who do not. • The mean score for P1 students reading to someone aloud at home was 7 cfwpm compared to 3 cfwpm for children in the same grade, but who do not practice reading to someone at home. • In the MLR analysis, reading to someone aloud at home was associated with reading an average of more 3.57 cfwpm for P2 students and 2.36 cfwpm for P1 students. 65 (iv) Having a favorite book P2 and P3 students who reported having a favorite book scored significantly higher on the familiar word reading sub-task than their respective peers who do not. • The mean score for P2 students who reported having a favorite book was 11 cfwpm compared to 5 cfwpm for their peers who don’t have a favorite book. • P3 students who reported having a favorite book read an average of 16 cfwpm compared to their peers who don’t have a favorite book. • In the MLR analysis, having a favorite book was associated with an increased mean score on the familiar word reading by 3.16 cfwpm for P2 students and 2.81 cfwpm for P3 students. (v) Taking Kinyarwanda books home from the classroom for reading practice after school P1 students who reported taking Kinyarwanda books home for reading practice after school significantly scored higher on the familiar word reading sub-task than their peers who do not. • The mean score for P1 students with this practice was 8 cfwpm compared to 3 cfwpm for their peers who do not. • In the MLR analysis, taking Kinyarwanda books home from the classroom for reading practice after school was associated with and increased mean score on the familiar word reading by 2.10 cfwpm for P1 students. (vi) Having a place in the community where children can go to read or borrow books P2 children who reported having a place in the community for reading or borrowing books after school significantly scored higher on the familiar word reading sub-task than their peers without this place in their neighborhoods. • The mean score was 12 cfwpm for P2 students with a place in the community for reading or borrowing books compared to 8 cfwpm for their peers without this place. • From the MLR analysis, having a place in the community for reading or borrowing books after school was associated with an increased mean score on the familiar word reading by 1.49 cfwpm for P2 students. (vii) Enjoy reading in a group with other children P3 students who reported to enjoy reading in a group with other children significantly scored higher on the familiar word reading sub-task than their peers who do not. • On average, P3 students who enjoy reading in a group with other children read 15 cfwpm compared to only 6 cfwpm for their peers who do not. • In the MLR analysis, enjoy reading in a group with other children was associated with an increase in the mean score on the familiar word reading by 4.95 cfwpm for P3 students. With the relative importance of predictors of higher/improved scores on the familiar word reading sub￾task, the three top-ranked KAP predictors, were (see Table 8 in Annex VII): • reading to someone out loud at home, reading independently at home, and taking Kinyarwanda 66 books home from the classroom and using them to practice reading accounted for 18.1%, 10.9% and 10.8% of the predicted variance in EGRA scores on familiar word reading for P1 students, respectively. • reading to someone out loud at home, reading independently at home, and participating in reading activities after school accounted for 21.1%, 16.0% and 7.9% of the predicted variance in EGRA scores on familiar word reading for P2 students, respectively. • reading to someone out loud at home, reading independently at home, and having a favorite book accounted for 22.9%, 12.0% and 10.0% of the predicted variance in EGRA scores on familiar word reading for P3 students, respectively. LINKING ANALYSIS TO PROGRAMMING In summary, our findings reveal that: • Participating in reading activities after school is particularly correlated with reading fluency (higher/improved scores on ORF and reading comprehension) for P1 students. Unlike children in higher grades who can improve their reading skills by reading a variety of books using community￾based libraries or setting up a time for reading independently, children in P1 may potentially benefit more from organized or supervised community-based reading activities. 67 • Reading to someone out loud at home is also correlated with higher/improved scores on reading fluency subskills for P2 students. • Having a place in the community where children can go to read or borrow books and time for reading independently at home were more beneficial to P3 students for reading fluency. This may suggest that older children (in P3) are able to use the community-based libraries and reading places because they can set a plan to read on their own at home. Interestingly, Save the Children’s own monitoring data shows that the attendance rate at reading clubs for P3 students is just 8% (48% for P1 students and 44% for P2 students). • Taking Kinyarwanda books home from the classroom and using them to practice reading is an important predictor of higher scores on all lower order reading subskills (letter name, syllable sound and familiar word reading) for P1 students. This may suggest that, unlike older children who could effectively use community-based libraries, children in P1 highly benefit from taking selected Kinyarwanda books home from the classroom and using them for reading practice to improve their reading skills. • Consistently, the three top-ranked predictors of higher/improved scores on low order reading skills (letter name identification, syllable sound identification and familiar word reading) for all grades, were: o reading to someone out loud at home, o reading independently at home, and o participating in reading activities after school. • Reading to children at home and exchanging books or other reading materials were the only KAP factors not correlated with improved reading skills. 67 Schickedanz, J. A. (1999). Much More than the ABCs: The Early Stages of Reading and Writing. NAEYC, 1509 16th Street, NW, Washington, DC 20036-1426. Ewing, R., Callow, J., & Rushton, K. (2016). Language and literacy development in early childhood. Cambridge University Press. 67 EQ3. WHAT ELEMENTS OF THE MUREKE DUSOME PROGRAM ARE LIKELY TO BE SUSTAINED? HOW COULD THE SUSTAINABILITY OF THE MUREKE DUSOME PROGRAM BE FURTHER PROMOTED? In this section, we turn to the issue of the sustainability of the Mureke Dusome programs. We consider what aspects of Mureke Dusome are likely to be sustained as well as what else could be done to further promote sustainability. Findings are organized into three sub-sections: sustainability at the national level, local level, and within the book sector. In each sub-section, we draw from key themes and program components that emerged from the qualitative fieldwork. As bedrocks to ensuring long-term sustainability of interventions, we examine issues concerning: ● ownership and demand; ● skills and capacity; and, ● financing. Summary: key findings around sustainability • The National Literacy Policy is the key to sustaining key elements and activities of Mureke Dusome. • Sustainability has been promoted by working within existing systems, delivering training based on REB standards, and working through schools for community outreach • Relying on volunteers will be an ongoing and inherent challenge. Potential lessons can be learned from the experience of other sectors in Rwanda that have also utilized volunteer models for service delivery. For Literacy Champions to be sustained and capacity strengthened, they should be incentivized. • There is demonstrable ownership and demand for Mureke Dusome interventions. Energy should be place on promoting literacy activities as a community-driven effort rather than an INGO initiative. Ownership could be improved by including literacy-focused indicators in imihigo. • Sustainability could be also enhanced by articulating the link between early literacy activities and other GOR priorities. This could include sector-focused activities like reducing drop out, as well as other national aims such as improving human capital. • With such limited staff at MINEDUC, the level of skills and capacity to sustain Mureke Dusome interventions is currently limited. Effective implementation of the National Literacy Policy will require MINEDUC investment in human resources. • The recurrent financial costs of Mureke Dusome interventions include a continued challenge to improve access to storybooks, particularly outside of Kigali. The key lesson learned from further examining the question of sustainability is that Mureke Dusome has helped to foster a strong demand for early grade reading opportunities at the national and sub-national levels, particularly in the local areas where implementation was most concentrated. The task now for Mureke Dusome, USAID and Rwanda’s early grade literacy movement more broadly, is to continue to build on the momentum that the program has begun. While Mureke Dusome has successfully laid the groundwork in many respects, long-term sustainability of interventions will require a further focus on embedding ownership and greater investments 68 in human and financial resources. SUSTAINABILITY AT THE NATIONAL LEVEL THE CULTURE OF READING HAS BECOME A NATIONAL FOCUS The Ministry of Education is prioritizing literacy, as evidenced by the country’s first National Literacy Policy which is currently in the final stages of validation. As a high-ranking government official put it, “Sustainability can only be ensured when the program is owned by the government.”68 Mureke Dusome made important contributions to the content of the National Literacy Policy, instrumentalizing its programming experience to shape future interventions. The policy offers the programming architecture and mandate for the government to improve literacy promotion, particularly for young children. Thus, when it comes to sustainability of some of the core ideas underlying Mureke Dusome, the National Literacy Policy holds a lot of promise. It offers the potential to help make a stronger case for many dimensions of the program such as reading clubs, Abana Writers Café, Literacy Champions, and working with community libraries. The National Literacy Policy also mentions the importance of better utilizing community-based mechanisms such as umuganda and umugoroba w’ababyeyi (parents’ evenings) to improve literacy attitudes, behaviors and practices. Many key informants at the national level expressed enthusiasm about the development of this policy, yet noted that sustained engagement and advocacy from MINEDUC, USAID, Mureke Dusome, and its partners would be required for implementation. The strategic plan has already been costed and will be presented to Cabinet for approval. MINECOFIN will then need to allocate sufficient funds to MINEDUC. Again, this requires consistent monitoring and advocacy. As it currently stands, the draft National Literacy Policy is emblematic of the widespread support for a culture of reading in Rwanda. Advocacy for the National Literacy Policy involved collaborating with key individuals from national institutions like MINEDUC. One key informant credited the national level success of Mureke Dusome as the “enthusiasm and personal investment” of some individual leaders who strongly advocated for the policy. However, they cautioned that sustainability could not always rely on the same set of enthusiastic individuals, because sometimes they end up shifting positions or getting replaced. For this reason, Mureke Dusome worked with these key individuals in order to bring about a national policy that could offer changes to literacy practices through the National Literacy Policy that could potentially be sustained, regardless of any personnel changes. Many respondents noted that the education space has become increasingly crowded by new organizations, posing difficulties for complementarity and avoiding duplication. Technical working groups and task forces have been set up throughout Rwanda’s education sector, and according to a key informant, such coordination meetings are a key avenue through which other actors can be made aware of Mureke Dusome’s approach, so that they can include literacy promotion as part of their planning programming or training. The office of the Director General at REB is also developing a system to guide programming by development partners according to REB priorities, and monitor their contributions. Coordinated service delivery was seen as an important strategy for the sustainability of Mureke Dusome activities. For example, thanks to strong coordination and communication by Save the Children as a participant in the School Leadership and Management Task Force (co-chaired by REB’s School Leadership Unit & VVOB) as well as bilateral meetings Save the Children has had between VVOB and REB, Mureke Dusome initiated content about how school leaders can support school-community partnership has been integrated into the school 68 KI, National. 69 leadership training modules delivered by an international NGO called VVOB with Mastercard funding. AN EVIDENCE BASE CAN STRENGTHEN CLAIMS TO EARLY LITERACY AS A POINT OF URGENCY Many actors at the national level felt that one area where increased demand for literacy activities could be generated – with subsequent improvements in long-term sustainability - was through bolstering arguments for its prioritization with a strong evidence base to national level actors. Several respondents noted that actors in the literacy space could produce more factual and advocacy-oriented information which links literacy to national economic growth and development. Tying early literacy to the achievement of key milestones for country’s future is a concrete way to ensure its prioritization by government. This strategy has already been successful in Rwanda, including in discussions around Early Childhood Development (ECD). Research on human capital and economic growth has helped strengthened the case for ECD, and the same logic could be applied to early literacy. One way to do this is for Mureke Dusome and others to produce more factual information and data on the impact of lack of literacy on national development in Rwanda. The Soma Rwanda platform could be used as a vehicle for documentation, dissemination and advocacy on this basis. SUSTAINABILITY AT THE SUB-NATIONAL LEVEL This section examines sustainability in district, sectors, schools and communities. As we saw in Evaluation Question #1, most respondents spoke highly of the program’s goals to improve early grade literacy and were enthusiastic about continuing the program. However, Mureke Dusome will come to a close. As the program is scaled down, an informant in the Ministry of Education suggested that this will present a “true indicator of sustainability”69 - specifically whether districts will continue Mureke Dusome activities now that the program is no longer active in these districts. WORKING WITH LOCAL GOVERNMENT Incentivizing literacy through performance contracts When it comes to engaging with local officials, there was near-uniformity on the approach to be adopted in order to sustain Mureke Dusome interventions – i.e. to ensure its visibility on imihigo performance contracts. Without formalizing activities through imihigo, there was a risk of inconsistent implementation and follow up. It is important to understand that local education officials (DDE, DEO and SEIs) are actually employees of MINALOC (even if they receive technical oversight from MINEDUC through REB). Their work is assessed and ranked in relation to their imihigo, and so this framework strongly guides their prioritization of tasks and time. Broadly speaking, there was general appreciation for Mureke Dusome in the sense that local education officials felt that community-based literacy activities did positively impact school performance as well as attendance and retention– indicators which already appear on their performance contracts. While these would contribute to positive assessments of their job performance, the fact that specific Mureke Dusome interventions do not explicitly feature in their imihigo limits the extent to which local officials will be incentivized to continually engage even after program closure. National level officials noted that districts had their own revenues and budgets. As one national key informant said, “We can only ask them to prioritize the education sector, but what we find is that each district has many variations in terms of the social problems which they face and may prioritize other needs, 69 KI, National 70 such as shelter and so on.”70 The official noted that the decision-making of mayors about planning and budgeting is driven by how they can accomplish their imihigo, because this is what they will be evaluated on. Their own budgeting decisions are also reviewed and approved by MINECOFIN to ensure alignment with national priorities and policies, as well as, district development plans. He explained that “while we cannot ask each district to allocate a fixed percentage on education, what we can do is to encourage them to prioritize education and to demonstrate its importance.”71 One education official in the Ministry of Education said that it is the job of district level officials to monitor Mureke Dusome activities. However, “they fail to do it,” the respondent said, “because they are overloaded. They do not see this as their first priority or their primary role. Their performance contracts and imihigo checklists are very long, and already their reports are not satisfactory, so they cannot be expected to take on much more effectively.”72 Given the multiple demands placed on local officials, development and implementing partners – in close collaboration with Government of Rwanda - must develop a strategy to present one basic education plan for each district. This plan could then clarify key objectives for the year, assign responsibilities with corresponding budgets, and so on. Performance indicators must realistically be achieveable, respondents cautioned, otherwise officials will not want to see them included. For example, reading clubs were identified by some as one way to monitor school dropout. If reading clubs could receive broader institutional and financial support to curb dropout, this may be one way to help infuse funding to sustain them with strong government buy-in. Other types of literacy indicators could potentially be added if they were achievable. Other forms of accountability While literacy-based goals or outcomes were not an explicit part of imihigo for either schools or local government, local officials and school leaders pointed to the fact that Mureke Dusome activities indirectly contributed to their broader efforts around school quality that they were sometimes evaluated on. The improved family engagement with schools supported by Mureke Dusome activities has positive effects beyond improved literacy skills. I think literacy is [implicitly] included in educational performance contract commitments of the local government authorities and parents because they commit to send their children to school and the reading clubs. The school also commits to improve the quality of education for their reading clubs. To improve the quality of education we have meetings and we commit to seek school material which include the above mentioned we provide children at reading events.73 Mureke Dusome-supported reading clubs played a role in reducing school dropout, which was a central component of the performance contract for local government officials. The local stakeholders we interviewed suggested that Mureke Dusome helps to curb school dropout by improving overall interest and engagement with learning, engaging with them in productive activities of reading clubs rather than wandering, and even enticing children who had already dropped out to re-engage in schooling. 70 KI, National 71 KI, National 72 KI, National 73 KI, Head Teacher 71 “Another success of Mureke Dusome is that the program contributed in reducing the dropout rate that was high in the sector. The fact that children were helped to raise their reading proficiency, this also contributed in their success in other courses, and therefore they are no longer missing classes.”74 Some suggested that the reading clubs could act as an incentive to draw children who had dropped out already back to the education system. Mureke Dusome in last three years helped in building children literacy capacity of school drop outs through the reading clubs that helped them to learn writing and reading differently than how they were learning at school. They saw they were taken care of with Literacy Champions volunteers who encouraged them to like studying to the extent they agreed to go back to school.75 Local government does not have strong demand to improve learning, in part because demand from MINALOC to promote literacy is not strong yet Sustainability of Mureke Dusome activities requires buy-in from MINALOC given their hierarchical links to local officials, from district mayors to sector education inspectors. But a recurrent theme among interviewees was that MINALOC staff members often had limited interface with Mureke Dusome. Technically, monitoring the activities of Mureke Dusome was the mandate of district education officials, but because of competing demands for their time, they were often minimally invested. Others pointed out that the Vice Mayor for Social Affairs was often unaware of Mureke Dusome’s activities in their district. The lack of interface with local officials underscored the limited collaboration that occurred between MINALOC and MINEDUC. Within Rwanda’s decentralized model, most line ministries have sought greater linkages to local government, so this issue is not limited to the education sector. Enlisting a stronger joint collaboration between the two ministries was seen by some to be essential to ensuring smooth handover for the continued implementation of literacy activities at the sub-national levels. “In terms of sustainability, one concrete way to make this happen would be to establish an MOU with MINALOC, whereby Mureke Dusome does direct implementation of literacy activities for a certain period of time before handing over to government through MINALOC.”76 Sustaining literacy practices in schools and communities Schools and SGACs were characterized by some respondents as holding the potential to be a standing institution for community literacy activities. Schools were considered by some as a hub through which trainings could be held for groups like Literacy Champions and SGACs. In addition, they could help to sponsor literacy activities such as umuganda, reading competitions, and other literacy activities. One of the key features to sustainability were the modules that Mureke Dusome and REB worked together to develop. These modules have been officially approved by REB, and they elaborate particular ways that 74 KI, Sector Official 75 KI, Village Leader 76 KI, National 72 schools and communities can partner in meaningful ways. These modules present another avenue for sustainability of certain elements of Mureke Dusome. Five of the six modules are designed to be self-study, which means they have the potential to be low cost and scalable while also embedding key ideas around how to promote effective parent and community engagement. Other respondents noted that the initial training provided by Mureke Dusome could have cascading effects for the relationship between schools and communities, because of the way it transformed their working relationship. Prior to Mureke Dusome, many Head Teachers and SGACs were unsure or unfamiliar with their roles and responsibilities. SGACs would typically focus on financial oversight and be less attuned to issues that impacted education quality and reading. However, through Mureke Dusome trainings some respondents said that the interaction between these groups has been transformed and that this, paired with the modules, can lead to sustainability. New school leaders and SGAC representatives will enter into a culture of collaboration and expectation for school-community partnerships. “Mureke Dusome interventions have opened a door for us and we have entered. It means that Mureke Dusome helped us to start and to learn from their best practices. Let me say that all Mureke Dusome reading activities will continue as we have Literacy Champions and parents committee at school which are very involved and has been trained by Mureke Dusome, so those who have been trained will also train others parents.”77 The sustainability of the Literacy Champion model was an area that produced strong and sometimes divergent views. To be sure, relying on volunteers is challenging in any context. Key informants noted that relying on volunteers is sustainable in the sense that it is low cost. Employing a volunteer-based model is a popular approach in Rwanda across sectors through what is called ‘home grown solutions.’ There have been several examples of how volunteers have managed to reach national scale using little resources. But on the other hand there was a large amount of concern about the prevalence of dropout and discontinuation of Literacy Champions, given lack of incentives or recognition associated with the uncompensated position. Respondents offered a number of ideas about how to ensure the Literacy Champion model could potentially be sustained. The most common solution to sustainability was to re-evaluate the Literacy Champion model. In particular, many respondents said that Literacy Champions could shift to an approach to volunteer work similar to that of the Community Health Workers, whose well-respected role is now embedded in communities nationwide. Volunteers in the education space could potentially follow up on dropout and promote literacy in ways that could align with local imihigo. Along with this, many respondents suggested that additional support could be provided to these volunteers. Examples of these provisions included subsidized health insurance, airtime, and/or a transport allowance. Many Literacy Champions also mentioned that income generating activities had been initiated for Literacy Champion volunteers. Most expressed appreciation for this effort and said they would like to benefit from similar types of support. As part of a pilot initiative to include income generating activities in the current Mureke Dusome cost extension, one Literacy Champion remarked that: “As Literacy Champions, we have an association that is about to be a cooperative. We gather in this association to share ideas and experience and to join hands together for we may not be discouraged later after Mureke Dusome. In our Literacy Champions' association we meet on the last week of 77 KI, District Official 73 each month and we save and offer credit among us and money is repaid back with a 5% interest. Mureke Dusome offered a support of 200,000 Rwandan francs to raise our savings. We are 18 Literacy Champions from nine schools in our sector.”78 Several respondents pointed out that one of the most important characteristics of Literacy Champions was that they were selected by communities themselves. That they came from and were selected by communities was seen as a way of improving retention. At the same time their demographic characteristics were sometimes significantly different. Some worried that younger Literacy Champions were more likely to leave the program early in favor of more lucrative opportunities, thus disrupting reading club continuity and functionality. “When Mureke Dusome will not be in our district, I am quite sure that reading activities will be sustained by relying on Literacy Champions who have been selected from the community and by other parents. They have experience on reading with the children and also got trained by Mureke Dusome. So relying on them will be very impactful in the community. In addition, Literacy Champions are trusted by parents. They don’t have any concern or worry to send their children to reading clubs. We have local structure called “Urugerero” [i.e., a national program for youth in Rwanda; Mureke Dusome is piloting how to leverage this program to support literacy goals]. Those young people who have completed secondary school before starting their university studies, they have to complete voluntary work in the community. We will use them to lead reading clubs and to sustain reading activities in the community with the support from Literacy Champions who will train them and lead them based on their experience.”79 Certain Head Teachers valued the contribution of Literacy Champions so much that they worked to find ways to motivate them through school budgets: “They will not possibly be motivated as they used to be during Mureke Dusome intervention, but we are trying to see how we can include Literacy Champions allowance in schools’ budgets and mobilize parents to contribute financially for a small amount of motivation - not like salary but as motivation for Literacy Champions as they have been very helpful in this activity. These Literacy Champions also used to mobilize the community with other national education initiatives; they could play the same role as Community Health Workers. More encouragement and motivation to Literacy Champions could be [like] doing teachers ceremonies where the best teachers are awarded. During this ceremony the best Literacy Champions can be also rewarded. This will definitely encourage them.”80 Discussions about the sustainability of Literacy Champions often occurred alongside that of the sustainability of reading clubs. A reading club was said to only be as effective as its Literacy Champion was committed. Yet there were other sustainability considerations concerning reading clubs, too. One of the most common concerns is that there were not enough reading clubs. Evaluation participants, 78 FGD Parents, Ruhango District 79 KI, District Official 80 KI, District Official 74 particularly parents, said they wanted reading clubs across all villages, not just the villages where schools were located. Some also suggested that reading clubs should even go to the sub-village (Isibo) level. While scaling to this level opens itself up to its own set of challenges, their rationale for sustainability was that the proximity of the club could introduce greater levels of ownership and accountability, particularly if the presence of reading clubs became a feature of local level imihigo. Under Evaluation Question #1, one of the concerns raised by some parents was that that reading clubs lacked materials and that those they had were falling into disrepair. At the same time, there were also instances of innovation to provide materials. For example, some community members said they pitched in to collect materials for reading club activities or built benches for children to sit on during the club meetings. This type of community-led contribution holds promise for sustainability. It could also extend to some of the concerns respondents had (including children) about the lack of infrastructure for reading clubs. Some of respondents’ ideas about spaces where reading clubs could be held included office space at local government, empty classrooms, community libraries, or churches. One father explained: “We have predicted that time will come and the program will end. As parents we took the lead where some of us started making mats to sit on when they are reading books in the club. Other parents also make for the children small bags they use to carry the books to prevent them to be damaged. And young men also help in repairing books that were damaged. We were also planning to create a fund that will be helping to support the reading club when the program ends. We will make contributions among parents, even if it would be a hundred Rwandan francs per person, for a certain period. When the program ends, we will keep supporting and facilitating the children because they are ours.”81 Reading clubs could also be sustained through continued parent buy-in. Some parents noted that after seeing the effects of the program, they will work to find ways to continue the reading club. One mother said: “The end of Mureke Dusome will not be the end literacy intervention it started, because it will not be the end of reading for the children. We mobilized parents to support their children’s literacy through encouraging them to go to the reading club and they collaborated. I hope they will not be discouraged. They will keep doing what is possible for the club activities to sustain.”82 This perspective aligns with community-based approaches to improve sustainability. It focuses on increased opportunity and time for children to spend practicing reading outside of school. Reading clubs appear to be an effective mechanism through which to do so because they align with community culture. Parents viewed the program positively because they can see children more productively engaged. Reading clubs also provide an alternative way for children to access to books through book banks, which are much more widely available than community libraries. In this way, reading clubs appear to be a feasible and sustainable option for increasing opportunities for children to improve their literacy skills outside of the classroom. 81 FGD, Parents, Burera District 82 FGD, Parents, Kirehe District 75 Other ideas included encouraging children to attend the clubs, collecting prizes to offer at reading competitions, or collecting donations during umuganda to replenish or repair books. Another father stated: “We have already started to prepare the end of the program even if we did not know that the program intervention is about to end. At Umuganda day in our community we contribute 100 RwF per each person, then that money will be used to repair books when they will be damaged or even buying other books. In such case Mureke Dusome before leaving us should increase books we have. They should also help us to find good location in which the club should gather.”83 Mureke Dusome has created a demand within households for literacy skills for children Mureke Dusome enlisted the support of communities and parents to improve children’s literacy-related knowledge, attitudes, and practices. Participants felt that this, in turn, created demand, because it helped convince parents and children that the program was relevant to their lives in the first place. Local respondents, particularly parents, said that they appreciated Mureke Dusome because it produced dividends that that were discernable and that they could appreciate. In other words, Mureke Dusome helped children gain the skills that parents valued, both in the classroom and in life. “Mureke Dusome have been important for children from the cities. There are new words they couldn’t imagine to know, but at least with Mureke Dusome they can see images and know these new vocabularies. Some words are like “umuvure,” (a traditional utensil used to make banana juice), “isekuru” (a traditional utensil used to grind peanuts or cassava), goats, etc.”84 Children and parents also appreciated Mureke Dusome because it presented an opportunity where many parents could help children build upon the literacy skills that they were learning through school and the reading clubs. Reading clubs reinforced what they learned in school and vice versa. In focus group discussions with children, we asked them who, if anyone, helps them read. Participants across all focus groups said that their parents were helpful in this regard. “My mother helps me to read and sometimes comes in Mureke Dusome to see how we are doing.”85 “When I leave reading club I tell my parents where I had difficulties while I was reading and they help me.”86 “My parents tell me ‘take a book and go in a living room and read.”87 83 FGD, Parents, Burera District 84 FGD, Parents, Ngororero District 85 FGD, Children, Gasabo District 86 FGD, Children, Kirehe District 87 FGD, Children, Ngororero District 76 Illiteracy among parents was identified as a challenge by some parents in focus group discussions. But even in these cases, parents still explained how they can help their children. “Not all parents know to read, but even those who do not know to read, they request children to read which is also to support them to learn.”88 As a result of Mureke Dusome parents saw their role in interventions that they may not have otherwise been concerned with. In the reading clubs, for example, parents who may have been otherwise reluctant to send their children to the clubs now gave them permission. In other cases, parents provided what they could in the forms of direct material support, helping to provide mats and benches for their children to sit on while they read. “Parents’ behavior changed. They understood the reading books and contribution of reading clubs because they agree to release their children allowing them to go to the reading club.”89 BOOK INDUSTRY SUSTAINABILITY Thanks to Mureke Dusome, the children’s book industry “now has teeth.”90 As was clear from findings in relation to EQ#1, the book sector has increased its capacity to supply Kinyarwanda storybooks for children. Thanks to Mureke Dusome’s other interventions at both community and national levels, there is also increased demand for these materials. But whether it can be sustained is unclear. Book industry ‘wins’: According to key informants at the national level, Mureke Dusome has already done a good job of handing over many of the structures and innovations upon which sustainability would be predicated. It did so in the following ways: ● Mureke Dusome introduced new ways of making books in Rwanda, connecting aspiring writers and illustrators to work with established publishing houses to produce locally relevant storylines and images. Abana Writers Café has been credited with providing a concrete platform through which viable, vibrant and innovative ideas for content can continue to be produced in the future. ● Private sector actors have begun to think through ways in which books can be used as a jumping off point for more creative interactions with storylines, using games and other play-based methodologies to engage with children. ● Publishers have been inculcated with an intrinsic responsibility to contribute to a wider reading culture. Thanks to the work of Mureke Dusome, this passion goes above and beyond their commercial motivations to increase book sales. ● The Rwanda Children’s Book Organization (RCBO) has spearheaded the drafting of a National Book Policy, with support from Mureke Dusome. 88 FGD Parents, Kirehe District 89 KI, Head Teacher 90 KI, National 77 Challenges to sustainability for the book industry: Key informant interviews also revealed a number of challenges that will need to be overcome in order for the book industry to continue to be viable, including: ● Mureke Dusome’s unique contribution to the book industry in terms of offering training and expertise has left the industry with a strong desire to continue learning and growing. While learning needs have been identified, there are no resources to fulfil in the same manner as before. These include topics such as issues of copyright, market research and marketing, sales and distribution, as well as, digital media, amongst others. ● While the technical side has been supported by Mureke Dusome already, the business side needs more support and attention. Publishers need to be able to operate as stable business entities. They need to establish a distribution network that is less charity-oriented and more commercially driven. Some respondents doubted about the industry’s ability to reduce costs and make books financially accessible, although there are some efforts underway. Some publishing houses are working with selected supermarkets in urban and peri-urban areas across the country to sell books on a consignment basis, using their existing infrastructure to support sale points outside of Kigali. Others are also experimenting with the use of ‘briefcase’ street sellers to resolve the issue of lack of hard infrastructure for distribution. ● The RCBO has developed a strategic plan with Mureke Dusome’s support, but there is little accompaniment for follow up of the related action plan now that the program is drawing to a close. ● The Abana Writers Café has been handed over to Sankofa Creatives, but no earmarked budget has been allocated thus far for its continuation. ● During the program, publishers were in a position to offer a more advantageous end-user price since they were sure to sell a certain number of copies of any given title to Mureke Dusome (thus reducing manufacturing costs). This effect was even more pronounced by the fact that Mureke Dusome was the second bigger purchaser of books in Rwanda at the time (after REB). This is now no longer guaranteed. Publishing houses will need to explore possibilities to reduce their pricing, possibly by bringing printing in-house to control production costs. ● Mureke Dusome was in the unique position of stimulating both demand and supply of books. At times, it was the buyer of the same books that the program encouraged to be published in the first place. Some informants worried that publishing houses are producing books to the tastes and price points of international NGOs rather than local populations. While some Kigali-based residents can potentially afford these prices, they are inaccessible for the vast majority of households in Rwanda. Some respondents felt that publishing houses need to start orienting their book production towards rural communities, not just NGOs. ● Schools remain a reliable market for Rwandan publishing houses. While REB has decided to print textbooks in-house, there is still the possibility that Head Teachers could choose to use their capitation grants to purchase books for school libraries should they see the value in it. ● The potential for using digital media and IT-related platforms has yet to be fully explored, although certain publishing houses are exploring the possibility of online sales (with home delivery). Others are looking into audio books and game applications, given the expanding internet coverage across the country. 78 CONCLUSIONS 1. There is now a strong appetite for early grade literacy in communities. Stakeholders at multiple levels have begun to understand, appreciate and value literacy. 2. Positive changes in literacy-related knowledges, attitudes and practices at home and in the community did, for the most part, translate into better student performance in the classroom. 3. The work of Literacy Champions was valued by schools and communities, but the mechanisms through which they work needs to be further institutionalized if it is to be sustained. 4. Without clear incentives and accountability measures, local government engagement on early literacy issues will likely be less consistent or in-depth. 5. The Rwandan book sector has the capacity to produce higher quality, age-appropriate Kinyarwanda titles for P1 to P3 children. However, gaps remain in terms of accessibility of books outside of urban centers in Rwanda that, if left unchecked, could reinforce existing inequalities. 6. The drawdown of Mureke Dusome’s continue support to Rwanda’s nascent book industry is a point of inflection, requiring more (and likely new) innovative ways to sustain and to grow. 7. The National Literacy Policy offers the potential policy architecture to increase lifelong literacy skills starting from an early age, thereby contributing to a wider culture of reading. 8. The global evidence base and rationale for early grade literacy is strong, and more can be done to apply this evidence to advocate to key policy and funding decision-makers to sustain and scale early literacy activities in Rwanda. 9. The arrival of new actors within the literacy space, which is indicative of a higher profile and prioritization of literacy promotion, requires improved communication and coherence of approach – as well as greater ministerial leadership – to avoid duplication of efforts while maximizing the positive impacts for children and their communities. 79 RECOMMENDATIONS Recommendations for ways forward stem directly from the findings of this evaluation. They are informed by the many respondents who offered their insights and expertise, as well as population-based findings from the EGRA data. Because many of the findings cut across multiple issue areas (i.e., across evaluation questions and sub-questions), recommendations have been organized thematically to inform decision￾making on approaches to adopt in future programming. FOSTER GREATER CROSS-MINISTERIAL COMMUNICATION AND COORDINATION • USAID and its partners should engage in advocacy to MINEDUC to increase recognition of the importance of community engagement to support children’s learning. This should include building human resource capacity within MINEDUC and REB as necessary, for the implementation of community-related aspects of the National Literacy Policy. In addition, activities to monitor early literacy should be made explicit priorities on the job descriptions of education-focused MINALOC employees such as District Directors of Education and Sector Education Inspectors. • Consider further research to better understand the interface between MINALOC, MINEDUC and other line ministries. Specific strategies should be developed to engage with MINALOC at centralized and decentralized levels to ensure effective implementation of the National Literacy Policy and to fully embed literacy activities such as Umuganda Literacy within existing local structures at national scale. • Given the multiple demands placed on local officials, development and implementing partners must work with government to develop a strategy to present one basic education plan for each district. This plan could then clarify key objectives for the year, assign responsibilities with corresponding budgets, and so on. • Adding early literacy to imihigo performance contracts – managed through MINALOC - will be an important step to ensure local ownership and sustainability. Literacy-related indicators should be comparable, measurable, and realistic (i.e., achievable). Future engagement with MINALOC can be potentially achieved through a dual approach. Firstly, MINEDUC should take a lead in direct advocacy to MINALOC at central level. Secondly, local officials and actors can share success stories from the field for more evidence-based advocacy. Umuhuza, for example, has been successful in gaining recognition for Urugerero youth volunteers under the National Itorero Commission by providing evidence on field-based successes from the pilot phase. • Development partners should channel further resources towards supporting community libraries, including advocacy to MINEDUC, MINISPOC and MINALOC for budget prioritization. The number of community libraries across the country must grow and their management be strengthened through adequate resourcing and greater visibility. Advocacy should emphasize that community library success is a MINEDUC mandate that requires broad community engagement with specific dividends for children’s literacy. This can build on success stories from the field, including locations in which sector officials have been supported by committees to manage community libraries in order to overcome challenges such as payment of librarians. EXPLORE DIFFERENT OPTIONS TO FURTHER STRENGTHEN THE WORK OF LITERACY CHAMPIONS • Continue to work with MINEDUC and partners to advocate for the approval and implementation of the National Literacy Policy. 80 • Examine ways for the Literacy Champion volunteer model to emulate the existing successful model of Community Health Workers program in Rwanda. Enhancements should focus on improving satisfaction, motivation and retention possibly through: • offering a package of incentives, such as formal certification through the National Itorero Commission, financial remuneration and/or income generating activities, or other forms of practical support (health insurance, airtime etc.); and, • providing non-material forms of recognition that would be meaningful for them, such as public appreciation during various community meetings. • Improve access to, and viability of, income generating activities for Literacy Champions as means to gather them together, keep them motivated and as a platform to discuss and problem-solve literacy-related challenges they are facing in their work. • Revisit the profiling of an ideal Literacy Champion, ensuring that the position can cater to their personal situation from the outset. Recognize that the set of incentives and motivations may look different for a young person compared with a retired teacher, for example. • Reconsider whether any existing positions at school or community level could adopt additional responsibilities in terms of literacy promotion, thereby investing in state-mandated structures such as school-based mentors or classroom teachers, for example, rather than creating parallel systems. • Determine how best to infuse literacy activities into existing community structures at the village level, e.g, umuganda, inteko y’abaturage, and itorero while establishing clear channels for accountability and reporting. • Strengthen parental education to provide more opportunities for children to practice reading at home. Our findings have highlighted reading out loud to someone and reading independently at home is an important predictor of improved reading skills and later classroom performance and should not therefore be overlooked. • Strengthen community-based reading activities led by Literacy Champions, encouraging all children to participate. Findings show that participating in reading activities after school is correlated with reading fluency. Nearly all children reported enjoying reading in a group setting with other children such as those provided in reading clubs, indicating that the continuation of reading clubs should be prioritized in future programming approaches. • Reinforce the key principles of SGACs when it comes to community literacy. Work to ensure that everyone knows that Literacy Champions are the responsibility of both schools and parents. While REB has already developed and distributed guidance on the roles and responsibilities of Literacy Champions, these appear to not yet be fully adopted. Further work needs to be done to better articulate and promote accountability of Literacy Champions at decentralized levels. This will also support implementation of the National Literacy Policy. HARMONIZE SERVICES TO AVOID DUPLICATION • Consider possible entry points to better harmonize service delivery to fully maximize the positive impact of programming. One example is of reading clubs, which could service not only P1 to P3 students, but also act as a gateway to other target groups and issues areas. These include providing Early Childhood Development (ECD) and Early Childhood Education (ECE) services to younger siblings who are – in any case – often present at the reading clubs. It could also be surveillance point for early identification of children with developmental delays, learning difficulties, or who 81 are at risk of dropping out. • Similarly, the link between reading clubs and community libraries could be made stronger. While the former were introduced uniquely for P1 to P3 students, the latter is a Government of Rwanda supported initiative that serves the entire community. Ideally, functional community libraries could serve as sites for reading clubs while new libraries are subsequently introduced according to current strategic planning. • District level coordination should be improved. District and other local officials are under a great deal of pressure. The idea underlying better coordination is to improve efficiency, impact and performance. Examples of improved coordination could include establishing one plan for early literacy that encompasses all actors and clearly feeds into broader plans for education; sharing calendars to better coordinate meetings and trainings to avoid duplication; establishing a district level equivalent to the national level School Leadership Unit (SLU) to help harmonize the work of development partners in line with key priorities; and, working with MINALOC to improve mechanisms to ensure that all early literacy activities are reflected in district action plans and captured in imihigo performance contracts. • Monitoring early literacy activities is expensive. Activities may be planned but implementation can be inconsistent due to lack of capacity, transport, and so on. This was reportedly a concern particularly for both central and local government officials. Human and financial resourcing for monitoring and evaluation of education activities at community and school levels needs to be reconsidered. The possibility of conducting joint monitoring through field-based staff funded by other related USAID and non-USAID-funded programs could also be explored. • Consider the potential advantages of linking adult literacy programs with early literacy initiatives that could then be placed under the performance contracts of local education officers. • Examine ways to further scale up activities by working with faith-based organizations and churches. Many of these organizations have longstanding relationships with community members. Furthermore, religious organizations often have a literacy-based component through reading of sacred texts. USAID and partners could explore how scale up existing work in this area by harmonizing the work of Literacy Champions and reading clubs with churches, Sunday schools and religious youth organizations. Sports and culture, particularly at school level, is another avenue to potentially leverage in order to further spread literacy messages. IMPROVE ACCESS TO BOOKS AND OPPORTUNITIES FOR READING FOR ALL CHILDREN • Given their correlation with improved reading skills, the following practices should be included as part of future interventions for lower primary children: • Increased participation in community-based reading activities after school, especially for P1 students. • Improved accessibility to reading materials outside of school by increasing community￾based places where children can go to read or borrow books, especially for older children in P3. • Advocacy to school leadership and encouraging children to take Kinyarwanda books home from the classroom and to use them for reading practice, especially for P1 students. • A particular focus on three most important practices for improved lower order reading skills that lead to reading fluency for all grades: 82 o reading to someone out loud at home, o reading independently at home, and o participating in reading activities after school. • Development partners should continue to advocate to donors, private sector actors and MINEDUC/REB for increased accessibility to reading materials at home. Access to reading materials creates opportunities for children to practice reading and become fluent readers, but a significant proportion of children do not take Kinyarwanda books home from school despite the existence of classroom libraries for P1-P3 students (funded through USAID Soma Umenye). Places in the community where children can go to practice reading or borrow books are still limited. • Development partners should work together with private sector actors to create a strategy to ensure books reach local communities. Specific ideas to improve book distribution could include establishing community savings groups for books. These groups could focus on groups of parents who could work on supplying books to libraries and households through savings groups. For quality control, the continuation of Abana Writers Café is essential to ensure production of literacy materials with locally relevant storylines and images. The Café can also help to monitor quality of writing in term of grammar and orthography. In addition, books using locally made materials can also be further explored. Finally, development partners should advocate to MINEDUC for school capitation grants to include a minimum allocation for the purchase of books. • When it comes to book distribution, development partners should enlist greater involvement of the Private Sector Federation and MINECOFIN to improve access to books for poor households. Examples of this could include: • Provide subsidies or grants to reduce endline costs. For instance, Save the Children is already applying for a World Bank education grant to strengthen corporate social responsibility and community engagement among companies in Rwanda. • Reduce taxes for books. This idea could help to engage other ministries who are not engaged in the day-to-day experiences of book procurement but who nevertheless have an immense impact on book accessibility through taxation policies. • Further engage multi-national actors in corporate social responsibility activities. Examples of businesses could include Volkswagen, MTN, and others. • Provide capacity-building for implementing partners in the area of private sector engagement. • Development partners and donors should consider onboarding different actors in government and the private sector who represent areas such as finance, culture, taxes, infrastructure, investment, gender, health, or youth. These actors could potentially sponsor the development and distribution of books that include key messages and content which align with their area of interest. Children’s exposure to new ideas at an early age could expand interest. • Development partners and donors should consider digital library solutions by Kigali Public Library in collaboration with REB. There may be unexplored synergies between the smart classrooms being promoted by MINEDUC/REB. However, digitization needs more research. For example, there may be challenges around electricity in rural areas, internet access, lack of IT equipment, and costs associated with digital book development. Further research should also seek to understand what lessons could be learned from the experience of the One Laptop Per Child and other similar digitally-reliant programs. Understanding past challenges and promising practices can help inform future literacy-related programming that employs digital technology. 83 ANNEX I: EVALUATION STATEMENT OF WORK I. PURPOSE The purpose of the performance evaluation is 1) to document successful approaches and lessons learned to inform the design of similar community reading/family engagement activities, and 2) to provide recommendations to promote the sustainability of Mureke Dusome activities. Findings and recommendations will be used by USAID/Rwanda, Save the Children/Rwanda and Government of Rwanda to finalize the Mureke Dusome sustainability plan; by USAID/Rwanda and Government of Rwanda to inform future project design; and by USAID and others to inform the design and implementation of similar programs in other countries. II. SUMMARY The activity to be evaluated is an initiative to improve the reading skills of students in the lower primary grades by strengthening partnerships between schools and communities. The table below summarizes basic information about the award. Activity Title Mureke Dusome (“Let’s Read”) USAID Office Education Office, Rwanda Implementing Organization Save the Children Award Number AID-696-A-16-00002 Award Dates January 4, 2016 - January 3, 2020 Funding $8,698,793 Geographic Regions Nationwide - All public and government-aided primary schools in Rwanda and the surrounding communities USAID expects to extend the award by approximately 18 months, for a revised award completion date of July 11, 2021. III. BACKGROUND A. Description of the Problem and Context Many children in Rwanda are not learning how to read well in the early grades. Mureke Dusome was designed to improve early-grade reading skills by creating school-community partnerships to support high￾quality reading instruction in school, and opportunities for reading practice outside of school. Teachers and educational leaders have the most direct influence on instructional events, but are also well-positioned to encourage school-community partnerships and promote literacy. The community is well-positioned not only to advocate for quality instruction, but also to create an environment beyond the school walls 84 that is conducive to student reading and achievement. However, school-community partnership in education is not easy to achieve. The following key challenges to achieving effective school-community partnership were identified during the design of Mureke Dusome: ● A widespread feeling among parents that involvement in schools is not their business; ● Among parents and community members, lack of time and/or lack of confidence in their ability to contribute to schooling and student learning (partially because of low levels of literacy and lack of educational attainment); ● Difficulty recruiting qualified people to serve on school committees. Community participation in school governance has been characterized by low capacity and high turnover; ● Exclusive association of school involvement with financial contribution. Even with the elimination of tuition fees, other schooling costs including financial contributions collected through School General Assemblies, strain the budgets of people living in poverty. This association is at the expense of a more open discussion on the mutual accountability of parents and teachers to achieve learning outcomes; ● Reluctance of management and leadership within schools to involve parents in two-sided dialogue and exchange; ● Inadequate communication and collaboration among educational stakeholders (i.e. parents, teachers, school leadership, local officials) to advance pupils’ learning; ● Competing/conflicting demands on the time of head teachers, such that building school￾community partnerships and promoting high-quality reading instruction are not given priority; ● Lack of evidence regarding strategies for effective school-community partnerships for improving reading skills in the Rwandan context; ● Lack of access to relevant, high-quality Kinyarwanda-language reading material, associated with the lack of capacity of the local publishing industry; and ● Lack of a nationwide culture of reading. B. Description of the Intervention to be Evaluated and Theory of Change Mureke Dusome (“Let’s Read”) is a nationwide community-focused literacy activity in Rwanda which is funded by USAID and implemented by Save the Children in collaboration with local partners Umuhuza and Urunana DC. Mureke Dusome’s key activities include: ● Working with the Rwanda Education Board to develop modules to train head teachers and school general assembly committees (SGACs), who, in turn, elect Literacy Champions and promote regular reading activities in their communities, ● Providing book banks to communities to promote reading outside of school, ● Implementing a social-behavior change communication campaign to popularize key literacy messages and content through media including print and audio-visual methods such as popular radio programs, and ● Training and encouraging actors in Rwanda’s publishing industry to develop more high-quality storybooks for children. Through these activities, Mureke Dusome aims to create and strengthen partnerships between schools and communities, increase effective community and parental involvement in children’s reading activities, and, ultimately, foster a “culture of reading” throughout Rwanda. Mureke Dusome defines supportive school-community partnerships and community/home-learning environments with the following measurable characteristics: 85 ● Students participate in community reading activities; ● Students spend time engaged in reading practice outside of school; ● School General Assembly Committees are active and discuss literacy at the meetings; ● Head Teachers encourage teachers to communicate with parents regarding their children’s reading progress, support community literacy activities, and communicate literacy promotion messages to parents; ● Parents/families are aware of their children’s progress in reading and take specific actions to support their children’s literacy development at home; and ● Parents and students have positive attitudes regarding reading. The theory of change is that children learn to read better in these supportive conditions than if they are receiving classroom-based instruction alone. Mureke Dusome is designed to complement the classroom￾based interventions of USAID Soma Umenye. IV. EVALUATION RESEARCH QUESTIONS USAID/Rwanda has identified three primary evaluation questions. Evaluators will review and finalize questions in collaboration with USAID prior to finalizing the evaluation design. 1. How has Mureke Dusome improved literacy-related knowledge, attitudes, and practices? To the extent possible, analysis of improvements in literacy-related knowledge, attitudes and practices (KAP) should include analysis about the extent to which various sub-populations (e.g., people with disabilities, and people across the gender spectrum) have benefited from Mureke Dusome implementations. Sub-questions: ● How have the most successful communities targeted by Mureke Dusome transformed their community culture to be more supportive of children's literacy? ● What were the biggest perceived contributors to literacy behavior change among head teachers, parents and children? ● What has been Mureke Dusome’s contribution to the capacity building and system strengthening of the Rwandan publishing industry? ● What has been Mureke Dusome’s contribution to the capacity building and system strengthening of schools and local government? ● Which aspects of Mureke Dusome were not successful and why? 2. Which knowledge, attitudes and practices are correlated with higher [or improved] student reading skills? USAID’s Soma Umenye activity has collected data about student reading scores; the student context survey for Soma Umenye’s Early Grade Reading Assessment (EGRA) also included questions about KAP targeted by Mureke Dusome. Analysis of data sets collected through Soma Umenye (and Mureke Dusome) should be used to answer this question. 3. What elements of Mureke Dusome program are likely to be sustained? How could the sustainability of the Mureke Dusome program be further promoted? Mureke Dusome interventions have included various activities, such as facilitating children’s participation in reading clubs, supporting the development of strong partnerships between Head Teachers and SGACs, building the capacity of the children’s book industry to produce more and better books, and messaging 86 through various campaigns (e.g. Gira Igitabo, Urunana radio, other Rwanda TV/radio announcements, Rwanda Reads National Literacy Month, etc.). Mureke Dusome’s general approach to achieve sustainability includes the following three results: 1. Sustained motivation of Communities and Community Based Literacy Volunteers (Literacy Champions) 2. Sustained technical and managerial capacity of head teachers, School General Assembly Committees and local leaders to cooperate and promote literacy supportive practices; and 3. Ongoing supply of accessible high-quality age-appropriate Kinyarwanda reading materials for children. Sub-questions: ● How does Mureke Dusome define sustainability? ● Is there demonstrable ownership and demand for Mureke Dusome interventions? How could Mureke Dusome further promote ownership and demand? (cover the following stakeholders) ○ Government of Rwanda officials at the national level ○ District and sector leaders ■ Are Mureke Dusome activities in district and sector plans? ○ Village and school leaders (including head teachers and School General Assembly Committee representatives) ■ How could Mureke Dusome be better embedded in local institutions like Umugoroba w’Ababyeyi and Umuganda? ○ Literacy Champions ○ Families and children ● What is the level of skills and capacity among stakeholders to sustain Mureke Dusome interventions? How could Mureke Dusome further build up skills and capacity among stakeholders? (cover the following stakeholders) ○ Book sector ○ Government of Rwanda officials at the national level ○ District and sector leaders ○ Village and school leaders ○ Literacy Champions • What are the recurrent financial costs of Mureke Dusome interventions, and will future revenue streams be sufficient to sustain them? What could Mureke Dusome do to ensure sustained financing for their interventions? V. EVALUATION DESIGN AND METHODOLOGY This evaluation will require a mixed-method approach, that may include (but not necessarily be limited to) the following: 87 ● Document review ○ Mureke Dusome program description ○ Mureke Dusome implementation plans ○ Mureke Dusome Monitoring, Evaluation and Learning plan ○ Mureke Dusome sustainability plan ○ Annual/quarterly reports, including performance monitoring data ○ Mureke Dusome/Save the Children baseline and midline KAP reports ○ Mureke Dusome/Save the Children baseline and endline impact evaluation reports ○ Mureke Dusome/Save the Children qualitative/process evaluation ● Key informant interviews ○ USAID/Rwanda management and staff ○ Save the Children/Rwanda management and staff ○ Urunana and Umuhuza management and staff (local subpartners) ○ Rwandan Government officials (MINEDUC and REB, MINISPOC, MINALOC, MIGEPROF) ○ Members of the Rwanda Reads steering committee ● Focus group discussions ○ Parents ○ School and community leaders (district, sector, village) ○ Literacy Champions ○ Children’s book sector ○ Children (optional) ● Quantitative data analysis ○ Soma Umenye baseline/midline student assessment and student context survey data ○ Soma Umenye Impact Evaluation baseline student assessment and student context survey data ○ Mureke Dusome KAP survey data Note that the Contractor will be expected to use existing data sets from the USAID Soma Umenye and Mureke Dusome activities; the Contractor is not required to collect additional quantitative data. VI. DELIVERABLES AND REPORTING REQUIREMENTS While USAID has provided the required deliverables with an illustrative schedule below; the offeror may suggest a more aggressive timeline. USAID requests the Offeror to include as an annex a complete and detailed evaluation implementation calendar showing the timeline for the achievement of major activities, including deliverables. 1. Detailed Evaluation Work Plan, including evaluation design: To be submitted to the Contracting Officer’s Representative (COR) at USAID/Rwanda for approval no later than the tenth day of work. The work plan will include: 1) the anticipated schedule and logistical arrangements for the overall evaluation study plan, including data collection activities and analyses as well as clearance of any local relevant IRB procedures; 2) a list of the members of the evaluation team, delineated by roles and responsibilities; and 3) the list of contacts and documents requested from USAID. 88 The evaluation design will become an annex to the evaluation report. The evaluation design will include: 1) detailed evaluation design matrix that links the Evaluation Questions from the SOW (to be finalized collaboratively between USAID and the Evaluation Team) to data sources, methods, and the data analysis plan; 2) the proposed tools (for interviews, focus groups, etc.); 3) a list of potential interviewees and sites to be visited and proposed selection criteria; 4) limitations to the evaluation design; and 5) the dissemination plan (designed in collaboration with USAID). The evaluation design may be shared by USAID with stakeholders including the Mureke Dusome and Soma Umenye teams for comment before finalization. The data analysis plan should clearly describe the evaluation team’s approach for analyzing quantitative and qualitative data, including proposed sample sizes, specific data analysis tools and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans should be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to help communicate evaluation findings and recommendations. See the Evaluation Toolkit (https://usaidlearninglab.org/evaluation-toolkit) for guidance on Developing an Evaluation Dissemination Plan (https://usaidlearninglab.org/library/developing-evaluation-dissemination-plan￾0). If applicable based on the Disclosure of Conflict of Interests Forms (https://usaidlearninglab.org/library/sample-disclosure-conflict-interest-form) submitted with the awardee’s proposal, the evaluation design will include a conflict of interest mitigation plan. USAID offices and relevant stakeholders will provide consolidated comments through the COR within 5 working days. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they will be expected to return with a revised evaluation design and work plan within 5 working days. 2. Data Collection: USAID/Rwanda will provide overall direction to the evaluation team, and identify and provide key documents. USAID/Rwanda will also facilitate communication with Save the Children/Rwanda, including about the Offerer’s requests for consultation and requests about recommended beneficiaries and other participants to interview. USAID/Rwanda will lead in arranging meetings with Government of Rwanda (GOR) stakeholders for the purpose of this evaluation. The Offeror will be responsible for the logistics for other interviews and focus groups (e.g. arranging meetings, transport, etc.). 3. Preliminary draft evaluation report: Submit an early draft or detailed outline that includes main findings and bullets to the USAID COR, who will provide preliminary comments prior to the presentation. 4. Preliminary presentation for recommendations development: The evaluation team is expected to conduct a preliminary presentation to discuss the summary of findings and conclusions with USAID and Save the Children/Rwanda, and to draft collaboratively any requested recommendations. 89 5. Draft evaluation report: The draft evaluation report should be consistent with the guidance provided in Section VII, Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The draft report should incorporate discussion from the initial presentation of findings. USAID will provide comments on the draft report within two weeks of submission. The submission date for the draft evaluation report will be determined in the evaluation work plan. 6. Final Report: The evaluation team leader will submit a final report that incorporates responses to USAID comments and suggestions to the USAID COR electronically, no later than ten days after USAID/Rwanda provides written comments on the team’s draft evaluation report. The evaluation report must meet USAID evaluation quality standards (see Appendix 1 of the USAID Evaluation Policy, http://www.usaid.gov/sites/default/files/documents/2151/USAIDEvaluationPolicy.pdf). 7. Submission of Dataset(s) to the Development Data Library: Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non￾proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. Please review ADS 579.3.2.2 Types of Data To Be Submitted to the DDL to determine applicability. 8. Submission of Final Evaluation Report to the Development Experience Clearinghouse: Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. 9. Dissemination: The evaluation team is expected to hold a final presentation to discuss the summary of findings, conclusions and recommendations with USAID and other stakeholders. VII. EVALUATION TEAM COMPOSITION The proposed evaluation team must include staff with demonstrated quantitative and qualitative analysis skills, excellent writing skills, and experience conducting focus groups and interviews. Experience conducting evaluations of education and/or community-based interventions, and evaluations in Rwanda, is preferred. USAID encourages that evaluation specialists from Rwanda lead or participate in evaluation teams when appropriate expertise exists or when engaging local evaluation specialists will facilitate institutional learning and/or capacity development. Proposed key personnel are expected to be the people who execute the work of this contract. Any substitutes to the proposed key personnel must be vetted and approved by the COR before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means. VII. FINAL REPORT FORMAT The final report should include the following sections, as described in the How-To Note on Preparing Evaluation Reports (https://usaidlearninglab.org/library/how-note-preparing-evaluation-reports) and ADS 201mah, USAID Evaluation Report Requirements. (https://www.usaid.gov/ads/policy/200/201mah). 90 The report should be formatted according to the Evaluation Report Template. (https://usaidlearninglab.org/library/evaluation-report-template) 1. Abstract 2. Executive summary 3. Evaluation purpose 4. Background on the context and the strategies/projects/activities being evaluated 5. Evaluation questions 6. Methodology 7. Limitations to the evaluation 8. Findings, conclusions, and recommendations 9. Annexes The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions as well as recommendations and lessons learned. The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. Annexes to the report must include: ● Evaluation SOW (updated, not the original, if there were any modifications); ● Evaluation design; ● All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● All sources of information or data, identified and listed; ● Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; ● Signed disclosure of conflict of interest forms (https://usaidlearninglab.org/library/sample￾disclosure-conflict-interest-form) for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and ● Summary information about evaluation team members, including qualifications, experience, and role on the team. IX. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, (https://www.usaid.gov/ads/policy/200/201maa) draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report; 91 ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ● Evaluation methodology should be explained in detail and sources of information or data properly identified; ● Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ● Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements (https://www.usaid.gov/ads/policy/200/201mah) and the Evaluation Report Checklist and Review Template (https://usaidlearninglab.org/library/evaluation￾report-checklist-and-review-template)from the Evaluation Toolkit (https://usaidlearninglab.org/evaluation￾toolkit) for additional guidance. X. OTHER REQUIREMENTS All modifications to the required elements of the SOW of the contract/agreement, whether in evaluation questions, design and methodology, deliverables and reporting, evaluation team composition, schedule, and/or other requirements will be agreed upon in writing by the COR. Any revisions made will be noted in the SOW annexed to the final Evaluation Report. 92 ANNEX II: APPROVED EVALUATION DESIGN USAID MUREKE DUSOME ACTIVITY PERFORMANCE EVALUATION DESIGN 93 Proteknôn Consulting Group hereby submits this Evaluation Design for the performance evaluation of the USAID￾funded Mureke Dusome project implemented by Save the Children in Rwanda. Proteknôn members have experience studying the challenges to improving early literacy in Rwanda. We understand that the purpose of this evaluation is: 1. to document successful approaches and lessons learned to inform the design of similar community reading and family engagement activities in Rwanda and elsewhere; and 2. to develop recommendations to promote the sustainability of Mureke Dusome activities. Our key personnel for this assignment – both national and international – have extensive experience working in Rwanda and have significant expertise in the education sector. Recent clients in Rwanda include Save the Children (conducting a systemic analysis of the Advancing the Right to Read project), Girl Effect, International Organization for Migration (IOM), the World Bank, and many others. 1. EVALUATION TEAM Our team of evaluation and education specialists has the skills and expertise needed to conduct this evaluation and make informed recommendations to ensure that Mureke Dusome interventions are both effective and sustainable. In particular, they have experience designing and implementing participatory, community-based, child-friendly qualitative and quantitative evaluation methodologies. They also have a deep understanding of the education sector and context in Rwanda. Our evaluation team includes key personnel who collectively offer skills, knowledge and personal qualities that add significant value to both the technical and operational aspects of the proposed performance evaluation: 1.1 KEY ROLES & RESPONSIBILITIES Our team members have clearly defined roles and responsibilities to ensure that we work together in complementary ways that make the best use of each person’s experience and expertise: Nidhi Kapur is a Senior Associate with Proteknôn and a child protection specialist with over ten years of field-based evaluation experience. Motivated by a strong interest in the complexities of protection programming in conflict and post-conflict zones, Ms Kapur has worked for international organizations such as the British Red Cross, Save the Children, War Child UK, Handicap International, and Right to Play. As part of emergency response teams, she has been deployed to various countries including Liberia, South Sudan, the Democratic Republic of Congo as well as Israel and the occupied Palestinian territories. She has experience leading projects on a multitude of child￾related issues, including education, disability and inclusion, gender-based violence, and sexual exploitation and abuse. She holds a graduate degree from the London School of Economics. Ms Kapur has lived in East Africa since 2008 and is based full-time in Kigali as a registered consultant. Team Lead: Nidhi will take overall responsibility for the day-to-day coordination of the assignment. She will ensure timely implementation, communication, joint planning and liaison with USAID and other key stakeholders, as well as quality control assurance. She will also work alongside other team members on the inception, analysis and drafting phases of the assignment, co-authoring and presenting the final report. She will carry out key informant interviews with relevant stakeholders at the national level. She will also oversee both field-based qualitative fieldwork as well as help guide the overall analytical strategy with the quantitative datasets. She will be also be responsible for managing and capacity￾strengthening of national counterparts within the team through training and supervision. Dr Timothy Williams is a Senior Associate with Proteknôn who has been a researcher and consultant in Rwanda’s education sector since 2010. His PhD thesis qualitatively examined Rwanda’s basic education policy. Tim’s recent projects include a learning paper for Save the Children’s efforts to Technical Specialist – Education: Dr Williams will significantly contribute to the overall evaluation using his expertise in the Rwandan education sector. He will lead the design of qualitative research methods, tool design, and analysis. He will also co￾author the final report. 94 promote systemic change in the area of early childhood literacy. This project included a focus on Mureke Dusome. Dr Williams has published extensively on these issues and is currently under contract with Cambridge University Press to write a book on schooling in Rwanda. Additionally, Tim has conducted analyses of education policies and programs globally for Harvard University, the World Bank and the Brookings Institution. He is an expert on qualitative research methods with significant experience designing studies that include children’s views and experiences. He has graduate degrees from the University of Bath (PhD), Harvard (MSc), and Boston College (MSW). Alphonse Nshimyiryo is a quantitative specialist with a commitment to social justice. He uses his data analysis skills to support evidence-based decision making in order to improve lives. He is proficient in the statistical packages (e.g. SPSS, STATA, R) used to analyze large datasets, such as those associated with this assignment. Since completing a fellowship with Global Health Corps three years ago, Mr. Nshimyiryo has been a research data analyst for Partners in Health/Inshuti Mu Buzima (PIH/IMB). He holds a bachelor’s degree in Applied Statistics from the University of Rwanda. Mr. Nshimyiryo is currently completing a Master’s in Science in Demography and Health by distance from the University of London through the London School of Hygiene and Tropical Medicine. Quantitative Data Analyst: Alphonse Nshimyiryo will lead on the analysis of existing quantitative data sets (provided by USAID) and drafting of preliminary findings to related research questions for incorporation into the final evaluation report. Jeanine Balezi is an Associate with Proteknôn. She is a researcher, cultural advisor, and translator with ten years of experience conducting and leading research teams, including monitoring and evaluation, on issues related to young people and families, education, gender, health, internally displaced people, and reconciliation in Rwanda. She has worked for scholars from Stanford University and the University of Oxford, and for many organizations, including UNICEF Rwanda, Save the Children, USAID, and the Clinton Health Access Initiative (CHAI). Jeanine is hardworking with a University degree (BA), an outgoing personality and excellent communication skills. She is fluent in Kinyarwanda (native), English (fluent), French (fluent) and Swahili (fluent). Field Coordinator: Jeanine Balezi will oversee sub￾national fieldwork in the selected districts. She will contribute to the design of the evaluation tools (including translation and field-testing) and to the training and supervision of four local Research Assistants. She will conduct key informant interviews at district and local levels as part of data collection. She will also contribute to, and supervise, data entry and analysis. 1. Hosanne Ingabire: A graduate in Mental Health from the former Kigali Health Institute (now University of Rwanda, College of Medicine and Health Sciences), Hosanne has more than eight years of professional experience in research related assignments, including with academic institutions, government ministries and agencies. She has a strong track record in Research Assistants: Sub-national fieldwork will be conducted with the support of a gender-balanced team of four national Research Assistants. Although they are already experienced researchers, they will participate in a two-day orientation, briefing and refresher training prior to commencement of fieldwork. This will ensure they have sufficient familiarity with the Mureke Dusome project in general, and the aims of the evaluation in particular, in order 95 qualitative data collection, transcription and translation. 2. Flora Mutimukeye: With over five years of experience in field research, including assistance to PhD candidates from overseas universities, Flora is familiar with data collection protocols, including facilitating focus group discussions and conducting interviews, data analysis and reporting. 3. Francois-Xavier Harelimana: With over ten years’ work in education, child protection and gender sectors, Xavier has acquired extensive monitoring and evaluation experience. He has excellent writing and facilitation skills. He is familiar with all aspects of field-based data collection including planning, preparation, logistics, use of tools, transcription and reporting. He has a bachelor’s from Kigali Institute of Science and Technology. 4. Olivier Hakizimana: Having worked for a number of INGO and UN agencies, as well as private consulting firms, Olivier has developed solid multi-sectoral expertise. Notably, he has administered focus group discussions and key informant interviews in relation to literacy, parenting curriculum and early child development projects across Rwanda. to improve data collection outcomes. They will work in pairs during each focus group discussion, and will also be responsible for transcription of research notes. Proteknôn consultants: We have provided the above profiles of the Proteknôn consultants who form the core team for this assignment, but we are also able to draw upon the skills and expertise of all Proteknôn consultants, as needed. Proteknôn Group is comprised of a diverse and skilled team of approximately thirty-five consultants with expertise in education, child protection, gender-based violence, safeguarding, and research methods. Collectively, our associates speak ten languages and have experience working in over 100 countries. 2. EVALUATION DESIGN 2.1 OUR APPROACH At Proteknôn, we approach our work in a way that is principled, participatory, appreciative, equitable, and rights￾based. We are always transparent about the purpose of our research and analysis and employ the highest possible ethical and safeguarding standards. We provide opportunities for stakeholders to contribute to, and meaningfully inform, our work. We do this by sharing findings and incorporating feedback, and by engaging collaboratively with a full range of stakeholders from a position of integrity, humility, and mutual respect. We will apply a rights-based, child-centered, equity-focused, inclusive, and gendered lens to this assessment. Incorporating these values into our work enables us to quickly build rapport and undertake solid evaluations in short timeframes. 96 2.2 METHODOLOGY This performance evaluation will take a mixed-methods approach, including a document review, analysis of existing quantitative data sets, as well as collection and analysis of primary qualitative data at national and sub-national levels. Quantitative data will be used to understand scale and scope, while qualitative data will illuminate on-the-ground perspectives, including those of children, parents, teachers and other community members and national stakeholders involved in Mureke Dusome. With due consideration to ethical and safeguarding considerations, we propose the following data collection methods: ● Document review: We will undertake an initial desk-based review of existing literature on Mureke Dusome provided by USAID and Save the Children. This will include project documents, such as the program description; implementation plans; monitoring, evaluation and learning plans; sustainability plans; and existing reports and assessments. A list of requested documents is included in Annex I. In addition, given our strong familiarity with (and contribution to) literature related to Rwanda’s education sector, we will undertake a detailed search for relevant peer-reviewed and grey material. This review will identify lessons learned and help to inform the study design, qualitative question guides, analysis of the datasets, and interpretation of qualitative data. A synthesis of the documentation review will be integrated into Part 4 of the final report under “Background on the context and the strategies/projects/activities being evaluated.” ● Key Informant Interviews (KIIs): Proteknôn will work with Save the Children and USAID to jointly identify national, district and local level key informants from a range of project stakeholders, including USAID, Save the Children, local partners, and authorities. Semi-structured interviews will revolve around the key research questions to gather expert insights not included in existing publications. ● Focus Group Discussions (FGDs): We will develop tools to conduct focus group discussions with parents and children who are familiar with the Mureke Dusome intervention through their participation in School General Assembly Committees (SGAC) and reading clubs, respectively. We aim for FGDs to be between 7 and 10 members. Discussions will last between 40 to 60 minutes in length. The tools themselves will aim to draw on the experiences and insights of participants to produce data relevant to understanding the effectiveness and long-term sustainability of Mureke Dusome. ● Quantitative Data Analysis: In light of Research Question #2 in the Statement of Work (SOW), we will analyze the Early Grade Reading Assessment (EGRA) data collected in 2018 through Soma Umenye. We will describe knowledge, attitudes and reading practices (KAP) of students and EGRA scores using frequencies and percentages for categorical variables as well as mean and standard deviations (SD) for continuous variables. Then, we will investigate KAP factors correlated with higher students’ reading skills using logistic regression for binary outcome variables and multiple linear regression (MLR) for continuous outcome variables. Two multivariable logistic regression models will be used to identify KAP factors associated with increased odds of higher oral reading fluency (ORF) and reading comprehension, categorized based on benchmarks validated for Kinyarwanda by Soma Umenye and the Rwanda Education Board (REB) in June 2019. The models will be adjusted for gender, disability and grade level and will be built using backward stepwise procedures for all KAP variables. All factors with a statistically significant association (α=0.05) with outcome variables will be retained in the final models. In addition, since the vast majority of children may not be meeting ORF and reading comprehension benchmarks with a significant proportion of students scoring zero on these components - and considering that other lower order reading skills can improve prior to an observable improvement in ORF and reading comprehension, we will use MLR analysis to assess KAP factors correlated with higher EGRA scores on pre-requisite literacy skills including letter identification, syllable identification, familiar word reading and ORF subtasks. We will exclude reading comprehension from the MLR analysis, because it may be inappropriate for a bound and discrete outcome variable with only 6 values (i.e. 0-5 possible comprehension questions correct). This quantitative analysis will be used to inform the qualitative research design. . 97 ● Validation and feedback: Preliminary findings will be shared ahead of the final report in order to allow key stakeholders the opportunity to provide feedback on our initial analyses and co-develop final recommendations. 2.3 PROPOSED TOOLS Question guides for the FGDs and KIIs are located in the annexes of this report. These tools were directly informed by Research Questions #1 and #3, while drawing - wherever possible - on existing tools developed by Save the Children for other evaluations of the program. Prior to the commencement of the fieldwork, Save the Children has agreed to work with members of the research team to help facilitate the piloting of the tools. Any final adjustments to the tools will be made at this time. These include the following tools: National KII question guides: 1) Government and other national-level stakeholders 2) Children’s book industry representatives Sub-national KII question guides: 3) Officials at district level 4) Local education officials (e.g., head teachers, Literacy Champion representatives) 5) Other local education stakeholders (e.g., local leaders, faith-based organizations (FBO), etc.) Sub-national FGD question guides: 6) Parent members of School General Assembly Committee (SGAC) 7) Children who participate in Mureke Dusome reading clubs 2.4 SAMPLING STRATEGY AND SELECTION CRITERIA Site Selection The national-level focus of the assessment will draw from existing literature and policy reports and semi-structured key informant interviews. KIIs will include representatives and members of government, civil society, development partners, local education officers and school officials, and representatives of the book industry, as described in further detail below. Given the aims of this project with respect to program effectiveness and sustainability, all participants must have a strong knowledge of and/or experience with Mureke Dusome interventions. Sub-national work will be carried out in 5 districts and 10 sectors, including one district per province and two sectors per district. These districts include: Gasabo District, Kirehe District, Ruhango District, Burera District, and Ngororero District. Districts and sectors were selected in consultation with Save the Children. The primary criteria for site selection was stakeholders’ familiarity with Mureke Dusome. This is to ensure that the sample population can speak knowledgeably about their engagement with the program with respect to priority research questions. In consultation with Save the Children, we have identified these sites to obtain local-level insights from children, parents, head teachers, and other education officials with good knowledge of Mureke Dusome interventions. We have worked with Save the Children to ensure that one of the sectors is what Save staff members would consider a community where Mureke Dusome’s work has been deemed particularly “successful.” Per our conversations with Save the Children, “successful communities” are those in which their involvement in Mureke Dusome is characterized by strong leadership, a functional reading club, and strong parental involvement. The second sector selected is not necessarily “unsuccessful,” per se, but instead may be characterized by more modest achievements in terms of program effectiveness. It may also be a sector in which particular challenges were faced, from which lessons learned can be drawn to inform future programming. For these reasons, site selection will aim to account for factors such as the length of Mureke Dusome intervention in the targeted area, and key personnel changes, among others. The breakdown of specific sites and criteria for their selection are outlined in greater detail below: 98 District Sector 1 (“Successful sector”) Sector 2 (“Other”) Gasabo District • Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome activities compared with other districts in Kigali City Nduba Sector (GS Nduba) • Functional reading club • Literacy Champions are active • School collaborates with Literacy Champions in promoting children’s literacy activities • Community supports the reading club Ndera Sector (GS Ndera Catholique) • Motivated Literacy Champions with moderate community support • Low attendance of children in reading club • Urban environment Kirehe District • Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Eastern Province Nyarubuye Sector (GS Migongo) • Reading club meets regularly • Community engagement with good collaboration between the school and Literacy Champions Gahara Sector (GS Gahara) • One Literacy Champion is committed, while another has dropped out Ruhango District • Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Southern Province Mbuye Sector (EP Kabuga) • SGAC is engaged, visits reading club and collaborates with Literacy Champions • Good community engagement, with parents supporting Literacy Champions • Literacy Champions are active • School is located in rural area Byimana Sector (GS Bukomero) • Literacy Champions are committed, with collaboration with the school • Children are enjoying the reading club but parents are not as engaged • School is located in urban area 99 Selection of Key Informants & FGD Participants: Fieldwork at the sub-national level will be prepared in close consultation with Save the Children and its partners in order to identify and schedule interviews with selected key informants. National level At the national level, we plan to interview approximately 21 total key informants (in individual and small group sessions), including the following specific individuals who can speak knowledgeably about Mureke Dusome from the following institutions: ● Government of Rwanda: o MINEDUC: ▪ Minister of State for Primary and Secondary Education ▪ Public and Community Libraries Officer o REB: ▪ Director of School Leadership & Management Unit ▪ School Community Partnerships Coordinator (embedded) Burera District • One of two districts of Rwanda where Mureke Dusome first commenced interventions • Shares border with Uganda (cross-border trade and linguistic specificities compared to other areas of Rwanda) • Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Northern Province Rugarama Sector (EP Gafumba) • Strong collaboration between local leaders, parents and Literacy Champions in promoting children’s literacy activities • Local leaders are motivated and work with parents to support Literacy Champions to develop materials • Parents attend the reading club Kivuye Sector (EP Rugarambiro) • Community speaks Urukiga dialect which presents difficulties to children when adapting to reading materials in Kinyarwanda • Reading club does not take place regularly Ngororero District • Has a successful school community in supporting children’s literacy and has benefitted from more exposure to Mureke Dusome compared with other districts in Western Province Nyange sector (EP Nyange) • Literacy Champions are committed • Children’s attendance is high • Community members are supportive • Rural environment Ngororero sector (Nyange Protestant) • Literacy Champions are committed • Children’s attendance is moderate • Community engagement is limited • Urban environment 100 ▪ Elisabeth Turner, Advisor to the Director General of REB (USAID Soma Umenye funded) o MINALOC: ▪ National Itorero Commission: Director of Mentoring & Volunteerism o MINISPOC: ▪ Personal Advisor to Minister ● Key INGO, civil society, private sector and other stakeholders: o Save the Children/Rwanda: ▪ Chief of Party Mureke Dusome ▪ Deputy Chief of Party o USAID/Rwanda: ▪ Senior Education Specialist o Additional member of Rwanda Reads: ▪ UNICEF Education Specialist ▪ Pacifique Mahirwe – Director - Rwanda Bookmobile o World Vision: ▪ Education Programme Manager o Soma Umenye: ▪ Deputy Chief of Party o Urunana DC: ▪ Director o Umuhuza: ▪ Executive Director ▪ Community Engagement Coordinator o Children’s book sector representative: ▪ Chairperson, Rwanda Children’s Book Organization ▪ Mutesi Gasana, Director of Arise Education ▪ Fiston Mudacumura – Director, Mudacumura Publishing House ▪ Martine Uwacu – Director Sankofa Creatives Sub-national level At the sub-national level, we plan to carry out work in five districts, with two sectors per district. In each district, we will employ a purposive sampling strategy that aims to consult with some of the key stakeholders of the Mureke Dusome intervention. Our sampling strategy for sub-national work is summarized in the table below: Interviewee # per unit # units type unit total # units District JADF representative 1 5 district 5 DDE or DEO 1 5 district 5 Local (2 sectors per district) Sector Education Inspector (SEI) 1 10 sector 10 Head Teacher or Deputy Head Teacher 1 10 school 10 Literacy Champion Representative 1 10 sector/school 10 101 Others (cell/village leader, FBO etc.) 1 10 sector/cell/village 10 Local level FGDs within sectors Parents (SGAC President, Vice￾President and parents whose children attend reading clubs) 1 7-10 sector/village 10 Children (reading club members) 1 7-10 sector/village 10 Total KIIs/FGDs 70 # per district 14 At the school level, we intend to work with Head Teachers to help schedule FGDs with parents and children. FGDs with parents will comprise of Presidents and Vice-Presidents of SGACs with whom Mureke Dusome worked directly, as well as parents of children attending reading clubs. FGDs with children will include those who have been exposed to Mureke Dusome for at least one year, including regular participation in reading club activities. Both parent and child FGDs will be mixed-gender groups, with equal representation of males and females. Any parents or children with disabilities will also be actively identified and included in these groups. The tools have been designed to examine gender and/or inclusion-related dynamics in relation to the proposed evaluation. Research Assistants who lead the FGDs will be trained to focus on ensuring that the voices of all participants will be heard. Our sampling strategy is purposive. It has been designed to strike a good balance between the breadth and depth that this evaluation requires. By selecting districts from across Rwanda’s five provinces, our sample offers sufficient geographic diversity to identify variation (e.g., urban vs rural). The study design also allows for a vertical investigation into our evaluation questions of interest, by speaking with stakeholders in various roles at the local levels. We believe that the sample size will be sufficient to achieve theoretical saturation of some of the core themes to be explored in this study. And while our methods are systematic, they are not rigid, in the sense that our study team will be trained to ensure that all those we speak with are indeed the most knowledgeable about Mureke Dusome interventions. 2.5 EVALUATION DESIGN MATRIX Evaluation Questions & Sub-Questions Data Sources Data Collection Methods Explanatory Variables of Interest, Data Analysis RQ1. How has Mureke Dusome improved literacy-related knowledge, attitudes, and practices? How have the most successful communities targeted by Mureke Dusome transformed their community culture to be more supportive of children's literacy? Qualitative data at national and sub￾national levels; site selection of ‘most successful communities’ within districts • National KIIs • Sub-national KIIs • FGDs with parents • FGDs with children • Document review Individuals, national/local incentives, and organizations that have been drivers for transformation and innovation; Thematic analysis of qualitative data; also identify ‘outliers’ and the factors that have facilitated transformation What were the biggest perceived contributors to changes in literacy behavior among head teachers, Qualitative data at national and sub￾national levels; site selection `of ‘most successful communities’ • Sub-national KIIs • FGDs with parents • FGDs with children • Document review Identify examples of successful literacy practices by teachers, parents, and children; locate linkages between these best practices and specific Mureke Dusome activities; Thematic analysis of qualitative data, disaggregated by sub-national level and demographic factors (gender, region, dis/ability as possible) 102 parents and children? What has been Mureke Dusome’s contribution to the capacity building and system strengthening of the Rwandan publishing industry? Qualitative data at national level; literature and document review • National KIIs • Review existing documents pertaining to children’s book sector Examine the technical successes of Mureke Dusome to improving Rwandan publishing; also identify other factors of success to building capacity and system strengthening; Thematic analysis of qualitative data; also identify strategies or ‘wins’ for Mureke Dusome that may not yet be documented, e.g., REB buy-in, working with private sector, etc. What has been Mureke Dusome’s contribution to the capacity building and system strengthening of schools and local government? Qualitative data at national and sub￾national levels; site selection of ‘most successful communities’ • National KIIs • Sub-national KIIs • FGDs with parents • Document review Thematic analysis of qualitative data, identifying concrete examples of how systems have been strengthened at sub-national levels (i.e., district, sector, school/SGAC). Which aspects of Mureke Dusome were not successful and why? Qualitative data at national and sub￾national levels • National KIIs • Sub-national KIIs • FGDs with parents • FGDs with children • Document review At the national level, focus on identifying unsuccessful aspects of program design, supported by national KIIs and existing documents. At local level, focus on understanding and explaining unsuccessful implementation; Thematic analysis of qualitative data, disaggregated by sub-national level and demographic factors (gender, region, dis/ability as possible) Examples of implementation issues could include political, social, technical (performance contracts) factors, among others. RQ2. Which knowledge, attitudes and practices are correlated with higher [or improved] student reading skills? Which knowledge, attitudes and practices are correlated with higher [or improved] student reading skills? Soma Umenye baseline/midline student assessment; student context survey data; Soma Umenye Impact Evaluation baseline student assessment and student context survey data; Mureke Dusome KAP survey data Analysis will draw from existing data sets to perform a bivariate and multi-variate statistical analysis, including regression analysis Focus on the extent to which knowledge, attitudes and practices are correlated with higher or improved student reading skills, and how this may be linked with Mureke Dusome. Consider the role of age, gender, disability. RQ3. What elements of Mureke Dusome program are likely to be sustained? How could the sustainability of the Mureke Dusome program be further promoted? What elements of Mureke Dusome program are likely to be sustained? How could the sustainability of the Qualitative data at national and sub￾national levels • National KIIs • Sub-national KIIs • FGDs with parents • FGDs with children At national level, identify how elements of program have been integrated into policy, adopted by other organizations, or incorporated into private book sector. At local level, focus on what elements of the program can (or could be) sustained through other 103 Mureke Dusome program be further promoted? • Document review efforts, e.g., government uptake through performance contracts, SGAC accountability, demand for literacy focused programming on radio, etc. Thematic analysis of data gathered, disaggregated into national / sub-national levels, and further disaggregated according to the program components of Mureke Dusome when possible Is there demonstrable ownership and demand for Mureke Dusome interventions? How could Mureke Dusome further promote ownership and demand? Qualitative data at national and sub￾national levels • National KIIs • Sub-national KIIs • FGDs with parents • FGDs with children • Document review At national level, understand investigate uptake and ownership of Mureke Dusome initiatives by government and nongovernment actors (including private sector). At local level examine what ownership and demand look like and how this can be improved; Thematic analysis of data gathered, disaggregated into national / sub-national levels, and further disaggregated according to the Mureke Dusome interventions when possible What is the level of skills and capacity among stakeholders to sustain Mureke Dusome interventions? How could Mureke Dusome further build up skills and capacity among stakeholders? Qualitative data at national and sub￾national levels within districts selected in consultation with SC and USAID • National KIIs • Sub-national KIIs • FGDs with parents • FGDs with children • Document review Analysis will focus on identifying what skills and capacity key stakeholders in the program have, and based on their experience/expertise, identify areas they feel they need further support in to sustain program; Thematic analysis of data gathered, disaggregated into national / sub-national levels, and further disaggregated according to the Mureke Dusome interventions when possible 2.6 QUALITATIVE AND QUANTITATIVE DATA ANALYSIS PLAN For the qualitative analysis (RQ #1 and RQ #3), interview and focus group transcripts will be uploaded to a qualitative software program called NVivo. They will then be analyzed thematically with respect to the research questions and sub-questions of the study in order to identify the most prominent themes, drawing attention to gender, geographic, and role (beneficiary, local official, etc.) as appropriate. For the quantitative analysis (RQ #2), we will use the 2018 EGRA data to investigate knowledge, attitudes and reading practices (KAP) correlated with higher students’ reading skills. Outcome variables - Binary outcome variables (Based on benchmarks for Kinyarwanda validated by REB in June 2019) o “Higher/improved reading comprehension” will be defined as an EGRA score on reading comprehension ≥60% for P1&P2 students and ≥80% for P3 students. o “Higher/improved oral reading fluency (ORF)” will be defined as an EGRA score on ORF ≥10 correct word per minute (cwpm) for P1 students, ≥25 cwpm for P2 students, and ≥40 cwpm for P3 students. - Continuous outcome variables o “Letter identification” i.e. number of correct letters per minute (ranging between 0 and 100) 104 o “Syllable identification” i.e. number of correct syllables per minute (ranging between 0 and 100) o “Familiar word reading” i.e. number of correct word per minute (ranging between 0 and 50) o “ORF” i.e. number of words of connected text read correctly per minute (ranging between 0 and 38) Descriptive statistics We will describe KAP and outcome variables using frequencies and percentages for categorical variables and mean and standard deviations (SD) for continuous variables. Analysis of factors associated with higher/improved students’ reading skills For binary outcome variables, we will use multivariable logistic regression to investigate KAP factors associated with increased odds of higher/improved EGRA scores on ORF and reading comprehension. The models will be adjusted for gender, disability and grade level and will be built using backward stepwise procedures for all KAP variables. All factors with a statistical significant association (α=0.05) with the outcomes will be retained in the final models. For continuous outcome variables, multiple linear regression (MLR) will be used to assess KAP factors correlated with higher EGRA scores on letter identification, syllable identification, familiar word reading and ORF subtasks. We will exclude reading comprehension from the MLR analysis, because it may be inappropriate for a bound and discrete outcome variable with only 6 values (i.e. 0-5 possible comprehension questions correct). This quantitative analysis will be used to inform the qualitative research design. The data will be analyzed using Stata v.15.1 (Stata Corp, College Station, TX, USA) 2.7 ETHICAL CONSIDERATIONS Proteknôn’s work is strongly rooted in the overriding principles of ‘do no harm’ and the ‘best interests of the child.’ We will therefore ensure complete compliance with any requirements in relation to informed consent and confidentiality policies and practices, particularly in relation to the safeguarding of children. After consulting with Save the Children and USAID, formal ethical clearance from the Rwanda National Ethics Committee (RNEC) will not be required for this assignment because this study is an evaluation of ongoing activities, and only direct stakeholders and beneficiaries of the program will be consulted as part of the evaluation. Similarly, clearance from the National Institute of Statistics Rwanda (NISR) was deemed unnecessary as we will be analyzing existing quantitative data sets and not collecting any primary quantitative data as part of this evaluation. In order to facilitate data collection at national and sub-national levels, USAID has agreed to provide us with an introductory letter. In addition, relevant representatives of both MINEDUC and REB will be informed by USAID of the proposed evaluation prior to any contact with local authorities. In addition, specific consideration will be given to the following ethical principles and in the following ways: ● Confidentiality of those participating in research ● Physical safeguards for those conducting research; ● Data protection and secure maintenance procedures for personal information; ● Age- and ability-appropriate consent and assent processes based on reasonable assumptions about comprehension of individuals involved in the research, including in the development of data collection tools; ● Child safeguarding, safety and privacy. We will attend to the ethical considerations above through the following activities: child safeguarding training provided by Save the Children, training in ethics and appropriate fieldwork methods provided by Proteknôn Team Lead and Field Coordinator; and the use of informed consent/assent in the field as per Save the Children protocols. In addition, Proteknôn will work in close collaboration with Save the Children to ensure that any participants (adult 105 or child) with disabilities are actively included FGDs. Any individual accommodations which may be necessary will therefore be identified and accounted for prior to the start of each session. Other key ethical, safeguarding and inclusion related considerations are further outlined in Annex II. These can be further refined in consultation with USAID and Save the Children to ensure appropriate contextualization and respect of internal organizational policies and best practices. All Research Assistants will participate in a two-day training. During this training, they will receive a detailed overview of Mureke Dusome and the aims of this evaluation. In addition, this training will review ethical data collection protocols, facilitation skills and the tools themselves. It will be an opportunity to highlight the need to ensure the participation of all FGD participants, regardless of gender and/or disability. The Team Lead and Field Coordinator will develop and deliver this training, as well as additional oversight during fieldwork for the purposes of quality assurance. Proteknôn’s internal safeguarding policy, as well as ethical research policy, will be adhered to throughout the course of this assignment, as well as any protocols or additional training required by USAID or Save the Children. In addition, prior to fieldwork, the Team Lead will complete a safety and security risk assessment to ensure that any identified risks are mitigated to the extent possible. 2.8 METHODOLOGICAL LIMITATIONS The strength of qualitative-focused work is that it enables us to examine questions such as ‘how’ and ‘why’ a program like Mureke Dusome operates as it does, because it draws from the expertise and insight of individuals and groups who have intimate knowledge and experience with the program. One challenge is that people experience the program differently, based on factors such as their position of authority (head teacher, government minister, education officer) and also how the program is shaped by factors such as gender, age, geographic location, and so on. While this can be read as a limitation, we also see this diversity of views as a strength in that it will allow us to unpack and report on the complexities of Mureke Dusome to offer targeted findings and recommendations for sustainability. Another challenge will be the possibility of a positive response bias. In other words, there may be little incentive for our respondents to speak critically of a program they have benefitted from in some capacity (e.g., services, employment, partnership, etc.). What we will do is be sure to stress that the overall aim of this evaluation is on understanding the factors that shaped the program’s effectiveness, with the goal of sustainability. Our study team will also stress the importance of informed consent and confidentiality. It will be difficult to generalize findings from a qualitative study across the country. However, by conducting in-depth work in 10 sectors across 5 geographically dispersed districts, and by developing a data collection plan that aims to arrive at conceptual saturation of key themes, we hope that this study can provide a good level of insight into Mureke Dusome that can be relevant to other parts of the country. Finally, the limited timeframe and parameters around budget will affect our fieldwork strategy. For example, we may be unable to interview a key informant if she or he is unavailable during our scheduled presence in the field. However, we are developing the evaluation in close consultation with USAID and Save the Children, and with their support, we are working diligently to identify key informants and schedule meetings in advance as a way to best ensure the evaluation stays on schedule. In these ways, we are confident that this our study design is capable of producing a high-quality evaluation that can contribute to broader discussions concerning project effectiveness and sustainability. 106 3. TIMELINE 3.1 PROPOSED WORK-PLAN The work-plan below is based on key activities and deliverables, start and end dates, as well as set turnaround times for feedback as per the final Purchase Order. Wherever applicable, a specific deadline has been included within the work-plan for ease of reference. KEY ACTIVITIES, DELIVERABLES AND MILESTONES SEPTEMBER OCTOBER NOVEMBER DECEMBER W K 1 W K 2 W K 3 W K 4 W K1 W K 2 W K 3 W K 4 W K 5 W K 1 W K 2 W K 3 W K4 W K 1 W K 2 W K3 INCEPTION ● Formal commencement of work by Proteknôn ● Request submitted regarding documents for review 2/9 ● Briefing and orientation with USAID COR 3/9 ● Desk-based review of project literature ● Revise evaluation work-plan ● Develop qualitative methodology, inclusive research protocol, and data collection tools (KII questionnaires, FGD guides etc) ● Establish list of key informants and site selection ● Clarify expectations related to quantitative data analysis and request relevant data sets accordingly ● Meeting with Save the Children for initial orientation and to refine evaluation design 11/ 09 ● Submit agreed components of evaluation design for initial review by USAID prior to deadline 11/ 09 ● Submit draft evaluation design, including work￾plan to USAID 16/ 9 107 ● Receive feedback from USAID 23/ 9 ● Integrate feedback received and re-submit revised evaluation design and work -plan for USAID approval 30/ 09 ● Agree work authorization protocol with Save the Children/USAID, including any authorization letters DATA COLLECTION AND ANALYSIS ● Translate data collection tools into Kinyarwanda ● Validation of Kinyarwanda version by Save the Children to ensure double -blind translation ● Pilot field -testing of proposed data collection tools, revise tools as appropriate to ensure quality control ● Recruit and train gender -balanced team of qualified local Research Assistants to work alongside Field Coordinator and Team Lead ● Coordinate and conduct sub -national fieldwork (KIIs/FGDs) [Note: relevant authorizations must have been received prior to start of fieldwork] ● Conduct national -level KIIs ● Analyze USAID -provided quantitative data sets ● Document key findings related to Research Question #2 [Note: data sets must be received prior to end of September] 108 ● Collation, processing and analysis of qualitative data: all transcriptions will be read by the Team Lead and Technical Specialist, with guidance for review provided by the Field Coordinator, in order to agree on analytical/coding categories REPORTING ● Develop, draft and submit preliminary findings in draft evaluation outline for review by USAID 4/11 ● USAID to provide feedback on draft evaluation outline 11/ 11 ● Finalize planning and circulate invitations to USAID/Save the Children and other relevant colleagues for preliminary presentation meeting (at least two weeks prior) ● Preliminary presentation to USAID/Save the Children to co -develop key recommendations for inclusion in evaluation report 11/ 11 ● Draft evaluation report to be submitted for review by USAID (using USAID guidelines on report format) 15/ 11 ● USAID to provide comments on draft evaluation report 29/ 11 ● Integrate feedback received and re -submit final evaluation report - including copy -editing, formatting and design - for USAID approval 13/ 12 109 DISSEMINATION ● Finalize planning and circulate invitations to USAID/Save the Children and key stakeholders (as per dissemination plan) for presentation of final report (at least two weeks prior) ● Final debrief with USAID, including dissemination presentation with key stakeholders ● Submit dataset to Development Data Library ● Submit final report to Development Experience Clearinghouse 19/ 12 110 3.2 FIELDWORK LOGISTICS The planning for field-based primary data collection, collation and transcription is outlined below. The dates are provisional. In the event we are able go to the field any earlier than planned, we will opt to do so wherever possible to allow for more time for data analysis and reporting. WEEK 1 Monday Tuesday Wednesday Thursday Friday NO OVERNIGHT NO OVERNIGHT Gasabo District Burera District TRANSCRIPTION DISTRICT LEVEL KII (JADF, DDE/DEO) DISTRICT LEVEL KII (JADF, DDE/DEO) Nduba Sector (GS Nduba) Ndera Sector (GS Ndera Catholique) Rugarama Sector (EP Gafumba) Kivuye Sector (EP Rugarambiro) LOCAL LEVEL KIIs & FGDs LOCAL LEVEL KIIs & FGDs WEEK 2 Monday Tuesday Wednesday Thursday Friday OVERNIGHT MUHANGA OVERNIGHT MUHANGA TRANSCRIPTION Ngororero District Ruhango District DISTRICT LEVEL KII (JADF, DDE/DEO) DISTRICT LEVEL KII (JADF, DDE/DEO) Ngororero Sector (Nyange Protestant) Nyange Sector (EP Nyange) Byimana Sector (GS Bukomero) Mbuye Sector (EP Kabuga) LOCAL LEVEL KIIs & FGDs LOCAL LEVEL KIIs & FGDs WEEK 3 Monday Tuesday Wednesday Thursday Friday OVERNIGHT MUHANGA TRANSCRIPTION Kirehe District DISTRICT LEVEL KII (JADF, DDE/DEO) Gahara Sector (GS Gahara) Nyarubuye Sector (GS Migongo) LOCAL LEVEL KIIs & FGDs 111 REQUESTED DOCUMENTS The list of documents below is requested for the purposes of conducting a desk-based review of relevant literature to inform the Mureke Dusome Final Evaluation. It is shared on the understanding that colleagues at USAID and/or Save the Children may potentially add or omit documents based on their knowledge of the project. MUREKE DUSOME CORE DOCUMENTS: ● Mureke Dusome project documents ● Mureke Dusome program description ● Mureke Dusome implementation plans ● Mureke Dusome Monitoring, Evaluation and Learning plan ● Annual/quarterly reports, including performance monitoring data ● Mureke Dusome/Save the Children baseline and midline KAP reports ● Mureke Dusome/Save the Children baseline and endline impact evaluation reports ● Mureke Dusome/Save the Children qualitative/process evaluation ● REB-approved training modules (developed as part of the project) ● Literacy Champions toolkit ● SBCC toolkit or key messages (developed as part of the project) ● Mureke Dusome sustainability plan or anything look at its sustainability or that of related programs NATIONAL EDUCATION DOCUMENTS AND OTHER RELEVANT POLICY MATERIALS: ● Most recent Education Sector Strategic Plan (ESSP) ● Other national planning documents regarding literacy ● Other national level outcomes data OTHER RELEVANT DOCUMENTS AND LITERATURE: ● Learning outcomes / literacy data (from REB, Literacy Boost, Mureke Dusome, and/or other) ● Relevant documentation from local partners, Umuhuza and Urunana ● Laterite School Dropout study ● Material relevant to Community-Based Literacy Volunteers and School General Assembly Committees ● District and sector level plans that feature Mureke Dusome activities ● Literacy Boost ● Advancing the Right to Read (e.g. learning documents) ● Research, Policy, and Institutions (political economy analyses) 112 KEY CONSIDERATIONS PRIOR TO AND DURING DATA COLLECTION PRIOR TO FIELDWORK: Research Assistants will ensure that the following key considerations are incorporated in the planning and preparation for the field-based data collection phase of the proposed baseline survey. ● Informed consent for adult participants ● Informed consent and assent for every child participant - see below for more guidelines ● Obtention of contact details for parents/guardians of every child participant, in case of emergency ● Knowledge of any particular needs of child participants, including those with disabilities. For example, any medication, mobility aids or translation needs that may be required. Research Assistants must be ready and willing to make necessary accommodations in order to ensure inclusivity ● Attendance sheets for each focus group discussion must be disaggregated for gender, age and disability ● Gender balance in the composition of the field team is maintained ● Familiarity with existing reporting and referral pathways within the field sites and schools in order to appropriately respond to any potential disclosures of abuse or distress ● The choice of venue has been checked to ensure safety, accessibility and suitability - including the availability of bathrooms ● Appropriate arrangements are in place for water and snacks - especially in the case of child participants ● Appropriate arrangements are in place for safe and appropriate transportation, wherever necessary, including reimbursement for actual costs and/or the presence of parents/guardians/agreed chaperones ● Confirmation that the proposed activity does not disrupt schooling, in the case of children, and is held at a reasonable time of day in order to avoid participants having to return home after dark INFORMED CONSENT AND ASSENT: Prior to participation in data collection activities, all adult and child participants must have given their informed consent in writing. For children specifically, both the child and their parents/guardians have given informed consent to participate . Consent means that the person has agreed to participate in the activity. It is informed because they understand what the objectivity of the activity, as well as, how the information they provide will be used. In the case of younger children, or children with developmental or cognitive impairments, who may not fully understand the purpose of the proposed activity, it is recommended that informed assent is sought instead. This informed assent may not be in writing, but rather can be provided in any way the child would like to communicate. It is understood that both informed consent and assent can be withdrawn at any time, including after the start of the activity. It is recommended that Research Assistants seek to confirm at the beginning of each session that participants understand why they are there, and their agreement to participate. Research Assistants can remind participants that their participation is entirely voluntary, and that they are under no obligation to continue if they would prefer not to for any reason. SPECIFIC CONSIDERATIONS FOR CHILDREN WITH DISABILITIES: In accordance with both the United Nations Convention on the Rights of the Child and the Convention on the Rights of People with Disabilities, the evaluation team will strive to ensure that all children with and without disabilities enjoy the same rights, including the right to participate. It is the responsibility of the Research Assistants to ensure that children are empowered, whatever their constraints, to express themselves and be heard during the proposed activities. 113 Considerations for different types of impairments include (but are not limited to): ● Use of written materials with larger fonts or translated into braille; Reading material out loud ● Use of a sign language interpreter, alongside pictorial or written instructions ● Giving children enough time to express themselves; Repeating back what you have heard to confirm accurate comprehension ● Providing instructions or information in simpler language and/or with the use of visual aids; Repeating yourself in order to be understood Save the Children has data on existing disabilities amongst members of reading clubs. They have agreed to share this information with us so that we can identify and make individual accommodations prior to any data collection session. This is outlined in greater detail above (please see section on ethical clearance). Similar accommodations could be made in order to promote the inclusion of adults with disabilities within the study design (parents with disabilities, for example). DURING FIELDWORK: What to do if a child is distressed Children may become distressed for a number of reasons. They may be overtired or frustrated, they may have suffered an injury, they may be angry or upset by something that happened during the focus group, they might have remembered a traumatic event that happened earlier. Whatever the reason, if a child becomes distressed, it is important to remain calm, and provide gentle, non-judgmental, non-punishing support. This may include giving the child additional privacy from the group, either during or after the activity for some one-to-one support. It may also mean allowing the child to take a break or stop the activity altogether. In every instance, the child should be allowed to decide what he or she needs, and to lead the conversation and next steps with age- and ability-appropriate support from the research team. What to do if a child discloses abuse If at any stage during the data collection a child reveals abuse that has happened to them or to someone they know, you should find an appropriate way to stop the conversation and take the child aside to a private space where you can talk without being seen or heard by others. Keep in mind that the child may not be aware that what they are describing is a form of abuse. It is important that the child knows that she or he has not said or done anything wrong. The identities of people who may have experienced or perpetrated sexual violence should always be kept confidential. This means not sharing the information with the group. Ensure the child feels safe and comfortable. Listen if the child wishes to speak, but do not ask any questions or try to get further information, and do not provide advice. It is not your role at this stage to investigate the disclosure further, beyond ensuring the immediate safety of the child in cases of imminent risk of harm. In adherence with the Proteknôn protocol and also in accordance with Save the Children’s Child Safeguarding Policy, reporting and referrals for further support will be made accordingly. 114 ANNEX III – QUALITATIVE DATA COLLECTION TOOLS KII Question Guide: National level key informants Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want 1. When it comes to improving children’s literacy, how would you characterize the successes of Save the Children/Umuhuza interventions in communities, particularly with regards to Mureke Dusome? (Probe: What would you say are some of the best practices that communities have adopted to support children’s literacy as a result of the program?) 2. As you know, Mureke Dusome sought to build capacity and strengthen the children’s book publishing industry. Please tell us what impact Mureke Dusome had in this area and also whether you think this impact can be sustained once Mureke Dusome has finished. 3. In your view, what were some of the successes for strengthening the capacity of schools and local government. What motivations or incentives were effective for local government, schools, and communities? 4. Which aspects of Mureke Dusome were less effective? Please tell us why. (Probe: planning, partnership, implementation) 5. [For Save the Children / USAID key informants] Now we’d like to talk with you about sustainability. How, if at all, did Mureke Dusome’s approach to sustainability change over the life of the project? (Probe: What effect did the cost extension have on this?) 6. Once Mureke Dusome ends, what needs to happen for the program interventions be sustained at national levels and in communities? (Probe: donors, private sector, policy, education sector strategies, performance contracts etc.) Are you aware of any efforts already being made in this area? 7. What needs to be done for Mureke Dusome to further build up skills and capacity among stakeholders? (Probe as relevant: book sector, national officials, sub-national leaders, village/school leaders, Literacy Champions) 8. When Mureke Dusome ends, what ideas do you have about how to sustain Mureke Dusome interventions as far as planning and budgeting are concerned? Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 115 KII Question Guide: National level book industry representatives Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want 1. What would you say are some of the best practices that communities use to support children’s literacy as a result of Mureke Dusome? Does the book industry have a role in these successes? If so, what is that role? 2. As you know, Mureke Dusome sought to build capacity and strengthen the publishing industry. What impact did Mureke Dusome have in this area? 3. What were the major successes in the work between Mureke Dusome and the book industry? What were the strategies for success? 4. What is the level of capacity of the book industry to supply high-quality children’s books to families in Rwanda? What needs to happen to sustain this effort? 5. When it comes to the book industry, what aspects of Mureke Dusome’s work were not as effective as they could have been? Please tell us why. 6. What will the close of the Mureke Dusome program mean for the children’s book sector in Rwanda, including actors like authors, illustrators, publishers, printers and book sellers? Will the close of the activity create any gaps for the book sector in terms of book sales? What specific ideas do you have to sustain and increase the capacity of the book industry in Kinyarwanda? (Probe: donors, policy, education sector strategies, private partners, etc.) Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 116 KII Question Guide: District level officials Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want ✓ Icebreaker question: Are you familiar with Mureke Dusome, Umuhuza or Urunana DC? Can you tell us about your involvement in the program? 1. When it comes to improving children’s literacy, how would you characterize the successes of Mureke Dusome interventions in communities? (Probe: What would you say are some of the best practices that communities have adopted to support children’s literacy as a result of the program?) 2. Now let’s turn to literacy behavior in communities. For head teachers, parents, and children, how did Mureke Dusome’s programs help to improve the environment for children to learn and practice reading? To what extent were factors like disability and gender addressed in the program? 3. To what extent are children able to access children’s books in your district? 4. What is your district doing to promote children's reading? What (if any) changes have you seen in schools and local government? In your view, what were some of the factors behind those changes? Will any changes be sustained after Mureke Dusome ends? (Probe: district planning/budgeting, Imihigo, relying on Literacy Champion volunteers?) 5. Which aspects of Mureke Dusome were less effective? Please tell us why. (Probe: planning, partnership, implementation) 6. How could Mureke Dusome be better embedded in local structures like, for example, Umugoroba w’Ababyeyi and Umuganda, FBOs, amongst others? 7. How could Mureke Dusome further build up skills and capacity among stakeholders in your district and communities to improve children’s literacy? (Probe as relevant: district/sector leaders, village/school leaders, Literacy Champions) 8. When Mureke Dusome ends, will the interventions it introduced also end? Do you have any other ideas about sustainability of the program? (Probe: Mureke Dusome programs cost money. Where will the financing come from for 1. Book replenishment, 2. LC training/motivation and 3. Transport for head teachers to meet with SGACs/Sector officials for learning, etc) Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 117 KII Question Guide: Local level officials (SEI, local leaders) Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want ✓ Icebreaker question: Are you familiar with Mureke Dusome, Umuhuza or Urunana DC? Can you tell us about your involvement in the program? 1. When it comes to improving children’s literacy, how would you characterize the successes of Mureke Dusome interventions in communities? (Probe: What would you say are some of the best practices that communities have adopted to support children’s literacy as a result of the program? 2. For head teachers, parents, and children, to what extent has literacy behavior changed and what would you attribute this to? To what extent were factors like disability and gender addressed in literacy programs? 3. What were some ways that local leaders government worked to improve children’s reading? What motivations or incentives were effective for local government, schools, and communities? (Probe: planning/budgeting, Imihigo, JADF, umuganda) 4. To what extent are children able to access children’s books in your area? 5. Which aspects of Mureke Dusome were less effective? Please tell us why. (Probe: planning, partnership, implementation) 6. Once Mureke Dusome ends, what needs to happen for the program interventions to continue within your sector and communities? (Probe: performance contracts, partnerships, etc.) Are you aware of any efforts already being made in this area? 7. How could Mureke Dusome be better embedded in local institutions like, for example, Umugoroba w’Ababyeyi and Umuganda, FBOs, and so on? 8. How could Mureke Dusome further build up skills and capacity among stakeholders in your sector or village? (Probe as relevant: district/sector leaders, village/school leaders, Literacy Champions) 9. When Mureke Dusome ends, will the interventions it introduced also end? Do you have any other ideas about sustainability of the program? (Probe: Mureke Dusome programs cost money. Where will the financing come from to support interventions in your district?) Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 118 KII Question Guide: Local level stakeholders (headteachers and Literacy Champion representatives) Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want ✓ Icebreaker question: Are you familiar with Mureke Dusome, Umuhuza or Urunana DC? Can you tell us about your involvement in the program? 1. When it comes to improving children’s literacy, how would you characterize the successes of Mureke Dusome interventions in your school or community? (Probe: What would you say are some of the best practices that have been adopted to support children’s literacy as a result of the program? 2. For head teachers, parents, and children, to what extent has literacy behavior changed and what would you attribute this to? To what extent were factors like disability and gender addressed in the program? 3. What were some strategies that schools did to improve children’s reading? (Probe: SGACs, head teachers, community outreach, imihigo) 4. Which aspects of Mureke Dusome were less effective? Please tell us why. (Probe: planning, partnership, implementation) 5. Once Mureke Dusome ends, what needs to happen for the program interventions to continue? (Probe: performance contracts, partnerships, Literacy Champions, etc.) Are you aware of any efforts already being made in this area? 6. How could the community literacy programs like Mureke Dusome be better embedded in local institutions like, for example, Umugoroba w’Ababyeyi and Umuganda, FBOs, and so on? 7. Now let’s think practically about continuing to promote children’s literacy in your community. Can you share any practical examples from this area for improving literacy that you think other schools and communities can learn from? (Probe as relevant: district/sector leaders, village/school leaders, Literacy Champions) 8. When Mureke Dusome ends, will the interventions it introduced also end? Do you have any other ideas about sustainability of the programs? (Probe: Mureke Dusome programs cost money. Where will the financing come from to support interventions in your area, like Literacy Champions, reading clubs, etc?) Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 119 FGD Question Guide: Parents/SGAC members Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want ✓ Icebreaker question: Are you familiar with Mureke Dusome, Umuhuza or Urunana DC? Can you tell us about your involvement in the program? 1. What is the difference between a community where Mureke Dusome was present compared to a community before Mureke Dusome was present? (Probe: availability of books, Literacy Champions, culture of reading, etc.) 2. What specific ways did Mureke Dusome’s programs work with the SGAC and school administration to improve children’s literacy? (Probe Literacy Champions, books, community-based work, etc.) Were any children excluded, for example because of their gender or disability? 3. To what extent are reading clubs helpful? How could they be improved? (Probe: How could they be improved to be more effective for girls? For boys?) 4. Which aspects of Mureke Dusome were less effective in your community or school? Please tell us why. (Probe: planning, partnership, implementation; exclusion due to gender/disability) 5. Once Mureke Dusome ends, will the program interventions also end? Or is there anything that can be done to continue the work in your community or school? (Probe: performance contracts, partnerships, LCs etc.) Are you aware of any efforts already being made in this area? 6. Do parents have the time and resources necessary to promote school-community partnerships? Is this different for mothers compared with fathers? Please help us understand. 7. Now let’s think practically about continuing to promote children’s literacy in your community. Can you share any practical examples from this school for improving literacy that you think other schools and communities can learn from? (Probe as relevant: Literacy Champions, SGAC members, local leaders) Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 120 FGD Question Guide: reading club members (children) Introduction: ✓ Who we are? ✓ What we hope to do together today and why? What we plan to do with the information we collect? ✓ Verbal confirmation of your agreement to participate voluntarily (informed consent/assent), usage of any photography or audio recordings & reminder that they can choose to pause or stop whenever they want ✓ Icebreaker question: What is your favorite thing to do at school? 1. As you know, we are here to talk about children’s literacy in your community. In your school or community, who encourages you and your colleagues to read and how do they do they do this? (Probe: Who are the people in school? Who are the people out of school?) 2. Now let’s talk about reading clubs in your school or community. What do you enjoy reading in the reading club? What do you think programs could do to make reading clubs even better? (Probe: How could they be made better for girls? Better for boys?) 3. Tell us about the kind of books you read. What kind of books do you read? Is there anything you would like to read more about? 4. We are interested in who participated in reading clubs. Were both boys and girls able to participate equally? Were there any children who were excluded from the club activities, for example, children with disabilities? 5. In your view, what could people in your community do to help girls and boys here to read more? Conclusion: ✓ Reminder of what happens next with the information we collected ✓ Thank you! 121 ANNEX IV SOURCES OF INFORMATION Government of Rwanda: ● Ministry of Education (MINEDUC): ● Minister of State for Primary and Secondary Education ● Public and Community Libraries Officer ● Rwanda Education Board (REB): ● Director of School Leadership & Management Unit ● School-Community Partnerships Coordinator (embedded Save the Children staff member and funded by USAID Mureke Dusome) ● Advisor to the Director General of REB (funded by USAID Soma Umenye) ● Ministry of Local Government (MINALOC): ● Director of Mentoring & Volunteerism, National Itorero Commission ● Ministry of Sports and Culture (MINISPOC): ● Acting Director of the National Library Key INGO, civil society, private sector and other stakeholders: ● Save the Children/Rwanda: ● Chief of Party Mureke Dusome ● Deputy Chief of Party Mureke Dusome ● USAID/Rwanda: ● Senior Education Specialist ● Umuhuza: ● Executive Director ● Community Engagement Coordinator ● Urunana DC: ● Director ● Chemonics: ● Chief of Party Soma Umenye ● Deputy Chief of Party for Field Operations Soma Umenye ● Additional members of Soma Rwanda Steering Committee: ● Education Specialist, UNICEF ● Director, Rwanda Bookmobile ● Education Programme Manager, World Vision ● Children’s book sector representatives: ● Chairperson, Rwanda Children’s Book Organization (RCBO) ● Director, Arise Education ● Director, Mudacumura Publishing House ● Director, Sankofa Creatives 122 ANNEX V – DISSEMINATION PLAN 123 ANNEX VII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Nidhi Kapur Title Team Lead Organization Proteknôn Consulting Group LLC Evaluation Position? Team Lead Evaluation Award Number (contract or other instrument) 720-696-19-P-00007 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mureke Dusome, implemented by Save the Children Rwanda I have real or potential conflicts of interest to disclose. Yes No XX If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or 124 disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date Jan 6 2020 125 Name Timothy P. Williams, PhD Title Education specialist, consultant Organization Proteknôn Consulting Group LLC Evaluation Position? Team member Evaluation Award Number (contract or other instrument) 720-696-19-P-00007 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mureke Dusome, implemented by Save the Children Rwanda I have real or potential conflicts of interest to disclose. Yes No XX If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or 126 disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12-11-19 127 Name Jeanine Mawazo Title Field Coordinator Organization Proteknôn Consulting Group LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) 720-696-19-P-00007 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mureke Dusome, implemented by Save the Children Rwanda I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 128 Signature Date 12 December 2019 129 Name Alphonse Nshimyiryo Title Quantitative Data Analyst Organization Proteknôn Consulting Group LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) 720-696-19-P-00007 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mureke Dusome, implemented by Save the Children Rwanda I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 130 Signature Date 12 December 2019 131 ANNEX VII: QUANTITATIVE ANALYSIS SUPPLEMENTARY TABLES 132 Table 1: Knowledge, attitudes and practices (KAP) about reading which are correlated with meeting the grade-level benchmark on reading comprehension P1 Students P2 Students P3 Students OR [95% CI] p-value OR [95% CI] p-value OR [95% CI] p-value Student's gender (Girls vs. boys) 1.26 [0.83, 1.92] 0.267 1.55 [1.07, 2.26] 0.021 0.81 [0.61, 1.07] 0.129 Student's age (in years) 1.17 [1.05, 1.32] 0.007 0.93 [0.82, 1.05] 0.227 1.05 [0.96, 1.14] 0.267 Variables on knowledge, attitudes and practices (KAP) about reading Take Kinyarwanda books home from the classroom and use them to practice reading 2.34* [1.09, 5.05] 0.030 1.17 [0.8, 1.7] 0.424 1.23 [0.83, 1.83] 0.299 A place in the community where children can go to read/borrow Kinyarwanda books 0.98 [0.61, 1.56] 0.921 1.61* [1.09, 2.36] 0.016 1.62* [1.11, 2.36] 0.012 Student lends or borrows a book or other learning materials to/from other students 0.93 [0.63, 1.38] 0.727 1.12 [0.77, 1.64] 0.549 1.19 [0.74, 1.92] 0.461 Student participates in reading activities after school 5.12* [2.22, 11.78] <0.001 2.40* [1.35, 4.28] 0.003 1.04 [0.67, 1.61] 0.865 Student enjoys reading in a group with other children 2.92 [0.36, 23.75] 0.314 3.35 [0.77, 14.49] 0.105 - - - Someone at home reads a story to the student 0.92 [0.47, 1.8] 0.806 0.96 [0.47, 1.96] 0.916 0.83 [0.56, 1.24] 0.367 Student reads to someone aloud at home 1.21 [0.74, 2] 0.447 2.14* [1.36, 3.36] 0.001 1.01 [0.54, 1.87] 0.986 Student reads independently at home 1.82 [0.84, 3.92] 0.127 1.97 [0.92, 4.21] 0.079 2.72* [1.42, 5.22] 0.003 Student has a favorite book 1.81 [0.97, 3.38] 0.062 2.16* [1.31, 3.57] 0.003 1.63 [0.9, 2.96] 0.105 Other possible confounding factors School location (Rural vs. Urban) 0.76 [0.49, 1.2] 0.235 0.78 [0.49, 1.23] 0.278 0.59 [0.36, 0.95] 0.029 Mobile/telephone at home 3.38 [1.75, 6.52] <0.001 0.98 [0.7, 1.39] 0.93 1.20 [0.86, 1.66] 0.277 Piped water at home 0.45 [0.27, 0.75] 0.002 1.27 [0.85, 1.9] 0.244 0.46 [0.20, 1.04] 0.061 Radio at home 0.69 [0.44, 1.09] 0.108 0.85 [0.63, 1.15] 0.289 1.21 [0.88, 1.67] 0.248 Television at home 1.32 [0.73, 2.4] 0.356 1.14 [0.73, 1.78] 0.557 1.24 [0.80, 1.91] 0.332 Bicycle at home 0.64 [0.43, 0.97] 0.034 0.6 [0.41, 0.89] 0.012 0.69 [0.49, 0.97] 0.033 Motorcycle at home 0.64 [0.29, 1.38] 0.252 1.47 [0.72, 2.98] 0.286 1.48 [0.83, 2.65] 0.182 133 A car at home 1.67 [0.79, 3.49] 0.175 0.66 [0.33, 1.32] 0.242 0.73 [0.33, 1.64] 0.445 Light at home: Electric light bulb 1.01 [0.58, 1.76] 0.958 1.81 [1.25, 2.63] 0.002 1.37 [0.98, 1.90] 0.063 Eat something before coming to school 0.92 [0.63, 1.34] 0.647 1.32 [0.81, 2.16] 0.264 1.18 [0.83, 1.67] 0.349 Drink something before coming to school 0.93 [0.63, 1.36] 0.704 1.03 [0.67, 1.58] 0.899 1.23 [0.93, 1.64] 0.142 Late to school yesterday 0.48 [0.23, 1.00] 0.050 0.64 [0.31, 1.30] 0.213 1.29 [0.75, 2.24] 0.354 Absent from school last week 0.53 [0.37, 0.75] <0.001 0.83 [0.62, 1.11] 0.217 0.6 [0.46, 0.78] <0.001 Notes: OR, Odds ratio; CI, Confidence Interval; *Statistically significant, p<0.05 134 Table 2: Knowledge, attitudes and practices (KAP) about reading which are correlated with meeting the grade-level benchmark on oral reading fluency (ORF) P1 Students P2 Students P3 Students OR [95% CI] p-value OR [95% CI] p-value OR [95% CI] p-value Student's gender (Girls vs. boys) 2.29 [1.49, 3.51] <0.001 1.76 [1.27, 2.43] 0.001 1.80 [0.8, 4.04] 0.156 Student's age (in years) 1.06 [0.97, 1.16] 0.199 1.09 [0.97, 1.23] 0.158 1.02 [0.8, 1.29] 0.889 Variables on knowledge, attitudes and practices (KAP) about reading Take Kinyarwanda books home from the classroom and use them to practice reading 1.76 [0.99, 3.11] 0.054 1.2 [0.8, 1.82] 0.38 0.99 [0.49, 1.98] 0.971 A place in the community where children can go to read/borrow Kinyarwanda books 1.27 [0.89, 1.80] 0.182 1.13 [0.73, 1.75] 0.589 2.49* [1.36, 4.57] 0.003 Student lends or borrows a book or other learning materials to/from other students 0.77 [0.49, 1.20] 0.240 1.05 [0.68, 1.64] 0.811 1.34 [0.56, 3.17] 0.506 Student participates in reading activities after school 3.93* [1.94, 7.99] <0.001 1.95 [0.99, 3.83] 0.053 1.14 [0.33, 3.89] 0.836 Student enjoys reading in a group with other children 3.00 [0.34, 26.45] 0.321 - - - - - - Someone at home reads a story to the student 0.60* [0.39, 0.92] 0.020 0.63 [0.33, 1.17] 0.143 1.23 [0.5, 3.04] 0.651 Student reads to someone aloud at home 1.55 [0.94, 2.56] 0.088 2.37* [1.35, 4.14] 0.003 2.07 [0.44, 9.75] 0.355 Student reads independently at home 2.81* [1.33, 5.94] 0.007 4.46* [1.47, 13.5] 0.008 2.5 [0.3, 20.83] 0.394 Student has a favorite book 1.40 [0.88, 2.24] 0.158 3.26* [1.45, 7.34] 0.005 5.06 [0.81, 31.39] 0.081 Other possible confounding factors School location (Rural vs. Urban) 0.71 [0.44, 1.15] 0.16 0.77 [0.45, 1.31] 0.330 0.68 [0.31, 1.47] 0.326 Mobile/telephone at home 1.87 [1.19, 2.95] 0.007 1.02 [0.65, 1.6] 0.932 0.81 [0.44, 1.5] 0.502 Piped water at home 0.59 [0.37, 0.95] 0.03 0.72 [0.41, 1.26] 0.247 1.54 [0.51, 4.65] 0.441 Radio at home 0.78 [0.59, 1.04] 0.093 0.86 [0.56, 1.3] 0.463 0.58 [0.28, 1.18] 0.129 Television at home 1.33 [0.8, 2.2] 0.265 0.93 [0.55, 1.55] 0.767 0.66 [0.22, 2.01] 0.465 Bicycle at home 0.50 [0.35, 0.71] <0.001 0.46 [0.23, 0.89] 0.022 0.71 [0.35, 1.45] 0.346 Motorcycle at home 0.52 [0.24, 1.11] 0.09 0.73 [0.37, 1.43] 0.362 0.48 [0.13, 1.82] 0.280 135 A car at home 1.88 [0.88, 4] 0.101 0.55 [0.15, 2.08] 0.378 0.29 [0.05, 1.66] 0.164 Light at home: Electric light bulb 1.65 [1.14, 2.39] 0.008 2.56 [1.49, 4.41] 0.001 1.83 [0.75, 4.44] 0.181 Eat something before coming to school 0.93 [0.5, 1.74] 0.83 0.72 [0.49, 1.05] 0.088 0.70 [0.31, 1.55] 0.371 Drink something before coming to school 0.84 [0.5, 1.42] 0.511 1.41 [0.93, 2.13] 0.101 1.55 [0.72, 3.34] 0.260 Late to school yesterday 0.75 [0.36, 1.55] 0.437 0.51 [0.27, 0.96] 0.037 2.31 [1.07, 4.99] 0.034 Absent from school last week 0.36 [0.2, 0.65] 0.001 0.78 [0.54, 1.11] 0.164 0.59 [0.3, 1.14] 0.116 Notes: OR, Odds ratio; CI, Confidence Interval; *Statistically significant, p<0.05 136 Table 3: Knowledge, attitude and practices (KAP) about reading which are associated with higher/improved scores on Letter Name Identification sub-task KAP Factors P1 Students P2 Students Coefficient (SE) [95% CI] p-value Coefficient (SE) [95% CI] p-value Student's gender (Girls vs. boys) 4.42 (1.21) [2.03, 6.8] <0.001 6.07 (1.86) [2.39, 9.75] 0.001 Student's age (in years) 1.51 (0.4) [0.73, 2.3] <0.001 -0.04 (0.39) [-0.81, 0.73] 0.926 Variables on knowledge, attitudes and practices (KAP) about reading Take Kinyarwanda books home from the classroom and use them to practice reading 3.36 (1.57)* [0.26, 6.47] 0.034 0.48 (1.9) [-3.27, 4.23] 0.800 A place in the community where children can go to read/borrow Kinyarwanda books -1.61 (1.46) [-4.5, 1.28] 0.273 1.78 (1.55) [-1.28, 4.84] 0.252 Student lends or borrows a book or other learning materials to/from other students 0.34 (1.26) [-2.15, 2.84] 0.786 -0.12 (1.77) [-3.62, 3.38] 0.947 Student participates in reading activities after school 6.38 (1.5)* [3.42, 9.33] <0.001 8.11 (1.96)* [4.24, 11.97] <0.001 Student enjoys reading in a group with other children -0.21 (3.19) [-6.52, 6.1] 0.948 3.73 (3.49) [-3.18, 10.64] 0.287 Someone at home reads a story to the student -0.16 (1.78) [-3.68, 3.36] 0.928 0.46 (3.47) [-6.4, 7.33] 0.895 Student reads to someone aloud at home 5.61 (1.57)* [2.51, 8.71] <0.001 6.36 (2.97)* [0.5, 12.22] 0.034 Student reads independently at home 4.55 (1.63)* [1.32, 7.78] 0.006 5.01 (2.77) [-0.46, 10.48] 0.072 Student has a favorite book 2.24 (1.23) [-0.18, 4.67] 0.070 5.49 (3.03) [-0.49, 11.47] 0.072 Other possible confounding factors School location (Rural vs. Urban) -1.93 (1.81) [-5.5, 1.64] 0.287 -2.76 (2.4) [-7.49, 1.98] 0.251 Mobile/telephone at home 4.4 (1.26) [1.91, 6.89] 0.001 3.17 (1.89) [-0.57, 6.92] 0.096 Piped water at home -1.88 (1.28) [-4.41, 0.65] 0.145 -1.43 (1.87) [-5.12, 2.26] 0.445 Radio at home -0.89 (1.28) [-3.43, 1.64] 0.487 -2.71 (1.56) [-5.78, 0.36] 0.084 Television at home 1.4 (1.64) [-1.84, 4.63] 0.394 -0.41 (2.25) [-4.84, 4.03] 0.857 Bicycle at home -4.34 (1.32) [-6.95, -1.74] 0.001 -4.87 (1.29) [-7.43, -2.31] <0.001 137 Motorcycle at home -1.59 (1.88) [-5.31, 2.13] 0.400 4.19 (2.9) [-1.53, 9.92] 0.150 A car at home 3.47 (1.99) [-0.47, 7.41] 0.084 -2.93 (3.06) [-8.97, 3.12] 0.340 Light at home: Electric light bulb 2.75 (1.38) [0.03, 5.48] 0.048 6.3 (1.91) [2.53, 10.07] 0.001 Eat something before coming to school -1.83 (1.21) [-4.22, 0.55] 0.131 4.82 (3.16) [-1.42, 11.05] 0.129 Drink something before coming to school -0.04 (1.08) [-2.17, 2.1] 0.974 1.26 (1.47) [-1.64, 4.17] 0.393 Late to school yesterday -1.18 (2.13) [-5.4, 3.03] 0.579 -6.4 (2.67) [-11.68, -1.12] 0.018 Absent from school last week -5.91 (1.46) [-8.8, -3.02] <0.001 -4.11 (1.74) [-7.54, -0.68] 0.019 Notes: SE, Standard error; CI, Confidence Interval; *Statistically significant, p<0.05 138 Table 4: Relative importance of predictors for higher/improved scores on Letter Name Identification sub-task Variables P1 Students P2 Students Standardized weight a Rank Standardized weight a Rank Student's gender 0.06 6 0.05 10 Student's age (in years) 0.05 8 0.00 21 Take Kinyarwanda books home from the classroom and use them to practice reading 0.10 4 0.07 5 A place in the community where children can go to read/borrow Kinyarwanda books 0.02 15 0.06 6 Student lends or borrows a book or other learning materials to/from other students 0.02 14 0.02 13 Student participates in reading activities after school 0.11 2 0.09 3 Student enjoys reading in a group with other children 0.00 19 0.01 17 Someone at home reads a story to the student 0.02 13 0.01 15 Student reads to someone aloud at home 0.20 1 0.21 1 Student reads independently at home 0.09 5 0.16 2 Student has a favorite book 0.10 3 0.05 8 School location (Rural vs. Urban) 0.04 10 0.05 7 Mobile/telephone at home 0.04 9 0.01 16 Piped water at home 0.00 21 0.00 22 Radio at home 0.00 23 0.00 23 Television at home 0.02 16 0.01 18 Bicycle at home 0.03 12 0.03 11 Motorcycle at home 0.01 18 0.00 24 A car at home 0.01 17 0.00 20 Light at home: Electric light bulb 0.03 11 0.05 9 Eat something before coming to school 0.00 22 0.00 19 139 Drink something before coming to school 0.00 20 0.02 12 Late to school yesterday 0.00 24 0.02 14 Absent from school last week 0.05 7 0.08 4 Note: a Standardized weight is the general dominance weight normed or standardized to be out of 100%. The standard weights might not add up to 1 due to rounding errors. 140 Table 5: Knowledge, attitude and practices (KAP) about reading which are associated with higher/improved scores on Syllable Sound Identification sub-task KAP Factors P1 Students P2 Students P3 Students Coefficient (SE) [95% CI] p-value Coefficient (SE) [95% CI] p-value Coefficient (SE) [95% CI] p-value Student's gender 3.93 (1.22) [1.52, 6.34] 0.002 2.97 (1.51) [-0.02, 5.97] 0.052 0.61 (1.56) [-2.47, 3.68] 0.696 Student's age (in years) 0.9 (0.31) [0.29, 1.51] 0.004 -0.69 (0.33) [-1.34, -0.04] 0.038 -0.18 (0.28) [-0.73, 0.38] 0.532 Variables on knowledge, attitudes and practices (KAP) about reading Take Kinyarwanda books home from the classroom and use them to practice reading 4.02 (1.9)* [0.28, 7.77] 0.036 0.44 (1.5) [-2.52, 3.4] 0.770 2.38 (1.41) [-0.4, 5.15] 0.093 A place in the community where children can go to read/borrow Kinyarwanda books -0.67 (1.26) [-3.16, 1.82] 0.596 2.88 (1.22)* [0.48, 5.29] 0.019 1.66 (1.06) [-0.44, 3.76] 0.120 Student lends or borrows a book or other learning materials to/from other students -1.03 (1.05) [-3.11, 1.05] 0.331 -0.35 (1.43) [-3.19, 2.48] 0.805 2.8 (1.73) [-0.62, 6.23] 0.107 Student participates in reading activities after school 5.48 (1.55)* [2.43, 8.54] 0.001 4.13 (1.32)* [1.51, 6.74] 0.002 1.61 (1.28) [-0.92, 4.13] 0.210 Student enjoys reading in a group with other children 0.61 (2.88) [-5.08, 6.31] 0.832 4.99 (3.32) [-1.58, 11.55] 0.136 7 (2.75)* [1.56, 12.44] 0.012 Someone at home reads a story to the student -1.64 (1.15) [-3.92, 0.63] 0.155 0.01 (2.96) [-5.83, 5.85] 0.997 -2.34 (1.66) [-5.62, 0.93] 0.160 Student reads to someone aloud at home 5.11 (1.34)* [2.45, 7.77] <0.001 5.53 (2.23)* [1.13, 9.93] 0.014 4.21 (2.27) [-0.27, 8.69] 0.065 Student reads independently at home 4.45 (1.16)* [2.17, 6.74] <0.001 5.85 (1.84) [2.21, 9.5] 0.002 7.33 (2.04)* [3.3, 11.35] <0.001 141 Student has a favorite book 1.99 (1.02) [-0.03, 4.01] 0.054 5.71 (2.03)* [1.7, 9.71] 0.006 3.64 (1.41)* [0.86, 6.42] 0.011 Other possible confounding factors School location (Rural vs. Urban) -1.27 (1.68) [-4.6, 2.06] 0.452 -2.96 (2.42) [-7.75, 1.83] 0.224 -4.52 (1.47) [-7.42, -1.63] 0.002 Mobile/telephone at home 4.83 (1.21) [2.44, 7.22] <0.001 0.87 (1.31) [-1.72, 3.47] 0.507 2.82 (1.53) [-0.21, 5.85] 0.068 Piped water at home -0.57 (1.45) [-3.44, 2.31] 0.697 -0.72 (1.52) [-3.72, 2.28] 0.637 -2.52 (3.32) [-9.09, 4.04] 0.448 Radio at home -1.78 (0.98) [-3.72, 0.15] 0.071 -2.11 (1.26) [-4.59, 0.38] 0.096 0.65 (1.05) [-1.42, 2.72] 0.538 Television at home 1.36 (1.55) [-1.7, 4.42] 0.382 -0.81 (2.06) [-4.89, 3.26] 0.694 -0.54 (1.73) [-3.95, 2.87] 0.755 Bicycle at home -3.58 (1.17) [-5.9, -1.27] 0.003 -4.25 (1.24) [-6.69, -1.81] 0.001 -1.02 (0.96) [-2.93, 0.88] 0.29 Motorcycle at home -3.67 (1.88) [-7.39, 0.04] 0.052 3.47 (2.35) [-1.17, 8.12] 0.141 -1.4 (2.05) [-5.45, 2.65] 0.496 A car at home 1.93 (2.44) [-2.9, 6.76] 0.432 -3.24 (2.58) [-8.34, 1.86] 0.211 -1.02 (2.77) [-6.49, 4.45] 0.714 Electric light bulb at home (Yes vs. No) 2.26 (1.46) [-0.63, 5.15] 0.124 5.94 (1.65) [2.67, 9.2] 0 4.59 (2.14) [0.36, 8.82] 0.034 Eat something before coming to school (Yes vs. No) -1.05 (1.2) [-3.42, 1.32] 0.381 3.67 (2.49) [-1.26, 8.59] 0.143 1.54 (1.07) [-0.58, 3.66] 0.154 Drink something before coming to school (Yes vs. No) -0.32 (1.4) [-3.08, 2.44] 0.82 0.95 (1.41) [-1.83, 3.73] 0.501 0.25 (0.96) [-1.65, 2.14] 0.797 Late to school yesterday -1.81 (2.07) [-5.91, 2.29] 0.384 -3.84 (1.91) [-7.62, -0.06] 0.047 0.68 (1.74) [-2.76, 4.12] 0.696 Absent from school last week (Yes vs. No) -5.1 (1.27) [-7.61, -2.59] 0 -3.09 (1.22) [-5.51, -0.67] 0.013 -3.19 (1.1) [-5.36, -1.02] 0.004 Notes: SE, Standard error; CI, Confidence Interval; *Statistically significant, p<0.05 142 Table 6: Relative importance of predictors for higher/improved scores on Syllable sound identification sub-task Variables P1 Students P2 Students P3 Students Standardized weight a Rank Standardized weight a Rank Standardized weight a Rank Student's gender 0.06 7 0.01 15 0.02 14 Student's age (in years) 0.03 12 0.01 18 0.03 11 Take Kinyarwanda books home from the classroom and use them to practice reading 0.10 3 0.07 5 0.04 7 A place in the community where children can go to read/borrow Kinyarwanda books 0.02 14 0.08 4 0.02 13 Student lends or borrows a book or other learning materials to/from other students 0.01 19 0.02 13 0.02 16 Student participates in reading activities after school 0.09 4 0.08 3 0.05 6 Student enjoys reading in a group with other children 0.00 20 0.01 17 0.02 12 Someone at home reads a story to the student 0.01 15 0.02 12 0.01 18 Student reads to someone aloud at home 0.19 1 0.21 1 0.25 1 Student reads independently at home 0.12 2 0.17 2 0.14 2 Student has a favorite book 0.09 5 0.07 6 0.10 3 School location (Rural vs. Urban) 0.03 13 0.06 7 0.07 5 Mobile/telephone at home 0.04 9 0.00 19 0.01 17 Piped water at home 0.00 22 0.00 20 0.03 9 Radio at home 0.00 21 0.00 23 0.00 24 Television at home 0.04 10 0.01 14 0.00 20 Bicycle at home 0.03 11 0.03 11 0.00 21 Motorcycle at home 0.01 17 0.00 24 0.00 22 A car at home 0.01 16 0.00 22 0.00 23 Light at home: Electric light bulb 0.04 8 0.06 8 0.04 8 Eat something before coming to school 0.00 24 0.00 21 0.03 10 Drink something before coming to school 0.01 18 0.04 10 0.01 19 Late to school yesterday 0.00 23 0.01 16 0.02 15 Absent from school last week 0.06 6 0.06 9 0.08 4 Note: a Standardized weight is the general dominance weight normed or standardized to be out of 100%. The standard weights might not add up to 1 due to rounding errors. 143 Table 7: Knowledge, attitude and practices (KAP) about reading which are associated with higher/improved scores on Familiar word reading sub-task KAP Factors P1 Students P2 Students P3 Students Coefficient (SE) [95% CI] p-value Coefficient (SE) [95% CI] p-value Coefficient (SE) [95% CI] p-value Student's gender 1.85 (0.66) [0.54, 3.16] 0.006 2.31 (0.83) [0.67, 3.95] 0.006 1.64 (0.88) [-0.1, 3.38] 0.065 Student's age (in years) 0.37 (0.15) [0.06, 0.68] 0.018 -0.25 (0.22) [-0.69, 0.18] 0.251 -0.06 (0.22) [-0.5, 0.38] 0.8 Variables on knowledge, attitudes and practices (KAP) about reading Take Kinyarwanda books home from the classroom and use them to practice reading 2.1 (0.86)* [0.4, 3.8] 0.016 0.22 (0.81) [-1.39, 1.83] 0.788 1.63 (1.03) [-0.4, 3.66] 0.114 A place in the community where children can go to read/borrow Kinyarwanda books -0.59 (0.75) [-2.07, 0.89] 0.431 1.49 (0.70)* [0.11, 2.88] 0.035 0.97 (0.73) [-0.48, 2.42] 0.188 Student lends or borrows a book or other learning materials to/from other students -0.65 (0.55) [-1.74, 0.44] 0.239 -0.26 (0.76) [-1.77, 1.25] 0.733 1.27 (0.97) [-0.64, 3.19] 0.192 Student participates in reading activities after school 2.81 (0.77)* [1.28, 4.33] <0.001 2.35 (0.73)* [0.91, 3.8] 0.002 2.04 (0.89)* [0.29, 3.79] 0.023 Student enjoys reading in a group with other children 0.06 (1.24) [-2.39, 2.51] 0.963 3.11 (1.72) [-0.29, 6.51] 0.073 4.95 (1.8)* [1.39, 8.5] 0.007 Someone at home reads a story to the student -0.70 (0.58) [-1.83, 0.44] 0.230 0.43 (1.83) [-3.18, 4.05] 0.813 -2.09 (1.48) [-5.02, 0.84] 0.161 Student reads to someone aloud at home 2.36 (0.57)* [1.23, 3.49] <0.001 3.57 (0.98)* [1.63, 5.51] <0.001 3.09 (1.64) [-0.16, 6.34] 0.062 Student reads independently at home 1.76 (0.56)* [0.65, 2.88] 0.002 3.08 (1.13)* [0.85, 5.32] 0.007 4.98 (1.43)* [2.16, 7.81] 0.001 144 Student has a favorite book 0.7 (0.48) [-0.25, 1.64] 0.146 3.16 (1.17)* [0.84, 5.48] 0.008 2.81 (1.06)* [0.72, 4.9] 0.009 Other possible confounding factors School location (Rural vs. Urban) -0.44 (0.75) [-1.91, 1.04] 0.562 -1.66 (1.2) [-4.04, 0.71] 0.168 -4.36 (1.26) [-6.86, -1.86] 0.001 Mobile/telephone at home 2.08 (0.62) [0.86, 3.31] 0.001 0.59 (0.81) [-1.01, 2.2] 0.466 1.8 (0.99) [-0.16, 3.75] 0.072 Piped water at home -0.61 (0.58) [-1.77, 0.54] 0.296 -0.3 (0.83) [-1.95, 1.34] 0.716 -3.38 (2.7) [-8.71, 1.95] 0.212 Radio at home -0.56 (0.43) [-1.41, 0.29] 0.196 -1.76 (0.75) [-3.24, -0.28] 0.02 0.63 (1) [-1.35, 2.6] 0.532 Television at home 0.82 (0.63) [-0.43, 2.07] 0.197 0.61 (0.99) [-1.35, 2.56] 0.54 0.55 (1.23) [-1.89, 2.99] 0.656 Bicycle at home -1.28 (0.6) [-2.46, -0.1] 0.034 -2.18 (0.65) [-3.47, -0.89] 0.001 -0.92 (0.76) [-2.41, 0.58] 0.228 Motorcycle at home -1.8 (0.84) [-3.45, -0.15] 0.033 1.74 (1.09) [-0.41, 3.89] 0.112 -0.4 (1.32) [-3.01, 2.2] 0.76 A car at home 1.95 (0.91) [0.15, 3.75] 0.034 -1.96 (1.27) [-4.48, 0.56] 0.126 -0.25 (2.21) [-4.62, 4.11] 0.909 Light at home: Electric light bulb 0.88 (0.72) [-0.54, 2.29] 0.222 2.95 (0.89) [1.19, 4.71] 0.001 3.49 (1.59) [0.35, 6.63] 0.03 Eat something before coming to school -0.59 (0.56) [-1.7, 0.52] 0.298 1.79 (1.36) [-0.91, 4.48] 0.192 1.08 (0.84) [-0.57, 2.73] 0.197 Drink something before coming to school -0.37 (0.7) [-1.75, 1] 0.591 0.63 (0.93) [-1.21, 2.48] 0.498 0.24 (0.73) [-1.2, 1.68] 0.742 Late to school yesterday -0.83 (0.92) [-2.66, 0.99] 0.369 -1.44 (0.98) [-3.36, 0.49] 0.143 0.06 (1.19) [-2.3, 2.41] 0.962 Absent from school last week -2.53 (0.62) [-3.76, -1.3] 0 -1.64 (0.67) [-2.97, -0.3] 0.016 -2.51 (0.79) [-4.07, -0.94] 0.002 Notes: SE, Standard error; CI, Confidence Interval; *Statistically significant, p<0.05 145 Table 8: Relative importance of predictors for higher/improved scores on Familiar Word Reading sub-task Variables P1 Students P2 Students P3 Students Standardized weight a Rank Standardized weight a Rank Standardiz ed weight a Rank Student's gender 0.04 8 0.02 12 0.04 8 Student's age (in years) 0.02 15 0.00 21 0.02 13 Take Kinyarwanda books home from the classroom and use them to practice reading 0.11 3 0.06 6 0.03 9 A place in the community where children can go to read/borrow Kinyarwanda books 0.02 14 0.07 4 0.02 12 Student lends or borrows a book or other learning materials to/from other students 0.01 18 0.02 14 0.01 17 Student participates in reading activities after school 0.10 4 0.08 3 0.06 6 Student enjoys reading in a group with other children 0.00 21 0.01 16 0.02 14 Someone at home reads a story to the student 0.01 16 0.02 15 0.01 20 Student reads to someone aloud at home 0.18 1 0.21 1 0.23 1 Student reads independently at home 0.11 2 0.16 2 0.12 2 Student has a favorite book 0.09 5 0.06 7 0.10 3 School location (Rural vs. Urban) 0.03 11 0.07 5 0.09 4 Mobile/telephone at home 0.04 9 0.00 19 0.01 19 Piped water at home 0.00 19 0.01 17 0.02 15 Radio at home 0.00 22 0.00 18 0.00 24 Television at home 0.04 10 0.02 13 0.01 18 Bicycle at home 0.02 12 0.03 11 0.01 21 Motorcycle at home 0.01 17 0.00 24 0.00 23 A car at home 0.02 13 0.00 23 0.00 22 Light at home: Electric light bulb 0.05 7 0.05 9 0.05 7 146 Eat something before coming to school 0.00 24 0.00 22 0.03 10 Drink something before coming to school 0.00 20 0.04 10 0.01 16 Late to school yesterday 0.00 23 0.00 20 0.02 11 Absent from school last week 0.08 6 0.06 8 0.09 5 Note: a Standardized weight is the general dominance weight normed or standardized to be out of 100%. The standard weights might not add up to 1 due to rounding errors. 147 148 U.S. Agency for International Development Rwanda 2657 Avenue de la Gendarmerie Kigali, Rwanda