Midterm Performance Evaluation: Soma Umenye Activity This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dexis Consulting Group: Kristin Rosekrans (Team Lead), Michael Tusiime (Education Specialist), Max Shanstrom (Evaluation Specialist), and Kathryn Norris (Research Assistant). MIDTERM PERFORMANCE EVALUATION: SOMA UMENYE ACTIVITY February 2020 Prepared for USAID Prepared by Dexis Consulting Group Research for Effective Education Programming – Africa (REEP-A) Contract No. AID-OAA-TO-16-00024 Cover photo: P1 students in the Kicukiro District, reading their newly distributed textbook during a break time; this is the first time in Rwanda that lower primary students have received Kinyarwanda textbooks at a ratio of 1:1. (Credit: Alain-Patrick Mwizerwa, USAID/Soma Umenye) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Abstract..................................................................................................................................................................... 1 Executive Summary ................................................................................................................................................ 2 Background............................................................................................................................................................... 9 Evaluation Purpose ...............................................................................................................................................11 Evaluation Questions...........................................................................................................................................11 Evaluation Methods..............................................................................................................................................13 Findings.................................................................................................................................................................... 19 EQ1. Implementing Partner.................................................................................................................19 EQ2: Implementation Challenges .......................................................................................................43 EQ3: Sustainability ................................................................................................................................. 47 EQ4: Collaboration, Learning, And Adaptation .............................................................................51 Conclusions ............................................................................................................................................................55 Recommendations................................................................................................................................................59 References.............................................................................................................................................................. 63 Annexes .................................................................................................................................................................. 64 Annex I: Evaluation Statement of Work ..........................................................................................64 Annex II: Evaluation Methods And Limitations...............................................................................65 Annex III: Data Collection Instruments............................................................................................68 Annex IV: Sources of Information ...................................................................................................112 Annex VI: Disclosure of Any Conflicts of Interest......................................................................115 LIST OF TABLES AND FIGURES TABLE 1. Soma Umenye’s Intermediate Results and Sub-Intermediate Results TABLE 2. Location of Schools, According to Province, District, and Sector TABLE 3. KIIs by Category TABLE 4. Focus Group Discussions (FGDs) by Category and Number of Participants FIGURE 1. Teachers’ Report of Who Coaches Them and Who Supervises Them FIGURE 2. Teachers’ Report of How Often They Assess Students ACRONYMS BLF Building Learning Foundations CBC Competence-Based Curriculum CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting COP Community of Practice CTLRD Curriculum, Teaching, and Learning Resources Department DA District Advisor DDE District Director of Education DEC Development Experience Clearinghouse DEO District Education Officer DFID United Kingdom’s Department for International Development DOS Dean of Studies EGR Early Grade Reading EGRA Early Grade Reading Assessment ESAD Examinations and Assessment Department ESSP Education Sector Strategic Plan FGD Focus Group Discussion GoR Government of Rwanda IP Implementing Partner IR Intermediate Result KII Key Informant Interview LARS Learning Assessments in Rwandan Schools LEARN Literacy Enhanced Across Rwanda Now LEGRA Local Early Grade Reading Assessment M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MINALOC Ministry of Local Government MINECOFIN Ministry of Economic Planning and Finance MINEDUC Ministry of Education MOU Memorandum of Understanding NGO Non-Governmental Organization NRTT National Reading Training Team P1 Primary Grade 1 P2 Primary Grade 2 P3 Primary Grade 3 PA Provincial Advisor REB Rwanda Education Board RENCP Rwanda Education NGO Coordination Platform SACMEQ Southern and Eastern Africa Consortium for Monitoring Educational Quality SBI School Based In-Service SBM School Based Management SEI Sector Education Inspector SEN Special Education Needs STTA Short-Term Technical Assistance SU Soma Umenye ToT Training of Trainers TTC Teacher Training College UNICEF United Nations Children’s Fund URCE University of Rwanda College of Education USAID United States Agency for International Development 1 ABSTRACT This formative, midterm performance evaluation of the USAID Rwanda Soma Umenye early grade reading activity examines the effectiveness of both mid-course corrections made following the initial year of the activity, and current activity implementation and performance. The six-member international and local evaluation team used a mixed-methods approach. Findings are that the materials provided by Soma Umenye are widely appreciated and used, with varying degrees of mastery. There are concerns about the sustainability and ownership of materials, which still need to be addressed. Sector Education Inspectors, Head Teachers, and Deans of Studies appear to be implementing their coaching duties, but still require additional support to be effective coaches to teachers. Soma Umenye has fostered a collaborative relationship with the Rwanda Education Board and Ministry of Education, and has built their staffs’ technical capacity. Due to its alignment with government priorities, there is strong demand for Soma Umenye at the central government and district levels. Soma Umenye’s District Advisors have been crucial for strengthening buy-in at the district level, but are stretched thin with their various responsibilities. Recommendations are to provide additional training for teachers on topics such as classroom management and remediation, enhance systematic documentation of lessons learned, afford more resources at the district level, take advantage of opportunities for collaboration with partners, and engage parents and the community in creating a culture of reading. 2 EXECUTIVE SUMMARY PROJECT BACKGROUND Soma Umenye (SU), “Read to Know”, is a $72.4 million, five-year activity (7/2016-7/2021) of the United States Agency for International Development (USAID) and the Rwanda Education Board (REB) under the Literacy Enhanced Across Rwanda Now (LEARN) project, implemented by Chemonics International (contract number AID-OAA-I-14-00055, Task Order No. AID-696-TO-16- 00001). SU aims to improve the Kinyarwanda language reading skills of at least one million children in public schools by the end of Primary Grade 3 (P3). The two main Intermediate Results (IRs) are: 1) classroom instruction in early grade reading improved, and 2) systemic capacity for early grade instruction improved. EVALUATION PURPOSE AND QUESTIONS The SU evaluation is a formative midterm performance evaluation aimed at assessing both the effectiveness of mid-course corrections made following the initial year of the activity, as well as the effectiveness of current activity implementation and performance. It seeks to inform USAID Rwanda, the implementing partner (IP), and the Government of Rwanda (GoR) of SU’s strengths, weaknesses, and potential ways to improve its effectiveness. The main evaluation questions (EQs) are: 1) How has the implementing partner and its team performed against the terms of the contract? 2) What are some challenges or obstacles to implementing the activity, and how could those challenges be addressed? 3) What efforts are being made within SU to ensure sustainability, and how could SU more effectively work toward sustainability? and 4) What evidence, if any, indicates that the activity has employed adaptive management approaches? Does existing evidence indicate the effectiveness of any changes made? EVALUATION METHODS AND LIMITATIONS The SU evaluation utilized a mixed-methods approach, employing primary data from key informant interviews (KIIs), focus group discussions (FGDs), a teacher questionnaire, and classroom observations. Secondary data were gathered through a comprehensive desk review of relevant documents. The evaluation team engaged a range of stakeholders, including personnel from the GoR, the IP, USAID Rwanda, and local partners and activity beneficiaries. In total, data were collected from 93 informants through KIIs, FGDs, and a questionnaire. Additionally, observational data from 11 classes were collected using a classroom observation protocol. Limitations include potential sampling and response bias, unavailability of some informants, and lack of audio recording of two focus groups and two interviews. FINDINGS AND CONCLUSIONS EQ1. IMPLEMENTING PARTNER: HOW HAS THE IMPLEMENTING PARTNER AND ITS TEAM PERFORMED AGAINST THE TERMS OF THE CONTRACT? IR 1: CLASSROOM INSTRUCTION IN EARLY GRADE READING IMPROVED The SU team worked in conjunction with REB to develop teaching and learning materials for P1-P3, including teacher guides, student textbooks, and read-aloud storybooks. SU also worked with REB to procure 65 titles of supplementary readers and to develop 24 decodable readers for P1-P3. Teacher guides and textbooks for P1were distributed in all schools and supplementary readers and bookshelves in 500 schools. P2 and P3 teacher guides, student textbooks, and read-aloud storybooks were procured in early 2019. Due to delays in 3 production, these had only been distributed to some schools by the time of data collection. All stakeholders expressed that availability of teaching and learning materials was an important contribution to students’ ability to learn Kinyarwanda. It is the first time in Rwanda that every P1 student in public and government-subsidized schools will have their own Kinyarwanda textbook. However, not all students in the observed classes had their own textbook. When they did have their own textbook, students were more engaged, and parents said that bringing textbooks home supported their children in learning. Teachers were grateful for and are becoming accustomed to using the teacher guide, though some stakeholders voiced concern about the frequency with which teachers use the guide. District and school stakeholders perceived that teachers are using and becoming comfortable with the guide. Several stakeholders said some teachers may be reluctant to use the guide due to receiving the training prior to having the guides, while other teachers want to utilize old practices. Teachers varied in their mastery of the guide; approximately one-third of the observed teachers constantly read from the guide, and thus did not sufficiently or adequately engage students. All P1-P3 teachers in Rwanda have been trained in how to use the materials and instructional approaches for literacy development, which, overall, is having a positive effect. Stakeholders from all levels recognized improvement in teaching practices and student learning; all teachers in the sample reported using new instructional strategies. However, most of the observed teachers used some practices that were not evidence-based, which can undermine learning. Such practices include over-emphasizing the “I do” component of the “I do, we do, you do” strategy; using recall questions, chorus answers, and frequent recitation; and not allocating sufficient time to independent work, such as writing. Head teachers and Deans of Studies (DOSs) seemed to be fulfilling their roles as coaches, although they are not all equipped to recognize evidence-based versus non￾evidence-based practices. Most consulted teachers appreciated the coaching received by head teachers and DOSs, although several said that they need more preparation to coach effectively. District Education Officers (DEOs) and Sector Education Inspectors (SEIs) appreciated their new coaching role as well, even though the role of SEIs is still not clear. Furthermore, no mechanism is in place for systematic coaching follow-up by head teachers, SEIs, and District Advisors (DAs). SU has supported REB in developing formative assessment tools and guidelines that teachers are starting to put into practice. However, they are still not doing this systematically and deliberately. While DEOs, SEIs, head teachers, and teachers all understood the importance of formative assessment, they generally agreed that it is still a nascent practice and that more preparation is needed to be able to analyze assessment results to inform practice. Large class sizes (i.e., students per class) make this challenging. However, some teachers and head teachers gave examples of having used formative assessment to identify struggling students. IR 2: SYSTEMIC CAPACITY FOR EARLY GRADE READING INSTRUCTION IMPROVED SU has participated in fora and disseminated messages through communication networks to promote EGR; however, there are untapped advocacy opportunities. SEIs, head teachers, and teachers noted existing mechanisms as opportunities for promoting EGR messages, such as Umuganda (the monthly community service day), where key messages are communicated from the central level to communities nationwide. SU has supported REB in developing P1-P3 Kinyarwanda benchmarks and student performance standards, which stakeholders acknowledged are important tools. Although the process was delayed, the Kinyarwanda benchmarks have been approved, and will be the first 4 introduced to schools. However, stakeholders at the school level were not yet aware of the benchmarks and their impending application. Additionally, SU has worked with REB to develop teacher performance standards, which are the key competencies and behaviors that lower primary Kinyarwanda teachers must display for learners to achieve their benchmarks. SU has built a collaborative learning agenda with key stakeholders including REB and is working to execute the necessary research. Stakeholders emphasized the importance of widely disseminating findings from the studies to inform policy and practice. In 2018, SU developed a learning agenda in consultation with USAID and other key partners to enhance ownership by all stakeholders involved. Several REB officials noted that results of the recently￾completed Instructional Time Study should be shared with teachers so as to make them aware of classroom time misuse and strategies to better engage students. GoR stakeholders also noted that information about SU’s teaching approach should be more widely available. SU has helped strengthen REB’s assessment systems through capacity building and support for the Early Grade Reading Assessment (EGRA), Local Early Grade Reading Assessment (LEGRA), and formative assessment. However, stakeholder sharing and analysis of data will likely be a challenge. Performance benchmarks and formative assessment are new in Rwanda, and district and school actors are not yet equipped to analyze and use the results. School and district actors are also not accustomed to sharing results, which may foster concerns about results being used punitively (as noted by several DEOs and SEIs). REB officials and other stakeholders appreciated that SU has contributed to Teacher Training Colleges’ (TTC) uptake of evidence-based EGR instructional approaches. SU has been working with TTC tutors and has trained final year students so that they will be prepared for the SU approach when entering the teaching force. All stakeholders consulted on TTCs expressed that these have a great potential for the sustainability of the SU intervention. However, several stakeholders emphasized the importance of formalizing the partnership between SU and the University of Rwanda College of Education (URCE) to ensure that roles and responsibilities are clear. CONTRIBUTION TO GOR PRIORITIES AND LEARNING OUTCOMES The SU activity is directly supporting GoR priorities as defined in the Education Sector Strategy Plan (ESSP) and GoR policy documents. Some of the most important priorities highlighted by stakeholders are improving Kinyarwanda literacy in lower primary, developing teaching and learning materials, training teachers, strengthening assessment, and supporting the implementation of the new curriculum. SU has incorporated gender sensitivity into its materials and trainings, but has provided limited guidance on supporting students with disabilities. Teachers have begun to implement some of the strategies such as equal participation for female students and use of gender inclusive language, however many teachers lack the ability to identify and support students with special needs beyond visibly evident disabilities. ORGANIZATIONAL STRUCTURE AND STAFF DAs have been vital to building capacity of district and sector GoR partners, however they are over-committed to other responsibilities. DAs lack sufficient time to provide adequate follow-up and support to head teachers and teachers. DAs overwhelmingly agreed that they do not have sufficient time for follow-up, and that 10 visits per month were insufficient to provide the appropriate level of support and coaching to each school. This sentiment was echoed by several head teachers. 5 EQ2: IMPLEMENTATION CHALLENGES: WHAT ARE SOME CHALLENGES OR OBSTACLES TO IMPLEMENTING THE ACTIVITY? HOW COULD THOSE CHALLENGES BE ADDRESSED? The most consistently raised challenge was the delay in production and distribution of SU materials to schools. This prevented teachers from delivering lessons with the necessary supporting materials, and many teachers reported that they forgot what they learned regarding their use. Most school staff members said that they do not know how to follow up on material delivery to ensure provision of adequate numbers of copies or to replace missing or lost materials. There was a lack of clarity about ownership of textbooks and concern that students will not take care of them, or forget them at home; parental involvement is key for this. All stakeholders raised concerns about the durability of the materials, especially student’s textbooks. Additionally, stakeholders expressed uncertainty about whether textbooks belonged to schools or to students, sometimes prompting school staff to keep children from taking books home for fear of damage or loss. Engaging parents was seen as essential to ensuring that SU materials are well maintained, and that students have support in developing literacy skills. SU has encountered several barriers regarding greater GoR uptake of evidence-based approaches. These have been on some technical decisions, which they have overcome through dialogue and negotiation. One example of this was the sequence of introducing letters in the student textbooks and teacher guides. Another was including comprehension in the LEGRA. In both cases, REB officials and other education stakeholders insisted on the importance of context. SU accepted their position, which REB appreciated. EQ3: SUSTAINABILITY: WHAT EFFORTS ARE BEING MADE WITHIN SU TO ENSURE SUSTAINABILITY AND HOW COULD SU MORE EFFECTIVELY WORK TOWARD SUSTAINABILITY? Implementers defined sustainability as behavior change, particularly at the school level, and the Communities of Practice (COPs) were considered a viable mechanism to sustain good EGR instructional practices. Head teachers as well as district and sector level officials cited the COPs as providing sustainability of SU trainings. Although COPs are in early stages of implementation, respondents observed that teachers and head teachers are already enthusiastic about sharing best practices. District and school-level stakeholders expressed that after SU support ends, COPs could provide a way for teachers to continue to build on what they learned, and to share challenges and best practices. Incorporating these activities and other SU-related interventions into performance contracts (Imihigos) at central, district, and school levels were also seen by most stakeholders as a means toward sustainability. EQ4: COLLABORATION, LEARNING, AND ADAPTATION: WHAT EVIDENCE, IF ANY, INDICATES THAT THE ACTIVITY HAS EMPLOYED ADAPTIVE MANAGEMENT APPROACHES? DOES EXISTING EVIDENCE INDICATE THE EFFECTIVENESS OF ANY CHANGES MADE? SU’s successes were attributable, in large part, to the strong collaboration that SU has established with their GoR counterparts in REB and at district offices. All REB officials emphasized that working in tandem on SU-supported interventions has built REB capacity and helped to accomplish their goals. This was also seen at the district level, through DAs being highly integrated into the district offices. According to district staff, this has helped better tailor support to teachers. 6 SU has been adaptive in responding to changes in context or operational challenges, however internally there are opportunities to establish a greater culture of learning. SU has adapted to GoR requests, such as tailoring interventions to their technical vision of literacy development and assessment. However, SU does not have a mechanism to systematically consolidate and analyze data and information from central and local levels to promote learning and adaptation. SU has numerous opportunities to build upon its current collaboration with partners, including Building Learning Foundations (BLF) and Mureke Dusome. While coordination with donors and partners has proven successful in avoiding overlap and duplication, not enough has been done to develop plans to collaborate on shared priorities and objectives. Working toward parental engagement to support children’s literacy is one key area for collaboration that will also lay groundwork for the sustainability of SU. RECOMMENDATIONS IMPLEMENTING PARTNER’S PROGRESS TOWARD GOALS IR 1. CLASSROOM INSTRUCTION IN EARLY GRADE READING IMPROVED 1. DAs should work through SEIs and the School General Assembly Committees (SGACs) to clarify student and parent responsibilities around materials ownership and to encourage their maintenance. 2. Rapid distribution of the remaining materials should be prioritized and a refresher training provided so that teachers can effectively use all materials. 3. Refresher training, or a session in each training, should be provided that addresses the overuse of the “I do, we do, you do” strategy; clarifying its scaffolding utility and the importance of leaving adequate time for the “you do” component, including time for students to both read and write independently. The training could also include measures for effective time use that were identified in the Instructional Time Study, particularly for large classes. 4. In training for teachers and head teachers, SU should address the practices that teachers use that are not conducive to student learning and literacy development, such as recall questions, chorus responses, and excessive repetition and recitation. 5. To address teachers’ need for frequent and high-quality coaching, it will be important to explore the provision of tablets to head teachers and teachers so as to make available videos of model teaching, contextualized to Rwandan classrooms, and other useful functions, which have been shown to be a cost effective way to improve literacy learning.1 6. SU should support REB to further define the role of SEIs and their specific responsibilities both in general, and as they relate to SU implementation. 7. DEOs, DAs, and SEIs should guide head teachers to hold regular COPs with teachers, and should encourage peer mentoring; these sessions should be documented to share with the SEI, DEO, and DA, such that subsequent COPs can build on each other. 1. Piper, B., Jepkemei, E., Kwayumba, D., & Kibukho, K., “Kenya’s ICT Policy in Practice: The Effectiveness of Tablets and E-readers in Improving Student Outcomes,” FIRE vol. 2 issue 1(2015): http://dx.doi.org/10.18275/fire201502011025. 7 8. SU should ensure that coaching provided by head teachers, SEIs, DEOs, and DAs, including COPs, is systematically documented and shared in order to inform subsequent coaching and training. 9. Refresher training should be provided to DEOs, SEIs, head teachers, and teachers on how to conduct formative assessment, including documenting and analyzing results to inform practice. IR 2: SYSTEMIC CAPACITY FOR EARLY GRADE READING INSTRUCTION IMPROVED 10. SU should establish a partnership with the Ministry of Local Government (MINALOC) so that important EGR messages are incorporated in the messages communicated during Umuganda. SU should also explore collaboration with Mureke Dusome’s Umuganda literacy activities. 11. Once benchmarks have been used to categorize students by literacy abilities, SU should ensure that DAs, DEOs, SEIs, and head teachers systematically monitor the quality of remediation that teachers provide to struggling learners. 12. SU should support URCE to document evidence-based EGR best practices, and develop policy briefs targeting audiences at national and local levels to influence policy and practice. 13. SU should execute sensitization campaigns to reassure district and school actors that LEGRAs will not be used as a punitive measure, but instead will serve as a tool to monitor and improve student learning. 14. SU should establish a MOU or similar agreement with URCE and TTCs so that both parties are formally committed to their tasks to mainstream EGR activities into pre-service teacher education curriculum. CONTRIBUTION TO GOR PRIORITIES AND LEARNING OUTCOMES 15. SU should document the approach taken to support GoR priorities and build GoR capacity in order to share these best practices nationally and internationally. INEQUITIES BASED ON GENDER AND DISABILITIES 16. SU should produce or adapt specific materials for addressing special needs in the classroom, and go beyond merely creating awareness. 17. Partnerships with the Department of Special Needs Education at URCE should be explored, in coordination with BLF if feasible, since BLF has special needs coordinators in their pilot districts. ORGANIZATIONAL STRUCTURE AND STAFF 18. For the remainder of the activity, SU should aim to ensure that DEOs, SEIs, and head teachers are equipped to sustain SU interventions. To do this, SU should consider providing staff to support DAs in capacity-building of DEOs/SEIs to conduct coaching visits, hold COPs, and collect monitoring and evaluation (M&E) data within the district. 19. SU should consider hiring a Monitoring, Evaluation and Learning (MEL)/Collaborating, Learning, and Adapting (CLA) specialist for each province to systematically capture and document school, sector, and district M&E data, assessment results, and coaching/COP data. IMPLEMENTATION CHALLENGES 20. Specific training and guidelines should be provided to head teachers and teachers on how to engage students despite large class sizes (e.g., student group work, peer teaching, and independent work). 21. Time should be included in teacher training for teachers to discuss their concerns about instructional strategies, such as covering all of the content in 40 minutes. 8 22. Condensed training on Sunday should be provided for the Seventh Day Adventist head teachers, teachers, DEOs, and SEIs. 23. DAs should support SEIs and head teachers to provide greater sensitization for parents through the SGACs, including provision of a guidance pamphlet about how to support children’s literacy development and care for textbooks. SU should explore coordinating these efforts with Mureke Dusome. 24. SU staff should draft templates for letters or memos that DEOs can use to discourage transfers of Kinyarwanda teachers trained by SU. SUSTAINABILITY 25. SU should work to incorporate SU goals and activities, including early grade literacy rates and COPs, into the strategic plans and performance contracts (Imihigos) of sectors, districts, and schools. 26. SU should prioritize COPs as a means of making SU interventions sustainable, thus ensuring capacity to identify evidence-based EGR practices, document and analyze shortcomings, and use M&E and assessment data to inform subsequent action. 27. A year before the SU activity concludes, SU should support REB in planning for the reprinting and redistribution of P1-3 teaching and learning materials. COLLABORATION, LEARNING, AND ADAPTATION 28. USAID and SU should harness potential opportunities with Mureke Dusome for joint advocacy of common causes, such as awareness for inclusive education and parental support for literacy. 29. SU should identify specific points of convergence with BLF, such as assessment, COPs, school leadership, and SEI’s roles, including establishing action plans for joint interventions and collaboration. 30. SU should hire a CLA and/or knowledge management specialist at a central level to improve collection and management of internal information, and to consolidate and synthesize information from all levels to inform adaptation through regular learning reviews. 31. To inform adaptation and build capacity to use data to inform practice, the Provincial Advisor, or a provincial CLA specialist, should lead provincial and district-level CLA meetings at the end of each term to analyze M&E data with district and sector staff and head teachers. 9 BACKGROUND Over the past two decades, Rwanda has made progress toward becoming a middle-income, knowledge￾based economy. The Rwanda Vision 2020 articulates the Government of Rwanda’s (GoR) ambitious development agenda and its strategy to achieve these goals by fostering and sustaining economic, social, and political advances. The Vision emphasizes human capital formation as one of the six strategy pillars, with particular focus upon enhanced education service delivery. Efforts to improve the education system in Rwanda are oriented toward ensuring universal education for all, improving education quality, and expanding technical and vocational training. Rwanda has made significant progress in reducing illiteracy, and ensuring universal access to basic education. Rwanda has achieved near universal access to primary education, increasing the primary net enrollment rate from approximately 83 percent in 2002 to 96 percent in 2016.2 According to an assessment of progress toward the Millennium Development Goals, as highlighted in a recent World Bank report, Rwanda is one of the four best-performing countries in Africa in increasing primary net enrollment, a key indicator of achieving universal primary education.3 Despite this progress, improvements to education quality have not kept pace with the GoR’s expansion of access, and learning levels remain low. For example, the 2018 Human Capital Index ranked Rwanda 142 out of 157 countries in the amount of human capital a Rwandan child can expect to accrue given the country’s current health and education outcomes.4 As reflected in this ranking, learning outcomes remain low, leaving youth largely unprepared for the formal labor market. Rwanda’s 2015-2020 Country Development Cooperation Strategy (CDCS) highlights similar challenges to quality education, including an inadequate supply of quality teachers and a high ratio of pupils to teachers. One key component for improving education quality in Rwanda is enhancing early grade reading (EGR) and literacy outcomes. Reading and literacy are foundational skills that contribute to learning in all academic subjects. Thus, investing in EGR and literacy outcomes can broadly improve educational outcomes. In support of these priorities, the United States Agency for International Development (USAID) is partnering with the GoR to improve EGR outcomes through strategic investment in the sector. ACTIVITY DESCRIPTION Soma Umenye (SU), “Read to Know”, is a $72.4 million, five-year activity (7/2016-7/2021) of USAID and the Rwanda Education Board (REB) under the Literacy Enhanced Across Rwanda Now (LEARN) project, which is implemented by Chemonics International (contract number AID-OAA-I-14-00055, Task Order No. AID-696-TO-16-00001). SU aims to improve the Kinyarwanda language reading skills of at least one million children in public and government-funded Rwandan schools by the end of their third year of primary school. Specifically, SU seeks to ensure that at least 70 percent of Primary Grade 1 (P1), Primary 2. UNESCO Institute for Statistics, Education: Net enrolment rate by level of education: accessed May 9, 2018, http://data.uis.unesco.org/#. 3. World Bank, Global Monitoring Report 2015/2016 (Washington, DC: World Bank Group, 2016): 252, https://www.worldbank.org/en/publication/global-monitoring-report. 4. World Bank, The Human Capital Project (Washington, DC: World Bank Group, 2018): 32, https://openknowledge.worldbank.org/handle/10986/30498. 10 Grade 2 (P2), and Primary Grade 3 (P3) students are able to read grade-level text with fluency and comprehension. To improve reading and literacy in the early grades, SU aims to strengthen the five “Ts”: time, text, teaching technique, test, and tongue. SU’s theory of change is that improvements to the five “Ts” and the promotion of leadership in early grade literacy from the school level to the national level will ensure that students’ literacy skills are improved. To reach SU’s goal of improved literacy outcomes for all children in early grades, – inclusive of boys and girls, children living in extreme poverty, and children with disabilities – the initiative centers on two Intermediate Results (IR). IR1 focuses on improving classroom instruction in the early grades through school-level interventions, including provision of reading materials, training and coaching, and the implementation and use of student assessment results. IR 2 focuses on improving the capacity of the Rwandan education system to support EGR instruction. This includes strengthening advocacy for EGR, implementing EGR policy and curricula that are supported by evidence, and supporting the development of assessment systems for EGR. (See Table 1 for a summary the sub-IRs that fall under the two Intermediate Results). Table 1. Soma Umenye’s IRs and sub-IRs CDCS Development Objective: Increased opportunities for Rwandan children and youth to succeed in schooling and the modern workplace Program Purpose: Improved literacy outcomes for children in early grades IR 1: Classroom instruction in early grade reading improved IR 2: Systemic capacity for early grade reading instruction improved Sub-IR 1.1: Evidence-based, gender- sensitive early grade reading materials available and used Sub-IR 2.1: National advocacy mechanisms for early grade reading interventions strengthened Sub-IR 1.2: Teachers’ use of evidence- based, gender-sensitive instructional practices in early grade reading increased Sub-IR 2.2: Student and teacher performance standards and benchmarks for early grade reading applied Sub-IR 1.3: Capacity of head and mentor teachers to coach and supervise early grade reading instruction strengthened Sub-IR 2.3: Research-based policies and curricula in support of early grade reading instruction implemented Sub-IR 1.4: Schools’ and teachers’ use of student assessment results improved Sub-IR 2.4: Early grade reading assessment systems strengthened 11 CDCS Development Objective: Increased opportunities for Rwandan children and youth to succeed in schooling and the modern workplace Program Purpose: Improved literacy outcomes for children in early grades IR 1: Classroom instruction in early grade reading improved IR 2: Systemic capacity for early grade reading instruction improved Sub-IR 2.5: Capacity of Teacher Training Colleges (TTCs) to prepare effective early grade reading teachers improved Cross-Cutting: gender and inclusion of students with special needs, ICT The initial year of SU was completed in July 2017. During this year, SU encountered performance and management issues regarding alignment with GoR priorities, and effective coordination with the education sector. To address these challenges, an adjusted approach was adopted for Year 2 (July 2017- July 2018). The updated approach was developed through close collaboration between USAID Rwanda and the implementing partner (IP) over several months. Efforts at course correction included a major revision to the roll-out plan in addition to repositioning the activity to better align with and work through GoR systems. EVALUATION PURPOSE The SU evaluation is a formative midline performance evaluation aimed at assessing the effectiveness of mid-course corrections made following the initial year of the activity as well as the effectiveness of current activity implementation and performance. The analysis presented below provides USAID Rwanda with detailed information about activity challenges and successes, and identifies areas for improvement in activity effectiveness and sustainability. In particular, this evaluation details the performance of the IP against the terms of the contract and the extent to which they were able to adapt to changes in the local context to achieve high-level results. The evaluation also focuses on challenges encountered in working effectively with the GoR and other systems in Rwanda, and stakeholder perception of IP performance, as this can contribute to buy-in and ownership of the activity. The report aims to inform USAID Rwanda, the IP, and the GoR on SU’s strengths and weakness, as well as potential ways of improving the effectiveness and sustainability of the activity. EVALUATION QUESTIONS The evaluation was guided by four primary questions, developed by USAID Rwanda in collaboration with the evaluation team, which address the key evaluation aims identified above. These questions are: 12 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? a. What progress has Soma Umenye made toward the stated goals and objectives of the contract? i. What progress has been made toward each of the sub-IRs (as shown in Table 1)? ii. Is Soma Umenye viewed as positively contributing to GoR priorities and learning outcomes among GoR stakeholders? iii. To what extent has Soma Umenye addressed inequities based on gender and students and teachers with disabilities? iv. Is Soma Umenye making effective use of its organizational structure and staff? b. Have teachers incorporated best practices from Soma Umenye’s trainings into their classrooms? If not, what have been the key impediments to doing so? How has Soma Umenye responded to those challenges? i. How do teachers perceive Soma Umenye interventions? Do teachers perceive that the Soma Umenye interventions have had an impact on their teaching techniques and their students’ learning? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? a. National level challenges i. Has Soma Umenye addressed barriers to greater GoR uptake of evidence-based approaches? b. District, sector, school-level challenges 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? a. How does Soma Umenye define sustainability? b. Is there currently demonstrable ownership and demand for Soma Umenye interventions? How could Soma Umenye further promote ownership and demand? (covering the following stakeholders) i. GoR officials ii. Local administrators iii. Teachers iv. Local NGO partners c. What is the current level of skills and capacity among stakeholders to sustain Soma Umenye interventions? How could Soma Umenye further build up skills and capacity among stakeholders? (covering the following stakeholders) i. GoR officials ii. Local administrators 1. What role are Soma Umenye district advisors playing in developing the skills and capacity of local administrators? iii. Teachers iv. Local NGO partners d. Given available resources of GoR and local organizations, which Soma Umenye interventions are most sustainable? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Does existing evidence indicate the effectiveness of any changes made? 13 a. Is Soma Umenye effectively collaborating with USAID, GoR, and other development partners in Rwanda? How could their collaboration be improved? b. How is Soma Umenye using assessments (both formative and summative) to spur adaptation? c. What opportunities exist to strengthen collaborating, learning, and adapting (CLA) practices in the activity? EVALUATION METHODS The SU evaluation utilized a mixed-methods approach, employing a combination of primary and secondary data. Primary data collection consisted of key informant interviews (KIIs), focus group discussions (FGDs), a teacher questionnaire, and classroom observations. Secondary data were gathered through a comprehensive desk review of relevant documents. The evaluation team engaged a range of stakeholders, including personnel from the GoR, the IP, USAID Rwanda, and local partners and activity beneficiaries. A four-member team of both international and local personnel conducted the evaluation. The team was comprised of a team leader, a local education specialist, an evaluation specialist, and a research assistant. Additionally, two local education specialists assisted in data collection at the district and school levels. The team leader was responsible for the evaluation design, elaboration of data collection tools, leading the data collection and analysis, and writing the evaluation report. Other team members participated in providing feedback and assistance in elaborating the tools, collecting data, analyzing data, and writing the report. The local education specialists conducted all of the focus groups and interviews in Kinyarwanda, including teacher focus groups, most of the interviews with head teachers, and some interviews with sector and district staff members. Data sources, data collection methods, sampling and site selection, and limitations are detailed below. (See Annex II for a Getting to Answers Matrix that provides a summary of these components). SAMPLING AND SITE SELECTION The evaluation utilized purposive sampling for KIIs, FGDs, classroom observation, and the teacher questionnaire. The team selected key informants from USAID, the IP, and partner organizations, as well as beneficiaries from the GoR (mainly REB), and other donors. USAID Rwanda provided an initial list of recommended informants, which was supplemented by input from the IP. Individuals from REB were selected based upon their knowledge of, and involvement with SU. Members of other donor and partner organizations were selected based on their roles in supporting primary education in Rwanda. The evaluation team also used purposive sampling to select provinces, districts, sectors, and schools where classroom observation would take place. After consultation with USAID Rwanda, the evaluation team collected data from 10 schools in four provinces and six districts that were representative of different regions, and urban and rural areas of the country. Six districts within the four provinces were selected for data collection – one in the Northern Province, one in the Western Province, and two each in the Southern and Kigali Provinces. The Provincial 14 Advisors (PAs) provided data on districts with higher overall literacy performance and those with lower performance, which the evaluation team used to select a higher and a lower performing district in Western and Kigali Provinces. Accessibility also influenced the choice of districts. For example, in situations where two districts had relatively similar literacy performance levels, the more accessible district was selected. For each of the selected districts, District Advisors (DAs) provided a list of schools with information based on the criteria (e.g., urban/rural, performance level). DAs were asked to select two schools in each district, but from different sectors – one school that was relatively more successful in implementing SU teaching practices, and one school that was facing challenges in implementing these practices. In only one case were two schools selected within the same sector (See Table 2 for a list of the province, district, and sectors). Table 2. Location of schools, according to province, district, and sector Province Districts Sector No. of Schools Western Province Nyabihu Mukamira 2 Kigali Province Kicukiro Gikondo 1 Nyarugenge Nyamirambo 1 Southern Province Huye Tumba Rusatira 1 1 Gisagara Save Ndora 1 1 Northern Province Musanze Kabaya Busogo 1 1 In each of the 10 schools, all P1-3 Kinyarwanda teachers were selected to participate in a FGD. The head teachers and/or Deans of Studies (DOSs) were selected for KIIs, as well. One class was observed in each school, with the exception of the first school, where both a P1 and P3 class were observed.5 In most schools, the team observed a P1 class because this was the only class to have both the teacher guide and student textbooks; several P2 classes had these materials and were also observed. During school visits, parents were selected for interviews based on availability. The Sector Education Inspector (SEI) in each sector included in the sample was selected for a KII, as well as the DA and the District Director of Education (DDEs) or District Education Officer (DEO) in the selected districts. Additionally, one PA was selected to interview, based on availability. One Vice Mayor was also selected in light of her involvement with SU. DATA COLLECTION The evaluation team collected data with a desk review, KIIs, FGDs, teacher questionnaires, and classroom observations. In total, data were collected from 93 informants through KIIs, FGDs, and a questionnaire. Additionally, the evaluation team collected observational data from 11 classes using a classroom observation protocol. These aspects of data collection are detailed below. DESK REVIEW 5. Though the evaluation team members had worked together to help ensure inter-rater reliability with the classroom observation protocols, the first school provided the opportunity to test this in the field. Therefore, two team members observed one class and three members the other, to then compare ratings and interpretations. 15 The team conducted a comprehensive desk review prior to fieldwork in order to inform in-country data collection, and fine-tune data collection tools and the sampling process. The team obtained documents relevant to the SU activity from USAID Rwanda, Chemonics, and USAID’s Development Experience Clearinghouse (DEC). Documents included work plans, quarterly and annual reports, assessments and evaluations, monitoring, evaluation and learning (MEL) plans and reports, and Ministry of Education (MINEDUC) reports. See Annex IV for a complete list of documents. KIIS AND FGDS The evaluation team conducted initial KIIs and FGDs with personnel from USAID Rwanda and the IP to better understand the SU activity and its stage of implementation for each expected result. Interviews were then conducted with key members of REB, such as the Director General, Deputy Director General, and directors and staff from departments involved in SU’s implementation (e.g. teacher training, curriculum, and assessment). The team conducted follow-up interviews with personnel from the IP, including the Chief of Party, activity managers, and relevant technical specialists. The team also conducted interviews with the main donors supporting primary education in Rwanda, such as the United Kingdom’s Department for International Development (DFID), United Nations Children’s Fund (UNICEF), the World Bank, and implementers of USAID’s activity to boost communities’ support for reading, Mureke Dusome (See Annex IV for a complete list of key informants by organization/institution and title). The team used semi-structured interview guides tailored to the different informant groups in order to conduct all KIIs and FGDs (See Annex III for the data collection tools). The team interviewed the DDE/DEO and DA in the selected districts as part of the sample, and the SEIs in the selected sectors.6 FGDs were conducted with the teachers of P1-P3 in each of the 10 schools selected for data collection. In seven schools there were three lower primary teachers (one for each grade, for P1-P3), while in three schools there were two teachers per grade, resulting in a six-person FGD. All FGDs were conducted in Kinyarwanda. The local education specialist piloted the teacher FGD guides prior to the arrival of the international team members and the team adapted the guide accordingly. Additionally, input was provided by the IP and further adaptations were made. The head teacher or DOS was interviewed in each of the 10 schools. These interviews were also conducted in Kinyarwanda, with the exception of two schools wherein interviews were conducted in English. Additionally, at each school an attempt was made to conduct a FGD with parents of students in P1-P3.7 Ultimately, it was only possible to interview one or two parents in four of the ten schools. (See Tables 3 and 4 for a summary of KIIs and FGDs). The team recorded all interviews and focus groups after requesting consent from the respondents. Two team members were present to conduct each interview, allowing for one person to take detailed notes on a computer, while the other person conducted the interview. The duration of the KIIs ranged between 45 and 60 minutes, and FGDs varied from 60 to 90 minutes in duration. 6. An attempt was made to interview all of the DAs, DDEs/DEOs, and SEIs in the selected districts and sectors, but this was not always possible given their lack of availability on the days of the school-level data collection. 7. Though the head teacher at all schools requested the presence of a small group of parents, only between one and three were typically present. In six of the schools, it wasn’t possible to speak with parents. 16 Table 3. KIIs by category Respondent Category No. of KIIs USAID 2 Other Donors and Partners 5 Government of Rwanda (REB and MINEDUC) 7 Soma Umenye Staff 5 SEIs 5 DEOs and DDEs 4 District Advisors 4 Others (e.g. Vice Mayor, PA, Publisher, URCE) 5 Head Teachers/DOSs 10 Parents of Students 2 TOTAL 49 Table 4. FGDs by category and number of participants Respondent Category No. of FGDs Total No. of Participants Teachers 10 40 Parents 2 4 Total 12 44 TEACHER QUESTIONNAIRE 17 Prior to the FGD, teachers were administered a structured questionnaire that consisted of multiple choice and short open-answer questions about teaching practices, use of materials, and other aspects related to SU implementation. Evaluation team members were present to answer teachers’ questions and explain any part of the questionnaire that required clarification. The IP provided feedback on the questionnaire prior to school-level data collection, and adjustments were made accordingly. As in the FGDs, typically three teachers from P1-P3 responded to the questionnaires, while in three schools, teacher questionnaires had six teacher respondents. In one of these a DOS also completed a questionnaire. Thus, a total of 40 teacher questionnaires were collected from the 10 schools. CLASSROOM OBSERVATION The team used a classroom observation protocol to observe P1-P3 classes in each of the selected schools. Protocol content was designed to capture SU’s target teaching practices and also included some elements such as student engagement, class management, and gender equity. The local education specialist piloted the protocol and the team made further adjustments. Adjustments were also made based on IP feedback. For inter-rater reliability, evaluation team members observed the first two classes together in order to compare choices made for recording the items that were more subject to interpretation. Two team members co-observed all other classes, which further helped assure inter￾rater reliability.8 In all cases, a Kinyarwanda lesson was observed in its entirety of 40 minutes. In total, the team observed nine P1 classes, one P2, and one P3 class.9 DATA ANALYSIS Data analysis consisted of data consolidation and organization, coding, identification of themes and patterns, and tabulation. The process for analyzing the different types of data is explained below. DESK REVIEW Evaluation team members used a matrix to conduct the desk review. The matrix was organized by evaluation questions and sub-questions, as well as SU’s expected results. Team members extracted and organized information from the desk review documents within the matrix. This was completed prior to in-country data collection. The desk review was instrumental to determining the stage of completion for the SU activity’s expected results and deliverables in addition to challenges and solutions to such implementation challenges. It evidenced information gaps, and informed all subsequent data collection. KIIS AND FGDS All detailed notes from KIIs and FGDs were uploaded to a Google drive where the team could access them to perform an initial (in-country) data analysis. The team used data consolidation matrices based upon the evaluation questions and SU expected results to extract information. The evaluation team used the data matrices for the first stage analysis, which helped to identify initial findings and recommendations for the in-country evaluation briefing with USAID and the IP. 8. There were six team members for data collection, three of whom were Kinyarwanda speakers. For the five days of data collection, the team divided into two groups to cover both schools in each district simultaneously. 9. The evaluation team decided to prioritize observing P1 over P2 and P3 because in Grade 1 teachers had teacher guides and student textbooks and had been implementing the SU practices the longest. 18 Audios from KIIs and FGDs were transcribed, and transcriptions in Kinyarwanda were translated to English. Using MAXQDA, the evaluation team created a coding structure based upon evaluation questions, sub-questions, and themes relevant to the evaluation. Segments from KII and FGD transcripts and notes were subsequently coded. Unanticipated themes identified in the data were added to the existing codes. Following completion of the qualitative coding process, convergence and divergence determinations were used to analyze qualitative data. Convergence and divergence analyses compare responses to common questions and themes across respondent groups to identify domains where responses align or differ. Using this method, the evaluation team identified common and recurring themes, and deviations between stakeholder perceptions. Comparing themes across respondents also helped to identify the frequency of expressed beliefs and opinions. TEACHER QUESTIONNAIRE Data from teacher questionnaires were consolidated in a Microsoft Excel file that allowed for registering each response by teacher. Although questionnaire information was collected in Kinyarwanda, it was entered into an Excel file in English. Response frequencies were tabulated, and the consolidated data illustrated patterns and trends in teachers’ perceptions of SU implementation. CLASSROOM OBSERVATION The team consolidated the classroom observation data in an Excel file that captured the quantitative and qualitative data for each classroom observation. Frequencies of classroom practices and student behavior were quantified, and patterns and trends in the multiple choice and short answer responses were identified. Following the analysis of both primary and secondary data, the evaluation team triangulated results based on data sources, geographic locations, and methods, to create a complete set of evaluation findings. LIMITATIONS There were several limitations that were mitigated to the extent possible throughout the evaluation process. First, as with any qualitative research using non-random samples, there was potential for sampling bias. For example, parents interviewed were those who responded to the head teachers’ request to meet the evaluation team. These parents may have been more involved in their students’ schooling than other parents. Despite the use of purposive and convenience sampling, the evaluation team aimed to ensure that respondents reflected a variety of perspectives, demographics, and geography, particularly among activity beneficiaries. Second, there was potential for response bias. Although interview questions were not highly sensitive, respondents may have had some incentive to not be fully forthcoming about their perspectives. The evaluation team attempted to mitigate this concern in the question design process, and by sharing strategies for how to frame questions to elicit more genuine responses. Similarly, in classroom observations, teachers may have demonstrated teaching practices that were not entirely typical because they knew about the team’s visit ahead of time, and were aware of our presence during the observation. There was no way to entirely mitigate this limitation. 19 There were several limitations in sampling due to the unavailability of some informants. Unavailable parties included parents in some schools, two of the DAs, DEOs/DDEs from two districts, and SEIs from four of the nine sectors. The team was able to mitigate this limitation at least to some extent, by interviewing other informants who could also speak to the issues from similar perspectives. Finally, teacher FGDs on the first day were not recorded due to a lack of audio recording devices.10 However, team members took detailed notes of these FGDs and added pertinent information to the data consolidation matrices. Additionally, one SEI, one Vice Mayor, and one government official did not want to be recorded. To ensure that all data were captured, detailed notes were taken, which were then added to the data collection matrices. FINDINGS EQ1. IMPLEMENTING PARTNER: HOW HAS THE IMPLEMENTING PARTNER AND ITS TEAM PERFORMED AGAINST THE TERMS OF THE CONTRACT? EQ1.A. WHAT PROGRESS HAS SU MADE TOWARD THE STATED GOALS AND OBJECTIVES OF THE CONTRACT? EQ1.A.I. WHAT PROGRESS HAS BEEN MADE TOWARD EACH OF THE SUB-IRS? IR 1. CLASSROOM INSTRUCTION IN EARLY GRADE READING IMPROVED SUB-IR 1.1. AVAILABILITY AND USAGE OF EGR MATERIALS The SU team has worked in conjunction with REB’s Curriculum, Teaching, and Learning Resources Department (CTLRD) to develop teaching and learning materials for P1-P3, including teacher guides, student textbooks, and read-aloud storybooks. Additionally, the SU team worked with REB staff to review and revise the leveling guidelines for supplementary readers. The revised guidelines were subsequently shared with publishing companies. SU procured three copies each of 65 titles for P1-P3 for a total of 390,000 supplementary readers. SU also developed 24 decodable readers for P1-P3. At the time of the evaluation, P1 teacher guides and textbooks had been distributed in all schools, and supplementary readers and bookshelves for P1 were provided to 500 schools. P2 and P3 teacher guides, student textbooks, and read-aloud storybooks were also developed by SU in collaboration with CTLRD; both were procured in early 2019 but experienced delays in production. At the time of the evaluation, student textbooks for P2 and P3 had been distributed in some districts, and P2 and P3 teacher guides were provided to the 1,300 teachers who attended catch-up training. Although alphabet charts and pocket boards have been produced for P1 classrooms, these were only distributed to the first 30 percent of schools that received P1 teacher guides and textbooks. Interview data revealed that, despite challenges encountered in the development and printing of EGR material, there has been steady progress in distribution of EGR teaching and learning materials in schools. P1 student textbooks and teacher guides were found in all schools in the sample. Some schools in the Southern Province had received P3 student textbooks, and most of the schools visited had 10. All team members downloaded a voice recording application to their phones to ensure that there would not be a lack of a recording device for the rest of the data collection period. 20 received P3 teacher guides. P2 textbooks were delivered to several schools during the data collection period. Across the board, stakeholders expressed that availability of teaching and learning materials was an important contribution toward students’ ability to learn Kinyarwanda, and improving literacy skills. The SU activity represents the first time in Rwanda that every P1 student in public and government-subsidized schools has their own Kinyarwanda textbook. Informants from all categories cite the importance of each student having his or her own textbook. As an SEI stated, “Soma Umenye has provided students textbooks throughout the country. By doing so, even the poor children have equality of educational opportunity, which was not the case in the past because textbooks are expensive and could only be accessed by parents who had money.” A REB official emphasized that REB and SU had a joint commitment to ensure that every child had their own book, and further expressed that this would be a considerable achievement in the context of the new competence-based curriculum, which requires learners to have materials in order to practice independent learning. All parents interviewed discussed the importance of their children having Kinyarwanda books to read and use for homework. They all stated that their children bring their textbooks home, and most said that their child’s ability to read had improved.11 As one parent relayed, “[the children] like them very much. When the child is at home, there is nothing else they look at.” Another parent discussed the importance of the textbooks for children’s learning and the need to take care of these books: “See for example, when I am at home, I see my daughter holding the book, she is so happy, and holding a notebook writing. They may keep them but rather we, as parents, should be the ones to make sure the books are fine as a gift to our children that make learning Kinyarwanda better; it is our responsibility as parents to take care of the books given to our children, we should make sure that if children come back home, they come back with the books and we keep them from any spoiling and take good care of them.” Teachers and other stakeholders noted the absence of readers and other materials in schools. Many head teachers and teachers discussed materials that they did not yet have, but needed in order to effectively teach. Several teachers expressed that it was difficult to execute a lesson if not equipped with the right materials to follow the lesson plan (such as supplementary readers). In the teacher questionnaire, 35 of the 40 respondents said that they lack sufficient materials for effective teaching. In observing classes, it was apparent that the lack of materials, such as read-aloud storybooks, made teaching the lesson challenging. An official from MINEDUC emphasized this, claiming “it took eight months to get the bookshelves out to classrooms. Now, they are there, but they are empty. They need to speed up the distribution process.” Students that had their own textbooks were more engaged in class. In 6 of the 11 classes observed, each student had her or his own book. In the other five classes (all P1 classes), students shared books, with three to four students using each textbook. In one class, two children shared one book. In classes in which students each had their own book, students tended to be more engaged, even if the classes were the same size. For example, in two classes of 55 students, the class in which students had their own textbooks had higher average student engagement throughout the lesson. 11. It is important to note that the four parents interviewed said that their children bring home textbooks, but they did not say that this was every day. Also, as noted in the limitations in the methods section, these parents were likely more involved in their children’s schooling and thus may not be representative of the majority of parents. 21 It is unclear how often students each use a textbook during class or how often they bring textbooks home, even in schools that have received teacher guides and textbooks. According to the evaluation data, students were encouraged to take their textbooks home in only 4 of the 11 observed classes. All parents interviewed indicated that their children brought the textbook home regularly, but did not specify the exact frequency. Teacher questionnaire responses indicated that teachers assign reading homework (from the student textbooks) two to five times per week. In schools that had textbooks delivered but in which not all students had their own textbook in class, head teachers were unclear as to why each student did not have their own book to use. Two head teachers remarked that some teachers might be withholding a portion of the textbooks from use in order to preserve them. Teachers were grateful for the teacher guides and the lessons they provided. As one head teacher expressed, “This teacher guide is like the head for putting the programs of Soma Umenye into practice.” Several teachers expressed that the teacher guide contained comprehensive information on how to instruct EGR: “When we have this teacher's guide, we prepare our lesson well and deliver it as it is prepared. Before Soma Umenye trains us, there is a book we used; while reading the sequence of lessons, we found terms like phonology and phonetics and we said ‘what is the importance of these things they inserted in teacher's guide?’ and we failed to know how to deliver it.” Several teachers mentioned the adjustment required to hold the teacher guide while teaching. As one stated, “We have the mindset that no teacher should teach with a document in their hands, and we are obliged to take it in our hands every time; but we will get used to it slowly by slowly as we do it every day; and we teach with it.” A DA and several other interviewees confirmed that they see the teachers trying to use the guides, and are becoming accustomed to the guides. The majority of head teachers reported that they see teachers trying to follow the steps in the teacher guides, and they think the guides help them learn. They seem to trust that if the teachers follow the teacher guides, students will learn. As this head teacher stated: “…the teacher’s guide published by the project Soma Umenye containing the methodology, how lessons follow one another considering the first term, in fact, all the units of each lesson and the methodological approach on how a lesson must be delivered. Once the teacher possesses the book and conforms to it, the objective of the project Soma Umenye will be achieved.” While teachers appeared to be using the teacher guide, mastery of the materials vary. All teachers who have teacher guides noted that they use the guide for every lesson they teach. Those that do not have guides noted that they use the student textbooks (if they had them). In the classroom observation, it was apparent that some teachers were more comfortable than others with the guide. About one-third of the teachers observed read from the guide constantly and did not pause to engage with the students (as will be discussed in the following section on instructional practices). Additionally, some teachers reported feeling unprepared to use supplementary materials, such as the readers. As one teacher stated, “There are additional small books, and there are some that the teachers uses to teach a lesson, and they do not understand them well.” While teachers are using the SU materials they have received, there is concern among some stakeholders about the frequency of use. Several interviewees, including head teachers, SEIs, and DAs expressed concern that the teachers did not always use the teacher guides even if they had them. The reluctance to use these materials can be partly attributed to the timing of distribution of the 22 materials, and the time when the evaluation was conducted. Most of the teachers had received training on using these materials just a few weeks before the evaluation, and others had been trained to use the guides before receiving them. A DA noted that training teachers on how to use the guide before they had the final version of it may have caused, what he called, resistance to its use among some teachers. In expressing his takeaways from SU’s experience, an IP staff member noted that training without all materials is problematic: “it is a huge compromise if materials are not available.” When asked whether teachers were open to using these newly-acquired teaching guides, an SEI responded: “There are still some challenges, some teachers don’t follow what they learned in the training when using teacher guides. Some have received teacher guides but put them away in the cupboard and teach what they want the way they want.” SU staff also perceived this challenge of using teacher guides. As one staff member stated: “A large number of teachers are able to use the teacher’s guides. There is a challenge here though because some appreciate the teacher’s guide, but still want to teach in their own, old way. Change is not something that happens overnight, so they are taking them through a process of behavior change. They are really a program of behavior change so people understand their roles in schools including teachers, head teachers, DOS, and SEIs.” Some teachers appear to disagree with the content sequence in the teachers’ guide because it is different from what they were previously accustomed. For example, in one school in the sample, teachers stated that they believe students should never read a word until they know all of the letters first, but in contrast, the guide did not follow that principle. Several teachers expressed concerns about covering the literacy skill content in one forty￾minute period. Two teachers mentioned there is too much content to cover in 40 minutes, and that they need more training. Three other teachers expressed concern about covering the content for the term because of the limitation of teaching only one literacy skill per 40-minute lesson. For example, in the SU curriculum, phonics and phonological awareness are taught in the same lesson. However, teachers found the combined teaching of these two concepts to be difficult in one forty-minute lesson. One teacher noted that, “…in discovering sounds you do not go in deep. We wish they should divide it into two parts. Forty minutes is not enough to teach discovering sound and linking sound at the same time and make them read because reading is required there. Reading words, reading syllables, reading sentences. Putting it in one lesson seems to us too long. It is too long.” SUB-IR 1.2. TEACHERS’ USE OF INSTRUCTIONAL PRACTICES IN EARLY GRADE READING SU staff has worked with REB to develop teacher training materials in the principles of effective EGR instruction, which are for trainings of trainers (ToTs) and trainings of lower primary Kinyarwanda teachers. Content includes the components of literacy development, such as reading fluency, vocabulary, reading comprehension, and assessment. SU also worked with REB to form the National Reading Training Team (NRTT) in order to provide in-service training across the country. This is composed of local teacher training experts, including professionals from REB, University of Rwanda College of Education (URCE), TTCs, local Non-Governmental Organizations (NGOs), Kinyarwanda national subject trainers, and district master trainers. Finally, SU has developed a strategy, and provided training to head teachers and teachers on Communities of Practice (COPs) to support teachers’ implementation of effective EGR instructional practices. 23 It is important to note that although training includes essential components of literacy development (e.g. phonological awareness, fluency, writing, etc.), the emphasis of training is not on theoretical aspects of literacy development or teaching methodologies, but rather on how to use the materials. It is assumed that if teachers follow the teacher guide, their instructional practices will be effective to help students learn to read and write. Informants who had observed classes prior to and after SU training noticed improvement in EGR teaching practices. One interviewee working with a TTC gave examples of observing the difference between teachers that had received SU training and those who had not. He visited two schools and saw that one had literacy-enhancing visuals on the walls and the other did not. He noted that students had different learning levels based on whether their teacher had received SU training, and had incorporated learning materials into their instruction. Similarly, all of the DAs interviewed commented on observing improved teaching practices. As one DA noted: “I used to observe lessons and saw that they [teachers] mixed all the five literacy skills in one lesson, and the children would get confused. After the training, they learned they needed to teach the skills in separate lessons. I have seen a big change in classroom instruction to reflect this, and children aren’t as confused as before.” Similarly, head teachers discussed what teachers seem to be doing well based on the training. These competencies included having students collaborate when reading and getting students actively involved in a lesson by asking questions before or during the lesson. As one head teacher stated, “When we compare how Kinyarwanda teaching was before SU and how it is now, we as head teachers find improvement. Teachers used to improvise but now they work closely with students to learn especially using the I do, we do, you do approach. The school has become a friend to students and students interact well with teachers. The number of students who can read improves progressively.” All teachers reported utilizing new teaching practices, and seemed to understand the importance of the different literacy components. One teacher shared her appreciation for the approach of teaching syllables instead of single letters. Another teacher explained how she has changed the way she teaches since the SU training, stating, “For me what I liked is the way they explain or interpret pictures that help us to read and write. Before gaining the knowledge from the training, I used to make learners write immediately if for example it is a letter that we are going to learn; we could end the lesson at once, and go to the next lesson and so on. Now these five pillars, phonology,…they progressively complement one another.” When asked which ones are the most challenging to work on, teachers tended to respond that reading fluency and comprehension are the most challenging, while phonics and phonological awareness are the least challenging. Most teachers utilized practices that support effective literacy instruction, yet some practices need further reinforcement. During the classroom observations, the evaluation team noted that all of the teachers followed the methodology of the teaching guide, and teaching and learning aids were used where applicable. All teachers used verbal reinforcement approximately five or six times per lesson, and all checked for understanding three to six times per lesson. Teachers adjusted their instruction to accommodate student understanding generally four to six times in a lesson. However, in five of the 11 classes observed, teachers did not have visuals on the walls, such as letters and words. Students were encouraged to take their textbooks home in only four of the 11 class observations, but all parents interviewed indicated that their children bring their textbooks home regularly. 24 Most teachers carried out some practices that did not cognitively engage students and were not evidence-based, effective practices for literacy development. In almost all of the observed classes, teachers used call-and-response questions and chorus answers, which did not challenge the students since they would simply recite the response in unison. When many students would become disengaged, some teachers did not notice or respond. Additionally, teachers did not always afford time to students to read individually. Notes from the observation protocols reflect these practices, such as “Students were interested with the lesson but were not given time to read by themselves”, and “Students were entering very late throughout the lesson. There was lots of reading directly from the teacher's guide and call and repeat. Students did not use any materials themselves.” In many of the observed classes, much time was spent on these practices, rather than being devoted to cognitively engaging students. Teachers showed variable awareness of areas in which their teaching practices could be improved. After observing a class, the evaluation team asked teachers to reflect on the effectiveness of the lesson delivery. Teachers mentioned certain practices that they thought would have improved the lesson, including giving students time to understand the story, saving time by handing out books ahead of time, assessing students individually, and helping students connect stories to real life. However, several areas of improvements that the team observed were not recognized or acknowledged by some teachers. For example, five of the 11 teachers did not assess students at the end of the lesson. While two of these teachers commented that they should have done this, the other three did not mention it. In only one case did a teacher refer to problems with student engagement and/or class management, even though this was a challenge for most teachers observed. There appears to be a fair amount of independent reading, but not so much independent writing by students. In the 11 observed classes, 10 had independent reading for students, which usually lasted for approximately five minutes.12 However, students were observed doing independent writing in only three classes. In looking at student notebooks, it also appeared that very little or no writing had taken place. One challenge mentioned was that students do not all have notebooks because the parents have not bought them. According to a DA, “Teachers are struggling with teaching writing. The teacher guide does have writing in it, but parents are often not involved so they don’t give their students notebooks. This is a major challenge for teachers and head teachers working with parents.” Teachers and head teachers value the scaffolding approach, though it sometimes seems automated, and the “you do” component is under-utilized. It was clear in classroom observations and in teacher FGDs that teachers liked and were easily able to implement the “I do, we do, you do” strategy. However, this approach seemed to be overused and to be something teachers did “automatically” throughout the lesson, sometimes even when they were not introducing a new concept. For example, this approach was used when singing a song that the children already knew. Furthermore, there were times when teachers read aloud to the students for long periods of time with no action by students, which tended to cause students to become disengaged. When students were asked to read aloud, it was often in unison, thus it could not be distinguished who was able to read or not, and in several cases students “read” aloud without looking at the book, but rather recited from memory. In sum, while teachers appear to be following the teacher guide lessons and implementing the corresponding teaching practices, they tended to focus more on their own actions and less on how 12. The IP explained that writing is included in lessons on particular days of the week, so this may explain why writing was seldom observed. However, based on the lack of writing in students’ notebooks and comments such as the DA’s teachers struggling to teach writing, it may also be an area for improvement. 25 these actions were engaging students in learning. In other words, classroom practices reflected a teacher-centered approach much more than a learner-centered approach. SUB-IR 1.3. CAPACITY OF HEAD TEACHERS TO COACH AND SUPERVISE EGR INSTRUCTION Under this sub-IR, SU and the GoR defined that SU would support the policy goals of enhancing school leadership quality and training and would implement a system of continuous professional development. This would be achieved by developing the capacity of DEOs, SEIs and head teachers to provide improved support to schools. SU provided initial orientation on the activity for all head teachers from public and government-aided schools with a lower primary stream, all DEOs and SEIs, and all principals of TTCs. Additionally, a school leader orientation module was developed, and head teachers, DOSs, SEIs, and DEOs were trained in evidence-based reading and writing instruction, scaffolding instructional strategy, formative assessment in P1 classrooms, and other key topics for improving literacy instruction. School leaders also created action plans to support the implementation of SU activities in their schools. SU has developed a template to collect information on the frequency of SEI visits to schools. SU also developed a school leadership training guide that includes the coaching tools and guidelines that were approved by REB. Additionally, modules were developed and validated for head teacher and DOS training, and head teachers and DOSs were trained in school leadership and teacher coaching as well as COPs. DEOs and SEIs were trained to provide support to head teachers who coach teachers, as well as in COPs. Finally, SU has developed and distributed to head teachers six videos addressing evidence￾based instructional strategies for teaching reading and writing that focus upon teaching the different literacy components. Although coaching trainings were still ongoing during the time this evaluation was conducted, most stakeholders expressed optimism about the improved capacity of head teachers to coach and supervise teachers. As expressed by a PA, “The testimonies they [head teachers and DOSs] have from training shows that they have an impact. Also, when head teachers visit schools, they see teachers putting the training content into practice and understanding that they need to give the books to students.” One of the DAs highlighted the importance of providing support to coaches as they begin to implement their coaching responsibilities. She noted: “I visit teachers five days per month and observe and discuss lessons and assess students. When I observe the classes, the head teachers or DOS usually come too. They collaborate well, and the teachers often have many questions. I even have a WhatsApp group with the coaches [head teachers, SEIs and DEOs] to discuss issues they may have or questions they have as an individual that any group member can answer spontaneously.” SEIs, DEOs, and DDEs appreciate that the training has provided guidance on going beyond their managerial and administrative roles to being able to evaluate teaching quality and provide constructive feedback. It was apparent from discussions with SEIs, DEOs, and DDEs that they appreciate being included in the training as a way to improve EGR instructional practices. As a SEI stated, “I remember very well learning how to teach Kinyarwanda…it is was very good as it helped me in classroom visit.” Similarly, a DEO commented on his job monitoring teachers and head teachers: 26 “There is no problem of being a DEO and coach at the same time, I see this like a positive thing in Soma Umenye, because I have to monitor what I master very well. If they have another coach, maybe I will miss this notion of coaching and if I go in one school, I will be monitoring what I don’t know. They know that it is the same person who trains them that also monitors them.” SEIs expressed that combined coaching and inspection duties are complementary and perceived their new role in a positive light. All SEIs spoke positively about COPs. Two SEIs indicated that the coaching relationship between SEIs and head teacher has helped improve the image of SEIs, as they were previously perceived by head teachers as being on a fault-finding mission every time they visited the school. One SEI stated, “Our role as coaches has changed our relationship with head teachers. They see us as trusting and present in school to help them and improve but not to judge them.” Another SEI commented: “I think that being a School Inspector and coach are complementary roles. I used to be School Inspector and my image in schools was that of a person going in to identify mistakes and hold people accountable. Now that I am a coach, I plan with the teacher and provide formative feedback when I visit a school. Teachers and head teachers see me as someone coming in to help them improve. The relationship is more mutual as I am now perceived as part of the team trying to improve performance on learning.” However, it also seemed apparent that the SEIs were still adjusting to this new role, and this required a paradigm shift. As a DDE stated: “They [SEIs] feel that their responsibilities are inspecting things which are wrong only. But we explained to them that somebody can teach in his methodology but when you come near him you can coach and show him a good methodology he can use just to help a learner to understand…In the first days it was not easy but today they understand it. But they still have this question: ‘Are we going to coach teachers or we join hands with the school mentors like headmaster or DOS?’ Today they say that inspection for a teacher is not necessary because in coaching everything is done inside.” Similarly, several head teachers also mentioned a positive change in the approach to supporting teachers. One head teacher noted, “We used to visit teachers just to inspect and give them marks. Now, we visit them, appreciate what they do and correct them where we think they are wrong.” Teachers reported that head teachers and DOSs coach them more often than just observing their classes; and all but four teachers reported being coached once or twice per month. While almost all teachers (34 out of 40) reported that head teachers provide coaching, 29 reported that the head teacher also observes their class. More teachers reported that the DOS coached more often than merely observed (24 and 20 respectively). Thus, it appears that, overall, head teachers and DOSs are fulfilling their role as coaches. Furthermore, according to these data, SEIs have some presence in the classroom, while the DA appears to have very infrequent presence in the classrooms (See Figure 1). This is consistent with the findings of the Instructional Time Study,13 which also revealed that head teachers most frequently provided class observation, followed by DOSs, and then SEIs. However, the reported frequency of classroom observation was much higher according to the evaluation’s teacher questionnaire than the Instructional Time Study: in the teacher questionnaire 53 percent and 38 percent of teachers reported being coached twice or once per month, respectively, and 13. Room to Read. A Study on Kinyarwanda Instructional Time in Lower Primary. Draft Version; July 19, 2019. Room to Read, p. 17. 27 only four reported never receiving coaching/observation, while in the Instructional Time Study 25 percent of teachers reported never being observed, 30 percent once per term, and 19 percent reported monthly observation. Figure 1. Teachers’ report of who coaches them and who observes them Teacher questionnaire data reflect SEIs as being both inspectors and coaches. Most teachers reported being observed and coached by the same individual(s), although this differs for SEIs. Eight teachers reported that SEIs observe but do not coach, while three reported that SEIs coach but do not observe. This may be due to the changing role of the SEIs in relation to inspection, monitoring, and coaching, as discussed above. They are no longer expected to observe classes as part of their monitoring, but rather are directed to coach the head teachers and DOSs in their teacher coaching (hence observing classes with them). Overall, teachers reported that coaching is helpful, yet several teachers expressed that head teachers need more guidance and training. All teacher questionnaire respondents noted that the coaching they receive is sufficient, except four who reported that they are never observed or coached. As one teacher expressed, “The head teacher; even class teachers [coach us]. They really help us. They gather us and speak with us. Like this we tell them the challenges we face so they advise. That’s enough [support]. And they are really skillful about it.” Other teachers commented that head teachers need more preparation to coach, stating, for example, “Maybe they [head teachers] might also need some guiding materials in mentoring us.” Another teacher commented, “Most of the time DOS and head teacher come to us but he/she has only recently been trained, he/she not yet used to doing it.” Head teachers demonstrated knowledge of general good teaching practices when providing coaching to teachers, but lacked awareness of key elements that prevent students from cognitively engaging. Some head teachers gave examples of how they provide coaching. For example, one head teacher explained, “Normally, when you are teaching a lesson like this one of reading that can make a student being bored, you can read and do a certain warming exercises and another thing that can make students waking up, yes, so that the students do not get bored.” Another head teacher mentioned that when coaching, he focuses on the lesson plan and instructional objective, how the teacher adapts content to the students’ level of understanding, the learning environment, how teachers use assessment, and how they use cross-cutting issues. However, although the head teacher/DOS observed the class with the evaluation team, in almost every observation, when asked about the quality of the lesson no head teacher mentioned the chorus answers, recall questions, and over-emphasis on the “I do” as being problematic for cognitively engaging students. In the majority of focus groups, teachers suggested that peer support is helpful and should be done regularly. An SEI discussed the importance of teacher peer learning groups: “I told you that we have SBM (School Based Management) and COPs. I think our task is to maintain those teachers in SBI (School Based In-service) and COPs where they share knowledge with others. Teachers come 29 20 16 0 2 3 4 0 5 34 24 13 3 0 0 4 1 5 HT DOS SEI DA REB BLF Nobody Other teachers Others Who Observes And Who Coaches Teachers Observes Coaches 28 together and teach each other, and they train each other.” A teacher stated that that teachers should be brought together to reflect on the SU methodology at least once a semester “in order to make sure no one is off-track.” This teacher thought that the SEI/DOS/head teacher, after visiting a class, should bring teachers together to reflect on what was observed. Another teacher stated, “The advice one may give is that we should have time to meet to read it through it together in order to have the same understanding on it.” Finally, one teacher said that all teachers in the sector should meet regularly to “remind one another steps that are followed in teaching.” There is no systematic coaching mechanism or protocol for follow-up on coaching recommendations. Head teachers explained how the DOS writes a “report of inspection” based on class observations and the SEI also does class observation and provides teachers with suggestions. Additionally, the DEOs and the DA observe classes. However, there is no mechanism that permits follow-up by the different individuals observing the class and providing feedback. It is also not clear how information in these reports is used to foster continuity of teachers’ improvement. The evaluation team was not able to identify the degree to which teachers put into practice the feedback they receive from coaches, beyond what teachers and coaches reported. One teacher gave the following example of coaching feedback: “There was a time the teaching aids were not enough, not the books, when you want to make the lesson practical, let’s say ‘trees’ then the head teacher tells you that ‘these teaching aids you brought are not enough, you could have uprooted a small tree so that you may show its parts, the way they appear, the stem, the branches, and the roots’ then you take that as a good example and next time you bring them.” School leaders, district officials, and SEIs perceived COPs to be helpful. A DEO explained that there is a COP for teachers at the school level, and one for coaches (head teachers) at the sector level. Three district officials, three SEIs, and half of the head teachers talked about the COPs being helpful for them. As one SEI explained, “For me the community of practice helps me as a coach. I cannot have time to coach all teachers from different schools… We sit together and they take time to coach their colleagues in their schools, that is how it helps.” A DEO stated, “It [the COP] empowered the teachers. Those teachers feel free to say what they, how they see they teach before the class, just the teachers feel free to express themselves in front of their leaders.” The five head teachers that discussed COPs said that they had recently put them into place and that they were important fora for discussing progress and challenges. As one head teacher explained, “Another important program that we have put in place is creating Kinyarwanda teacher’s forum where we meet one occasion every month to evaluate the Kinyarwanda teaching, and discuss the performance level of students.” SUB-IR 1.4. SCHOOLS’ AND TEACHERS’ USE OF STUDENT ASSESSMENT RESULTS SU’s support to REB seeks to provide teachers with guidelines, tools, and training to carry out formative assessment at the end of each lesson, week, and unit. Teachers are expected to monitor student progress, identify struggling students, and share assessment results in schools and with parents. SU has developed and piloted assessment tools and guidelines for P1. The guide is also aimed at helping teachers to analyze and share assessment results, as well as to identify children who need additional support. In addition, the teacher guides for P1-P3 include directions for conducting formative assessments, and scripted formative assessments at the end of each lesson, week, and unit. All P1-P3 29 teachers and school leaders (head teachers and DOSs) were trained to ensure that both teachers and school leaders can use the tools effectively. DEOs/DDEs and head teachers all demonstrated an understanding of formative assessment and its importance. All DEOs/DDEs and head teachers interviewed explained how often student understanding should be assessed, and why it is important. As one DEO explained, if formative assessment is done well, then summative assessments will show good results. A head teacher explained, “We mostly assess them on reading. We take a short story or words, depending on the level of the child, from P1 to P3, and then we give them like a short story and we assess how they read. There are children who read many times, others read very slowly, given children’s level, you may realize that he/she masters it and you find another who is not able to combine syllables, so that’s what we focus on.” Teachers have differing understandings of formative assessment, though they all expressed that it is important, and that they are trying to incorporate it in their teaching practices. Interviews revealed that some teachers specify its purpose as assessing literacy development, while others seem to think of it as checking for understanding during a lesson. A teacher explained, “The important thing I studied in it [SU training] is that after teaching a certain topic, I have to give an evaluation. I give an evaluation on understanding the story, I give another on discovering sounds, I gave it in reading without confusion and in writing. There are many evaluations in order to be able to know the levels of pupils.” Another teacher referred to “informal assessment” through asking questions. She explained, “I teach them and then I assess them to see whether they have understood what I have taught. You know it through their answers. When you ask a question the whole class raises their hands, especially in the informal assessment, when you are teaching you ask question like three children and when two of them are failing, you realize that the lesson is not yet understood and it requires to start the lesson again and make it understood to learners, that is the importance of informal assessment.” In the 11 observed classes, seven teachers checked for understanding frequently, and four checked moderately. In terms of literacy skills checked throughout the lesson, most frequent was vocabulary, followed by comprehension, and then phonics or phonological awareness. Reading fluency was checked in two cases, (i.e., the most infrequently one checked in the observed classes). Some teachers and head teachers have used assessment results to identify and support struggling students. As one head teacher stated, “There is improvement. Because if we find a child with a serious problem, we add some extra minutes on the normal time so that we can help that child particularly. A teacher finds time on Saturday like one or two hour(s) to help such children.” Another head teacher explained how he helped a struggling student: “For example, there is a child in P3 who did not know how to write and read, once I went to a class to do observation, by simple visit like that, then they tell me that the boy does not know to read and write, then I tried to focus on that boy, and now she, he knows to read and write. To mean that assessment help to communicate the problem that is found in learners, but when you do not do assessment, you do not have anything to communicate even to the parents and to the learners.” 30 Figure 2. Teachers’ report of how often they assess students Teachers understand how frequently they are supposed to assess student learning, but they do not implement assessments systematically. Most teachers stated that they know that they are supposed to assess their students at the end of each lesson, and several teachers specified that they aim to do this with a few students (to check if the students learned what they were supposed to learn). In six of the observed classes, the teacher assessed between three to five students individually at the end of the class. In the other classes, this did not occur. The six teachers who assessed students primarily did so by having students read. Teacher questionnaire data also reflected this as the most common method of assessment. The teacher questionnaire data also reflected that only eight percent of teachers reported that they assess their students at the end of each lesson and the majority of teachers assess students only once a week (See Figure 2). Additionally, assessment at the end of a lesson appears to be random, with teachers not documenting information so as to follow up on students’ weaknesses, or on how to adjust subsequent instruction. When asked how well formative assessments are being implemented in his school, a head teacher responded: “Normally it is 50 percent, because some teachers assess learners after three lessons or after the end of the unit, but, normally when it is possible for these students to give them assessment regularly, it could be very useful.” In sum, although head teachers and teachers know the importance of systematic formative assessment, such assessment is not currently the general practice. Teachers face challenges carrying out formative assessment, one of which is the large number of students in the class. Teachers, head teachers, and most other stakeholders interviewed expressed that formative assessment was a component of EGR, but also noted that there are still challenges in its implementation. One DA said that the last time they had teacher training, they cross￾checked and found that half of the teachers did not conduct the assessment because of time and class size. A teacher commented: “The time is short, because of what you prepared, saying that you will give it in 40 minutes, as you understand that if it is an evaluation it includes many things. There is writing, reading, it means he has to divide it into parts matching with the minutes reserved for that lesson, then you give another part we cannot give it at the same time, it cannot be possible because of time.” Several teachers thought that assessing their students once a week was a more realistic goal. One of the solutions that SU suggested to the teachers is to have other teachers help when assessing every learner in the class. However, this was only discussed as a realistic solution for one (Catholic) school that had other personnel available to assist: “..and here we are lucky we have Catholic sisters sometimes they take over and keep the class busy so that your absence does not create a big problem.” Assessment results did not seem to be widely shared at the school level or with parents, and did not appear to be used to inform teaching practices. Very few teachers and head teachers addressed how assessment results were being used beyond their own knowledge of the 8% 78% 6% 8% How Often Teachers Assess Students Every lesson Once a week Every unit Once to twice per month 31 students’ level/understanding. Most often, teachers report using assessment results to re-teach what students did not understand, and to ensure students learn what was taught. As one teacher mentioned, “We record them [assessment results] in notebooks, and we analyze their performance category and find out whether the lesson was well understood or not, or whether you have to revise for them depending on their results.” Three teachers stated that they discuss assessment results with parents if the student is failing to learn something or progress, and two of these teachers also say that they give the student homework aimed at helping them learn. As one teacher explained: “When I assess them and they fail, I give them a homework and ask them to show them to their parents that evening. I give the homework about the questions that I asked in assessment, for example if it about writing words, I give them words which are similar to those they dealt with in assessment and tell them to ask homemates to help them. So when they come back with good answers, then I know that they have shown their parents. When I enter the class and they remind me about that homework, I realize that they have done it and then correct them.” Some teachers said that they use assessment results to classify students according to their level of understanding. Several teachers also said that they share results with the head teacher: “The other thing is that we share these results with the school administration. And the school administration will decide if any particular efforts to invest in to uplift these students. And like this both sides contribute in moving these students from a particular level to another level.” According to a DA, teachers need to share the results more broadly and frequently. As he stated, “They just keep the results. In the training, they told them to do it and then share results with head teachers and parents and also identify the slower learners.” Interviews with parents confirmed that they do not learn about assessment results. Several parents remarked that teachers notify them if their child is struggling with learning or is distracted, but they do not receive specific feedback on how to support their child at home with learning to read and write based on her/his performance. Similarly, based on the interviews with stakeholders at the district and school level, it was not clear if and how assessment results are being analyzed to inform teaching, or if there was knowledge of how to do this. One DDE noted that both the DEOs and teachers need training to interpret the results of formative assessment to inform teaching and support teachers: “They [district staff] will analyze the results with head teachers and then find new strategies that work to improve EGR…But what we didn’t get it is how to make analysis of that the result, yeah how to make analysis of the result and we really need that kind of training…yeah after just making a formative assessment how can we really interpret the result so that they can provide constructive feedback to the students and to the schools.” As noted by a head teacher, another way of providing specific feedback for students is through homework. In all but one case of classroom observation, students were given homework, although it was unclear if teachers provide feedback on homework assigned and completed. One head teacher said that teachers often do not provide feedback on homework, which he was trying to change. 32 IR 2: SYSTEMIC CAPACITY FOR EARLY GRADE READING INSTRUCTION IMPROVED SUB-IR 2.1. NATIONAL ADVOCACY MECHANISMS FOR EGR INTERVENTIONS SU has carried out advocacy work on multiple levels, including with high-level policymakers such as MINEDUC and the Ministry of Economic Planning and Finance (MINECOFIN), to give voice to the importance of early grade reading to ensure quality education. SU has also aimed its advocacy work at the district, sector, and school-level as well as other local decisionmakers. To reach this variety of stakeholders, a range of activities have been implemented. These aim to provide opportunities for high￾level policymakers to recognize the importance of EGR in Kinyarwanda; promote a culture of reading in Kinyarwanda at the local and national levels; and to develop districts-level capacity to identify and respond to local and national EGR priorities. Progress on the aforementioned activities, as documented in SU quarterly and annual reports, includes developing a capacity-building plan, and strategy to strengthen national advocacy mechanisms. The SU advocacy strategy has sought to involve policymakers during the process of mainstreaming EGR initiatives, given that it is expected that they have the greatest potential to sustain EGR initiatives at a policy level. The strategy has also included supporting REB in the Andika Rwanda initiative: a writing competition for primary and secondary school students aimed at inspiring writers and readers to generate contextually￾relevant, original stories in Kinyarwanda. Working with the sub-contractor Never Again Rwanda, the winning stories are published in books that are made available for classroom libraries. SU has had several roles in this effort, including promoting the competition at a province and district levels, and providing training on editing and publication preparation for the winners. SU has used national communications networks to spread real time information on EGR instructional practices. SU has secured a channel to share audio and text materials by mobile phone with teachers and head teachers to enable them to access information about early grade reading. In support of this effort, SU has developed over 45 messages to share with teachers and head teachers regarding best practices in EGR, inclusion, gender, and utilizing classroom materials. Additionally, SU collaborated with Mureke Dusome to draft content for messages that will be sent to parents to promote reading at home. An IP staff member emphasized the importance of this approach: “Sending early grade reading best practices through such platforms has proven to be important considering that we want the short messages to reach as many people as possible. The people to whom the messages are addressed also instantly send them to other people in the networks or forums. We now send short messages to head teachers and teachers about different issues like how to best maintain textbooks, teacher guides, etc. By doing so, you end up having a pool of people who access such positive messages and very likely develop knowledge and positive attitudes on early grade reading issues.” As part of its advocacy strategy, SU has sought membership in pre-existing education coordination fora, although there have been missed opportunities for partnerships. These fora include the Rwanda Education NGO Coordination Platform (RENCP) and the Basic Education Strategy Group to sustain a collective voice on EGR best practices and to communicate issues related to EGR at the national level. By engaging in these fora, SU has secured leadership of various working groups, and is well-positioned to advance its position on EGR-related causes. For example, SU’s Monitoring and Learning Director was assigned to co-chair the assessment and curriculum working group, which brings together REB, MINEDUC, and education development partners. Such fora also 33 facilitate effective coordination of SU and other education development partners’ activities in districts, schools, and teacher training calendars. Several stakeholders discussed the importance of maximizing the use of existing mechanisms to promote EGR messages. For example, several head teachers, SEIs, and teachers noted that the Umuganda (the monthly community service day) is an excellent opportunity for this, as it is the means by which the central government – through the Ministry of Local Government (MINALOC) – communicates key messages. EGR communications can be integrated into these messages. According to stakeholders, parents attend these community meetings, and seriously regard and heed the adopted resolutions. Furthermore, Mureke Dusome already engages children and other community members with literacy activities during Umuganda that provides SU with an additional opportunity for collaboration. Some government stakeholders expressed that SU provides wider literacy support and promotion through initiatives such as Andika Rwanda. Several stakeholders at the district and sector levels discussed the importance of Andika Rwanda for promoting literacy in Kinyarwanda and motivating students’ learning. As a SEI stated, “I think that Soma Umenye is solution for Andika Rwanda. During the reading competition organized by Andika Rwanda, it comes and finds already Soma Umenye had trained students in reading and writing.” Similarly, a DDE noted that with Andika Rwanda “children improve the way of thinking, there are critical thinking minded children because they create themselves what they can present to others and if they think that the product is that they will provide it to others and will be rewarded, children really work hard.” This DDE suggested that even more such writing competitions should be held at district and school levels (e.g., one per term). However, although stakeholders valued SU’s support for the competition, according to several IP staff members, SU’s specific roles in supporting Andika Rwanda are not clear. SUB-IR 2.2. STUDENT AND TEACHER PERFORMANCE STANDARDS AND BENCHMARKS FOR EGR SU has worked jointly with CTLRD and the Examination and Assessment Department (ESAD) at REB to review and draft benchmarks for P1-P3 reading fluency and comprehension, and student performance standards. With the assistance of a consultant, SU analyzed available assessment data to identify tentative standards and design a pilot test of the standards in selected schools. SU then established standards for oral reading fluency and reading comprehension in consultation with representatives of REB, MINEDUC, URCE, and teachers. In early 2019, the Minister of Education asked the Director General of REB to respond to key priorities related to the Human Capital Index education indicators, including establishing a national reading target, and articulating plans to achieve the target. SU has offered technical support to enable REB to respond to this agenda, and the benchmarks will facilitate measuring these targets. Education stakeholders acknowledged that benchmarks will be an important tool once they are approved and operational in schools. SU has been at the forefront of ensuring that student and teacher performance benchmarks are developed and applied during teaching and learning, and (more specifically) during assessment. SU developed and submitted the student and teacher performance standards and benchmarks for EGR to REB for validation and approval before they are applied in schools. According to an IP staff member, “Benchmarks are only for Kinyarwanda and are the first benchmarks Rwanda has had. They were just recently approved. The next step for them is breaking annual benchmarks into term benchmarks.” 34 A REB official discussed the progress of the validation and approval of benchmarks, and how important they were: “By using benchmarks, teachers are able to provide help based on the students’ abilities. When I visited some of those schools I saw one school where a teacher was doing this and I knew that was from Soma Umenye’s trainings. The teacher was able to categorize the students based on their varied ability in reading and writing Kinyarwanda. The teachers had a plan on how to support the weak students.” A REB ESAD official expressed optimism that benchmarks will be a reference tool which can be used to assess students’ progress and learning. He was appreciative that the process of developing benchmarks had involved all stakeholders concerned, including the assessment department, the CTLRD, the quality assurance department from the MINEDUC, and lower primary teachers from all 30 districts. SU has worked with REB to develop teacher performance standards: the key competencies and behaviors that lower primary Kinyarwanda teachers must display in order to help learners achieve the approved learner benchmarks. SU plans to collaborate with REB to ensure that these EGR instructional competencies align with the overarching teacher competency framework that has been developed. REB has approved the SU-developed benchmarks, but stakeholders at the school level were not yet aware of the benchmarks and their impending application. Teachers and head teachers in the evaluation sample were not aware of the content of the benchmarks, even though they will be the actors who implement them. There was likely a missed opportunity to progressively inform head teachers and teachers about the process, updates, and what they would be required to do with benchmarks. This could have been done during the training on formative assessment, and would have been a good strategy to provide information at the school level, and prepare teachers and head teachers for its implementation. SUB-IR 2.3. RESEARCH-BASED POLICIES AND CURRICULA IN SUPPORT OF EGR The aim has been to develop and implement an EGR research and learning agenda with the REB in order to inform and strengthen curricula and policy frameworks. An additional goal was to disseminate evidence, and ensure findings are used to inform decision-making. SU has built a collaborative learning agenda and is working to execute the research in the agenda. In 2018, SU developed a learning agenda in consultation with USAID and other key partners in order to enhance ownership by all involved stakeholders. SU proposed a number of studies to USAID to inform the learning agenda. One of the completed studies is the Instructional Time Study. Its aim was to provide the REB with evidence to improve P1 and P2 Kinyarwanda reading outcomes by understanding currently allocated, engaged, and academic learning time, how these vary across schools, and how each category of time is associated with Kinyarwanda reading outcomes. According to SU staff, the research process was very collaborative. The study was designed with a reference group including REB and URCE. The extent to which time was an issue was a surprise, and SU is making sure this message is strongly articulated. An informant from URCE stated: “The time instruction study I find to be very interesting to see what teachers are actually doing in class. For example, a teacher may spend time reading by themselves, and the amount of time that learners are actually engaged is very small. Based on these findings, we have taken much time to think about how to improve time on task.” 35 Stakeholders emphasized the importance of disseminating findings from SU’s learning agenda. Stakeholders noted that information about SU’s teaching approach should be more widely available. One interviewee from URCE noted: “At department level, we have owned the activities and even used the approaches to inform our teaching. We have trained and conducted research and trainings on these new teaching approaches. However, we need to even do better. How about we develop policy briefs about the good things that are done by an organization like SU [for greater influence]. I would say we are silent on good things! We should do better.” Given the findings of the Instructional Time Study, another interviewee from URCE proposed conducting teacher workshops to increase awareness of time factors and how to more aptly manage time and better engage students. This training, he stated, could also be incorporated into pre-service teacher education. In collaboration with REB, SU plans to support the dissemination of findings of the study and to develop policy briefs aimed at informing MINEDUC and REB decisionmakers about the policy implications of these findings. SU has collaborated with URCE to conduct research, but the partnership is not yet formalized. SU has engaged URCE academic staff in development of training materials, research projects, and trainings. However, according to an official from URCE, a memorandum of understanding (MOU) was never established between URCE and SU to formalize the partnership. According to this URCE official, signing this MOU remains important because it would help to define the working relationship between URCE and SU. SU has supported the implementation of the new competence-based curriculum. A number of stakeholders commented on SU’s support for the implementation of the new curriculum. For example, an SEI said, “As partnership in education, it (SU) has helped in improving teaching methodology as we are shifting from former curriculum knowledge-based curriculum to competence-based curriculum.” SUB-IR 2.4. EARLY GRADE READING ASSESSMENT SYSTEMS STRENGTHENED REB has established an assessment system – the Learning Assessments in Rwanda Schools (LARS) – and has adopted a policy to conduct comprehensive assessment, in alignment with the newly-adopted competence-based curriculum. SU is supporting REB to further develop and strengthen their assessment systems through activities such as collecting LARS data, conducting a baseline evaluation of student learning outcomes, reviewing the results with REB, and conducting an EGRA. SU also collaborated with REB to identify a sustainable national EGR assessment model. In support of this goal, SU has developed a concept note on early grade assessment and accountability frameworks that has been presented to the REB-USAID-SU Steering Committee. SU also coordinated the development of assessment and accountability frameworks with staff from REB, MINEDUC, URCE, teachers and district education personnel. SU is working collaboratively to improve the overall capacity of REB technical staff overseeing assessment. For instance, the process of co-developing the EGRA has resulted in increased capacity of REB staff. As a REB official explained: “With EGRA she [a consultant] comes and she works with our staff to develop those tools. when they are developing together those tools with our staff at the same time there is transfer 36 of competencies of skills. I think this technical assistance or support is really very very important.” To further strengthen capacity, SU is providing REB with additional human resources. As stated by a staff member from this department: “To build systematic capacity of national-level staff in early grade reading assessment, Soma Umenye has supported REB to recruit an assessment expert who will be based at REB to support the Assessment Department to align mainstream EGR assessment practices in the department activities.” SU has increased the capacity of REB to execute the EGRA, which has strengthened assessment systems more broadly. According to a REB official, SU and REB worked closely together to develop, pilot, and conduct the EGRA, thus enhancing REB staff’s capacity to conduct future EGRAs. As stated by a REB staff member, “Staff from the assessment department and other stakeholders even inspectors from the Ministry and others from the department curriculum. We have been fully involved [with SU]… that was really a success about the outcomes of benchmarks on EGRA.” REB will use the P3 Kinyarwanda literacy benchmarks validated with the EGRA for the LARS, and these benchmarks will serve as a basis for REB to develop benchmarks in other subjects. As explained by a REB official, “We are going those categories for EGRA only in Kinyarwanda. But later on, in two, three or four years, we can adapt these categories and apply to other subjects especially in language subjects, English, French, etc.” In this way, SU is contributing to strengthening REB’s existing assessment systems, and is fortifying institutional capacity to execute other types of assessments, such as the EGRA. Finally, according to a REB official, the use of EGRA as part of the LARS has given more credibility to REB’s assessment system, prompting them to apply for membership with The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ). This will facilitate their participation in regional comparative assessments of student learning. SU’s support for formative assessment and Local Early Grade Reading Assessment (LEGRA) contributes to the comprehensive assessment model adopted by REB. The Rwandan government recently adopted a comprehensive assessment model, which aims to assess student learning in every subject in every grade, and the extent to which learning targets are being met. This new assessment system is based on and aligned with the newly adopted competence-based curriculum. In support of this aim, SU has collaborated with REB to develop and implement the LEGRA, an assessment similar to the EGRA, but intended to be formative and administered on a local level at the end of each term. According to a REB official, application of the LEGRA will directly support REB to execute comprehensive assessment. Furthermore, the official noted that SU has built the capacity of departmental staff in charge of assessment to implement LEGRA. The expected importance of LEGRA was captured in the following statement by a REB official, “With LEGRA, its expected that teachers, students and parents will be able to progressively see how students are achieving learning outcomes instead of waiting for results of summative assessment and where necessary provide remedial measures before learning gaps widen.” An official from ESAD explained how SU and REB are supporting teachers to conduct formative assessment, and how this also supports implementing the competence-based curriculum: “We have developed tools beside assessment guidelines and standards. We have uploaded on the REB website; assessment guidelines for P1 to P6. Those are standard questions that can help 37 student and teachers know how they can set competence-based curriculum. We have also set model questions and uploaded them on the REB website. These questions cover all the units to help teachers to use these questions and develop similar questions.” However, this same informant also noted that there has not been sufficient training for head teachers, teachers, and other education stakeholders to deeply understand the comprehensive assessment tools. He expected that, with time and training, all users should understand those tools, and he noted that one of the strategies to make them more accessible to a wide audience was to upload them on REB’s website. While LEGRA is promising, there may be reluctance to share assessment results, and it was unclear if there is sufficient capacity to analyze and use the data. Several DEOs and SEIs expressed concerns that LEGRA may also evoke negative repercussions when students’ low performance is misinterpreted by MINEDUC’S officials as a lack of commitment by DEO’s, SEI’s, head teachers, or teachers. One of the DEOs expressed this concern: “We have had DEOs and head teachers expelled or given serious warnings because students have scored low grades on national exams. There is a lot of pressure from MINEDUC and REB that students should score high performance on learning. Head teachers tend to compete with each other even when their schools don’t have the same teaching and learning materials. They would rather portray a false positive picture than a true negative picture of students’ performance to avoid negative consequences from higher officials.” Furthermore, as in the case of formative assessment results, it was unclear if capacity has been established at the district, sector, and school levels to analyze the data and use them to inform instruction and supporting teachers. As one district official noted, “But what we didn’t get it is how to make analysis of that the result…and we really need that kind of training.” SUB-IR 2.5. CAPACITY OF TTCS TO EFFECTIVELY PREPARE EGR TEACHERS In response to the concern by REB that TTC instructional practices were not aligned with competence￾based practices, and that instructional materials were not sufficient, SU provided Rwanda’s NRTT TTC tutors with SU training manuals to enable them to conduct four days of training for all Kinyarwanda candidate teachers. As a result, 669 Kinyarwanda pre-service teachers were trained on early-grade reading instruction. REB officials and other stakeholders appreciated that SU has contributed to TTCs’ uptake of evidence-based EGR instructional approaches. As stated by a REB official: “What I know is that at least Soma Umenye has been working with TTC tutors and even managed to train final year students in TTCs. You see that is a strategy to create what we can call a bridge. When you are final year with TTC, it means that next year, you are entering the teaching force. I think that they did that and we say that it is important that they inject their training approach. Soma Umenye’s approaches are used to train TTC tutors.” Another REB official, who is also a lecturer at URCE, also underscored SU’s role in training TTC Tutors and review of the TTC curriculum: “Soma Umenye initially worked with in-service teachers to teach Kinyarwanda in lower primary but after discussions, it was found that you cannot support in-service without pre-service. That is how SU came to support us in Kinyarwanda syllabus [curriculum review]. We wanted to align 38 TTC curriculum with primary and pre-primary education curriculum. So they have been helping us but I also know they support in-service teachers with trainings.” Stakeholders perceived the SU intervention in TTCs as having great potential for sustainability, but also suggested formalizing the partnership between SU and URCE. SU has supported REB to finalize the pilot version of the TTC lower primary Kinyarwanda curriculum. According to one official from URCE: “SU intervention in TTCs is seen in two ways: supporting and training in-service teachers and also supporting and training pre-service teachers. I look at this as strength – in-service teachers are being empowered with skills for teaching Kinyarwanda. But because Soma Umenye will not be staying for long, these pre-service teaching will be implementing what Soma Umenye is doing now based on the skills they are being taught. I know for example, that pre-service teachers are now using the 5Ts when teaching Kinyarwanda.” Additionally, some of SU’s EGR instructional approaches are gradually being integrated in the pre￾primary teacher education program. An official from URCE’s Pre and Primary Education Department noted: “At department level, we have owned the activities and even used the approaches to inform our teaching. When teachers in my department get involved in SU activities, they come and also start teaching according to Soma Umenye approaches. Soma Umenye active teaching approaches are embedded in our pre-service modules.” A formal MOU between SU and URCE has not been established, however, if put in place, it could help strengthen collaboration. An informant expressed that SU should have an MOU with URCE so that URCE leaders would be better informed on SU’s interventions, thus facilitating URCE’s contribution to SU activities: “SU has very good materials in their programs but these materials should systematically be incorporated in TTC curriculum so that they are used in the future,” he said. According to an URCE official, this would include SU training materials and EGR materials that need to be systematically mainstreamed in pre-service teacher education curriculum. EQ1.A.II. IS SOMA UMENYE VIEWED AS POSITIVELY CONTRIBUTING TO GOR PRIORITIES AND LEARNING OUTCOMES AMONG GOR STAKEHOLDERS? SU is perceived as comprehensively supporting one of the GoR’s main priorities, improving literacy skills in Kinyarwanda, as well as related priorities, to establish the conditions for continued literacy learning improvement at all levels. Since 2017, when it became evident that SU activities lacked alignment with GoR priorities, SU has made a careful effort to ensure that its activities directly support GoR goals and their implementation. One REB official highlighted SU’s support for their main priority, improving lower primary students’ Kinyarwanda literacy: “Early grade reading has always been in priority of REB because actually we have had problems with children who complete primary three….So Rwanda Education Board has always wished to make the priority to have children at least completing primary three to be fluent readers in the mother tongue…the importance of the language really cannot be overlooked because language of instruction and if you don’t understand the language of instruction you are not likely to understand even other subjects that are in that language. So I think we really value that input of Soma Umenye.” Similarly, a MINEDUC official emphasized the importance of SU providing each and every student with their own textbooks. He noted that this had always been a problem for them, and was something they 39 wanted to resolve. Prior to SU’s support, the ratio was approximately one book for every five students and that affected children’s level of interest in the books, and negatively impacted their learning. As he stated, “This issue will be resolved with SU, and I very much appreciate it. This improves reading because the child can take the book home and the parents can have some influence.” Another REB official explained that SU supports REB’s progress toward one of the goals in the National Strategy for Transformation: improvement of foundational skills in English, Math, and Kinyarwanda. Additionally, IP staff explained that SU is supporting goals in the GoR’s Education Sector Strategic Plan (ESSP), the development of a dashboard to monitor progress toward the ESSP goals, and SU is also contributing to elaboration of the National Literacy Policy and Teacher Development and Management policy. Several informants commented on the comprehensive nature of SU’s intervention. According to a REB official: “SU has chosen an area in which they hadn’t had interventions before, and lucky enough the scope is country-wide. It’s present everywhere, and they are leading in the work they are doing. Other partners do ‘soft interventions’ like just coming in and doing teacher training. But what does training leave behind? They need materials. SU is well-composed because it goes beyond training and enters the classroom.” REB officials valued capacity building in priority areas that has occurred through the close and collaborative working relationship with SU. One REB official remarked that SU’s recruitment of an assessment specialist was critical to build the capacity of REB staff to commence the new comprehensive assessment policy. Likewise, an IP staff member explained, “The expertise to develop and evaluate textbooks did not exist in REB before SU. SU got an international consultant and trained the REB team on how to evaluate textbooks and teacher guides (just the Kinyarwanda specialists). Through this process, REB learned what those textbooks should look like. All materials are co-developed with REB. For instance, Rwanda never had decodable readers before SU, so SU developed these tools with REB so that they are able to recreate the process themselves.” SU complements the local capacity-building work of MINEDUC. A member of MINEDUC explained that SU trained his inspectors, who have responsibility of training the SEIs and DEOs. Thus, SU is helping strengthen the capacity of these trainers, which is a priority for MINEDUC. SU’s alignment with GoR priorities extends to the district and sector levels. Several informants mentioned how SU alignment with GoR policy goals and priorities occurs at all levels. As one DA remarked: “It’s good because what we are doing is in line with the ESSP. That’s a government initiative, but it has to be implemented by the district office. Looking at the context of SU’s work, DAs, school administrators, and teachers have embraced the SU work because it is in line with the government priorities.” Another DA explained how the activities are aligned with the district strategy plan: “SU has trained SEIs and DEOs how to make action plans and lobby for Kinyarwanda in the last two weeks, and hopefully this will start to show results and mainstream these goals and SU activities in the district strategy plan. Imihigo is meant to achieve these overall district strategy plans.” 40 EQ1.A.III. TO WHAT EXTENT HAS SOMA UMENYE ADDRESSED INEQUITIES BASED ON GENDER, AND STUDENTS AND TEACHERS WITH DISABILITIES? GENDER SU has incorporated principles of gender sensitivity in its materials and trainings. Multiple stakeholders recognized the project’s efforts at gender sensitivity in the materials developed, noting that the materials are careful to avoid stereotypes and traditional notions of gender. One DA described an example of gender sensitivity in the SU-provided materials, noting that girls were depicted as having employment, and this avoids – and negates – the stereotype that females should stay in the home and do domestic work. Trainings by SU have encouraged teachers to treat male and female students equally in the classroom. For instance, multiple teachers noted having learned to call on male and female students equally, and not favor one over the other. Another teacher described a strategy she had learned for gender-sensitive teaching, “As far as gender balance is concerned, they taught a trick to congratulate a learner who performed well. Don’t tell the learner you are ‘akagabo’ (translates literally to growing man). When you say that, females will not feel involved, because the term doesn’t entail ‘female’ …. You just congratulate them in general, because they otherwise would not let females feel included.” The evaluation team’s classroom observations revealed that teachers were implementing such practices most of the time; gender inclusive language was used in all but three observed classes. In all cases it appeared that teachers were giving male and female students equal attention in the observed classrooms. SU has demonstrated gender inclusive practices in their trainings by providing childcare. Interviewees appreciated childcare because it allows mothers to focus on learning, knowing that their children were safe and secure. As one DEO described, “They provided special food for the kids and someone who is in charge of the kids, which is really important for us, because if you don’t consider the kids, you don’t consider the mother.” DISABILITIES Teachers have received some training on special education needs (SEN) and requested more training on how to identify and support students with particular special needs. Most teachers confirmed that they had received training on inclusive education. However, the evaluation team’s analysis of teacher questionnaires revealed that 6 of 40 teachers found training on inclusive education to be insufficient. As one teacher noted in the questionnaire, “Severity of special needs of individual learners varies, so we need more training.” Teachers appreciated the training they have received thus far, and have begun implementing some of the strategies learned in the training. One teacher described how she supports a student with special needs, “You don’t separate the child from others, that is one. In addition, you look at the disability that he or she has. For example the one I have, the way I help him, first of all I have to arrange for his sitting because he cannot sit on a desk like this …So I look for his own desk… Secondly, I must acquaint him with other children. I have to explain to other children how they must treat him without being scared of him or show him that he is not like them…” While few teachers gave such nuanced responses, the majority of teachers explained that their method of supporting students with special needs is to have the student sit in the first row. Teachers tended to identify students with disabilities only when they are obvious, rather than more subtle learning disabilities. 19 out of the 40 teachers who responded to the 41 questionnaire indicated that they had children with special needs; 10 teachers indicated that the disability was a hearing impairment; five teachers indicated that the student had a vision impairment; and two teachers indicated that a student could not speak. Four teachers who had indicated having a student/students with disabilities did not identify the type of disability. This may indicate teachers’ difficulty identifying disabilities. Furthermore, five of these teachers indicated having more than one student with a disability. In all but one of these cases, the teacher only listed one type of disability, further underscoring the possibility that there were unable to identify the type of disability. In other cases, teachers may have not recognized a condition as a disability. An example is a teacher who reported having no students with disabilities, yet it was observed that she had a student with severely crossed eyes. This student was not in the front row and the teacher did nothing to accommodate him throughout the entire lesson, even when students were supposed to read. Thus, it appears that teachers need help in identifying types of disabilities, as well as specific ways of accommodating these issues. In general, school staff requested more information on identifying different types of disabilities and strategies to support learning for students with such needs. Teachers explained that they struggled to identify disabilities that were not immediately apparent. As one teacher stated, “For example, there are children who have disabilities that we don’t know, yes, and you realize that there is something missing in what the child is doing but you don’t clearly know what is wrong. But because his parent didn’t explain to you, we find it as a challenge.” Stakeholders found it challenging to use SU materials for children with certain needs, and requested more accessible materials. School staff members found it difficult to implement inclusive strategies without having accessible learning materials that support students with special needs. As one head teacher noted, "This child with disability needs special school materials of his or her own linked to his or her disability, particularly physical disability. Sometimes he or she needs an appropriate desk for the disability. This problem of not having such kind of special school materials exists." Approximately two years ago, SU conducted a situational analysis of materials that were available in schools to support children with disabilities in learning to read. The analysis revealed that there are no standardized tools for school staff to identify children with disabilities, and that few materials exist to support children with special needs in accessing the curriculum, thereby echoing the evaluation findings. Similarly, other interviewees noted that it would be helpful for teachers to have materials that afford guidance on how to support students with particular types of disabilities. A DA remarked, “Each training session, they incorporate these issues into their handouts and materials. They also trained teachers and school leaders on how to care for gender issues and children with disabilities. Practically, I don’t think the teachers or head teachers know much about these issues. They don’t yet have materials dedicated to these issues – would be nice to have leaflets on these issues for teachers.” Remediation did not seem to be systematically addressed at the school level. With SU training, some interviewees acknowledged that teachers were better able to identify students with special needs, and students who are struggling with learning Kinyarwanda. One SEI found that, after the SU training on inclusive education, “…people get to understand the person who needs special education, whereas in the past they thought that special education should only go to the person who has a disability.” However, several stakeholders cited a need for a more structured approach to remediation for students who need additional or individual support with early grade reading, since it can be difficult for teachers to provide extra support on their own. As one head teacher noted, “…we have those weak learners, but it is not easy to spare enough time for them and pay special attention to them because we have a big class.” Given these difficulties, the SU project is currently preparing a structured remediation course to meet this need in the future. 42 EQ1.A.IV. IS SOMA UMENYE MAKING EFFECTIVE USE OF ITS ORGANIZATIONAL STRUCTURE AND STAFF? SU was designed to strengthen the capacity of state actors (namely head teachers, DOSs, and SEIs) to implement literacy improvement activities such as providing coaching, executing COPs, and conducting the LEGRA. The role of the DA was meant to support these state actors in such activities, ultimately leading to their strengthened capacity to conduct the activities without DAs' support. SU’s centralized organization has caused strain at the district level. IP staff members in Kigali argued that the role of the DA is ultimately to drive change at the school level, particularly through training of head teachers and SEIs. IP and GoR staff at all levels view the role of the DAs as essential to implementing the SU training and coaching model. However, it was unclear that DAs have sufficient time and resources to successfully accomplish this objective. DAs overwhelmingly agreed that they do not have sufficient time for follow up and that 10 visits per month were insufficient to provide the level of support and coaching needed by each school. Given that DAs have such a large number of schools, and are required to visit only five schools per month for coaching purposes, they would only be able to visit each school for coaching about once per year. Thus, although schools, and in particular head teachers and DOSs, may receive some coaching and feedback, DAs are unable to follow up, answer questions, and ensure that course corrections are made. As one DA explained: “A challenge with coaching is that I have so many schools. It would be better if I could coach one school a month, but I have 80 schools. This is too many to do in one month. They are expecting so much from head teachers and DOSs to implement coaching. A big challenge is that their expectations from head teachers and DOSs may not be possible, alongside all the other duties they have.” This view was supported by head teachers, with one respondent stating that they do not get enough support from DAs. While they are able to interact with DAs during training, they do not receive sufficient follow-up or direct coaching afterwards. Another DA expressed, “SU staff needs more time to follow up deeply in classrooms and conduct observations. For instance, I spend most of my time on coordinating trainings. I have little time for follow-up.” Despite these challenges, teachers and head teachers clearly valued the coaching they received from DAs. As one teacher recommended, “…I think Soma Umenye officers at district level would not be removed as soon as the project winds up its activities, they should be allowed to operate for another number of years so they can follow up these activities.” IP and district-level GoR officials agreed that the SEIs are essential to the implementation of SU. However, given their importance, there was a lack of clarity from DAs, DEOs, and SEIs in how SEIs fit into the overall implementation structure of SU. SU staff members at a central and district level agreed that the exact responsibilities of the SEIs and their support to SU is still unclear. Additionally, DAs are unclear as to their responsibilities to coordinate and support the work of SEIs. For example, there is no comprehensive sharing of data, or reporting between SEIs and DAs, although they both conduct coaching and school inspections. In interviews, SEIs spoke of the DAs’ role as complementary to their work. Two of the SEIs interviewed said that they visit schools with the DA when possible. One SEI explained that the DA was helpful because, after a school visit together, “they [the DA] give us feedback on what they realized about it so that you can start from it and take other strategies that can 43 help to improve more in education.” Despite these examples, interview data did not clarify the role of the DA in supporting the SEI. This lack of clarity is due, in part, to the relatively recent establishment of the SEI role, which was changed to an inspector position without providing clear guidance on roles and responsibilities. An IP staff member explained this lack of clarity, and the importance of the SEI’s role: “The [SEI] role has recently changed, as they are now accountable to a central Inspectorate. But, it’s not exactly clear what they are accountable for. Their position seems very reactive to whatever central action comes down, responsibilities seem kind of fluid and unclear even if there is a written-out job description. The comprehensive assessment system SU is trying to build only works if SEI role can support it properly.” SU has made extensive use of short-term technical assistance (STTA) in completing technical deliverables. To date, SU has made regular use of external STTA for technical development of key activities, including the development of SU’s core curriculum and training modules. While these STTA have brought high-quality expertise to technical areas of the SU activity, they are a costly use of time and resources, and this can lead to sustainability challenges. Given these issues, SU staff are exploring opportunities to build capacity and draw from in-country technical expertise. For example, SU staff have explored the potential of transitioning a percentage of SU’s technical team to become “super￾coaches”, who are responsible for one or two districts’ direct oversight, particularly during the LEGRA roll-out. They have also explored utilizing expertise from Rwanda’s NRTT for use in STTA. This would draw from existing GoR staff and resources, rather than external consultancies. EQ2: IMPLEMENTATION CHALLENGES: WHAT ARE SOME CHALLENGES OR OBSTACLES TO IMPLEMENTING THE ACTIVITY? HOW COULD THOSE CHALLENGES BE ADDRESSED? EQ2.A. NATIONAL LEVEL CHALLENGES The most consistently raised challenge was delays in production and distribution of SU materials to schools. As addressed earlier in this report, there were significant delays in the production and distribution of SU-provided materials. The evaluation team found that some schools had received textbooks for levels P1 through P3, while other schools visited were still waiting to receive textbooks in P2 and P3. Many of the production challenges have been overcome, but materials have not yet been fully distributed. Interview and observational data revealed inconsistencies in distribution of materials between schools, leaving some classrooms without particular materials. One teacher stated that, “They [the books] are few and they are not enough, and we don’t have the teachers’ books, there are only those of pupils.” Where materials had not been received, or had already been lost or damaged, school staff was unaware of how to request or follow up on their delivery. A head teacher remarked, “We don’t have enough books, and when you ask, they [parents] reply that they cannot find where to buy them so that when parents miss some books or don’t maintain it well, we can’t replace them because they are not available on the market.” There is a lack of clarity about ownership of textbooks, and concern that students will not care for them, or will forget them at home. Several head teachers and teachers expressed concern about textbooks getting damaged and lost. Additionally, stakeholders expressed uncertainty over whether the textbooks belonged to schools or to students. Due to this confusion, some school staff have not allowed students to bring textbooks home because of fears that they will not be returned, and not knowing how to replace lost books. As one teacher stated, “the way of safeguarding them is a problem because parents do not take care of them. You find a child coming in saying ‘I lost it’, then we 44 call the parents, looking after them is not done accordingly.” Furthermore, stakeholders reported being aware of some parents not allowing children to use the textbooks at home due to fear of having to pay for damaged or lost books. One head teacher remarked that parents are afraid of having to pay for materials if they are lost, and “Therefore, he [the parent] sees that the child will lose (the book), and he keeps it away from them.” Such restrictions from school staff and parents on the students’ use of SU materials presents a challenge, as it limits the amount of time students can engage with the materials and practice literacy skills. Numerous stakeholders expressed concern about the durability of the materials. Various stakeholders cited concerns that books would be damaged or destroyed within one to two years. Some interviewees reported seeing heavy wear-and-tear on materials already, even though materials have not been in schools for very long. Both teachers and head teachers remarked that they had already seen damage to books that had been distributed this past year, with students not taking proper care of the books to preserve their longevity. A DA noted that the “Primary challenge I’ve had so far is that they are distributing books, but they are already getting destroyed. What should be the replacement mechanism?” Similarly, a head teacher remarked, “[The parents] can help children to maintain books in good condition so that they last… if the book is not well maintained, it can’t last for a year, but if it is well kept, it can last for two or three years in good status.” Another significant challenge during the project’s implementation was teacher transfers to new subjects, grade levels, and even schools. Interviewees revealed that teachers are often transferred to new subjects or schools after they have been recently trained by SU. This jeopardizes the effectiveness of the trainings and sustainability, as it prevents teachers from utilizing the skills they may have gained during training. As one DA noted, “…some trained teachers were moved to new subjects, and I coordinated with the District Education Unit to make sure that the Kinyarwanda teachers were brought back to the subject so their training wasn’t for nothing.” Some stakeholders have implemented solutions to address the challenge of teacher transfers. However, the response has been uneven and uncoordinated in and across districts. In one province, the PA and Governor coordinated efforts to produce formal guidance for head teachers on this issue. However, based on interviews with IP staff and REB district officials, there did not appear to be a coordinated response across districts to ensure that trained teachers are not transferred. EQ2.A.I. HAS SOMA UMENYE ADDRESSED BARRIERS TO GREATER GOR UPTAKE OF EVIDENCE-BASED APPROACHES? SU has experienced some barriers to the adoption and use of evidence-based practices in the GoR. Interviews revealed that GoR officials have concerns about the source of evidence-based practices and whether they are suited to the Rwandan context. For instance, one REB official noted his reservations about adopting LEGRA, a type of international assessment adapted to local contexts: “Yeah. Like LEGRA. So when it comes here, we have to see if it can fit with our context. So, for example, they [SU] proposed to only assess comprehension because LEGRA it is like, it has been developed to assess only comprehension, so lastly when we discussed with the them, we told them that if LEGRA has to support comprehensive assessment. LEGRA have to fit in the structure of comprehensive assessment and the comprehensive assessment will have to assess all components in Kinyarwanda that is: writing, reading and comprehension. So, now they are, they are updating these tools according to our view and ideas on comprehension on comprehensive assessment.” Despite this challenge, SU has made significant advances in addressing these barriers. As demonstrated 45 in the case of LEGRA, SU has invested time in adapting to the needs of the GoR to ensure that evidence-based practices are still incorporated into the education system. GoR officials recognized the willingness of SU staff to discuss and adapt such practices to their context. As one official remarked, “…they’ve [SU] always, you know, discussed the evidence…what was contextually relevant to us, we had no problem in understanding and getting a common understanding on what was proceeding.” EQ2.B. DISTRICT, SECTOR, AND SCHOOL-LEVEL CHALLENGES Large classroom sizes presented a challenge for teachers in delivering the SU approach. Of the 11 classes that the evaluation team observed, the average class size was 41 students per classroom. The largest classroom observed had 70 students. Observation protocol data reveal that, in most classes, keeping students engaged throughout lessons was difficult. Teachers tended to resort to the same strategies across classrooms (such as chorus answers from students), rather than using other approaches that could engage all students in learning. Interviewees at all levels recognized how such numbers of students create challenges for teaching and learning in the classroom. One DDE remarked, “the big number of students hinders teachers to reach each student and know the difficulties of each child…” Given these conditions, school staff has experienced challenges implementing certain components of the SU instructional approaches. As one SEI stated, teachers say “the time is not enough due to learners’ questions, in fact there is no time for every child to fully participate because of a big number of learners in front of teachers.” In particular, teachers and head teachers noted that classroom overcrowding makes it difficult to implement the “I do, we do, you do” approach. The “I do, we do, you do” technique aims to ensure that each child has the opportunity to practice a new concept being learned. However, teachers expressed concern about whether they would be able to give each student this opportunity, given the number of students in their classrooms. For example, one teacher described how time-consuming the process of introducing a new word in a story can be, “…you have to explain words by using I do, we do, you do method, and you realize that the time for all that is very difficult due to the length of the story. For example, I may explain two words and when I start to read the story, I find another difficult word and I have to explain it to the learners, and for that reason, the time becomes short.” Class size also makes assessment and individualized support difficult to provide. As a head teacher noted, “Another thing is to evaluate the activity, to evaluate the students’ ability they have in the lesson of reading or sound awareness on which you provided guidance…You put these selected students aside immediately after supervision of the lesson so that you give them an assessment. As you do this, it takes a certain amount of time, although not much, but the students are deprived of their time to continue with the next lesson.” Head teachers and teachers expressed particular concerns over assessing writing skills and the amount of time this requires. The same head teacher remarked that “…when it comes to writing and even reading, this (assessment) would cost them a lot of time and it’s worse when their teacher needs to shift immediately to another lesson.” Stakeholders at the school, sector, and district levels cited issues with their availability for trainings. SU trainings for teachers and school leaders are typically held on weekends to avoid taking school employees away from their weekday duties. However, this training schedule presented issues for certain groups, in particular religious groups that worship on weekends. An SEI who had experienced this challenge remarked, “We don’t attend many trainings because of beliefs like Seventh Day Adventist, even me I am an Adventist and there are some trainings that I don’t attend. I think there is a gap of that challenge.” 46 To combat these challenges, SU has organized “catch-up” trainings for individuals who are unable to attend the weekend trainings. Yet, these sessions are not seen as sufficient to compensate for the missed trainings. A teacher who had completed one of the catch-up trainings found that “The short time, the time was not enough and so many lessons were given in a short time, people got tired, and they would not understand well.” Although recently started, stakeholders cited issues with insufficient time and depth of SU’s coaching component. It is difficult for DAs, DEOs, and SEIs to provide consistent coaching and follow-up to schools, given both the limited number of school visits they can make, and accessibility issues of certain schools. As one head teacher expressed “There isn’t enough support, and we don’t get to interact with SU staff enough. We only see SU staff at trainings, but not in meetings or visits. SEIs and DEOs are too busy to lend support as well.” Head teachers also expressed concerns about having enough time for their own coaching responsibilities. Lack of time and other responsibilities prevent them from being able to coach more often. Approximately half of the head teachers and DOSs questioned expressed that they would like to do coaching more frequently than they are currently able. Three head teachers said that they would like to observe and coach each P1-P3 teacher once per week, but that it is not possible. As one head teacher stated, “We should do it twice a week but we are so split into many things that we don’t do it as we should. I am like a power strip (multiprise). I am the school accountant, the head teacher, I manage this school and a sister satellite school…I am a coach but I am unable to effectively do coaching.” One head teacher explained that he does one class observation per week, and there are six classes (P1-P6), so presumably each teacher is observed every six weeks. Furthermore, the SU coaching model relies heavily on SEIs. SU anticipates that SEIs will provide coaching to head teachers, and follow-up on feedback given to schools during DA visits. Yet, as discussed, the role of the SEI has recently undergone changes, and it remains to be seen whether SEIs will have the time and latitude to complete these duties as well as their other responsibilities. Stakeholders saw parents as critical to ensure that materials are maintained and to foster a culture of reading in Rwanda. However, there have been challenges with engaging parents. Parents have a potentially critical role to play in the maintenance of materials. For instance, parents may be able to provide children with carrying bags and storage places in the home to ensure that materials are not damaged or destroyed. Teachers and head teachers highlighted this possibility, but noted that parents often do not appreciate the importance of maintaining books. As one head teacher responded, “A further reason why children do not keep books safe for home reading is that the majority of them come from rural villages and their parents never value the fact of allowing children go home with books for reading. So, the parents never think of planning to find for his or her child a small school bag to carry books and notebooks.” Multiple interviewees, particularly teachers and head teachers, suggested that SU could do more to engage parents to support the development of a culture of reading (as discussed in the section on Collaboration, Learning, and Adaptation). Likewise, stakeholders noted the potential of parents to support children’s development of literacy skills at home. Several teachers expressed that they would like parents to be more involved; as one teacher stated, “I think it’s important to train parents as well so that they may help children at home.” However, many raised concerns about parents’ awareness and knowledge of how to practically support reading at home. A head teacher stated, “The challenge that we have is that of parents who sometimes don’t help their kids to read books that we gave them to take home. Or you find that even a parent is not able to read.” 47 In the four schools in which the evaluation team was able to speak with parents, there were different degrees of awareness regarding SU, and how parents can support their children. In two schools, parents were very clear about SU’s goals and how they can support their children; one parent gave specific examples (that she devised on her own) of how she supports her child’s learning at home, such as having the child read aloud, and giving them time to read alone. In contrast, parents in the other two schools said that they knew little about SU, and that they thought parents need to receive training about what SU is trying to accomplish, how they can support that at home, and how to use the materials with their children. As one mother stated, “I recommend that they invite parents and then tell them about the project, its goal and its suggestions. So, a parent will be able to teach his or her child. Be it to train them, be it tell them about it in meetings, that parents become aware of it.” EQ3: SUSTAINABILITY: WHAT EFFORTS ARE BEING MADE WITHIN SU TO ENSURE SUSTAINABILITY AND HOW COULD SU MORE EFFECTIVELY WORK TOWARD SUSTAINABILITY? EQ3.A. HOW DOES SOMA UMENYE DEFINE SUSTAINABILITY? Implementers defined sustainability as behavior change, particularly at the school level. Stakeholders expressed varying definitions of what sustainability entails in the context of SU. USAID and IP respondents agreed that sustainability, in large part, will be realized through concrete changes in the behavior of teachers and head teachers in applying the SU curriculum and teaching practices. Two REB respondents echoed this vision of sustainability, arguing that sustainability involves teachers sharing the skills and approaches that they have learned from SU with other teachers. In this way, behaviors learned through SU would be continually passed on into the future. In examining sustainability, the team used the broadest definition possible to encompass the perspectives of all respondents and stakeholder groups queried. Therefore, the findings in this section encompass any view of sustainability that captures how the approach, interventions, resources, behaviors, or partnerships created by SU will be maintained beyond the period of performance. EQ.3.B. IS THERE CURRENTLY DEMONSTRABLE OWNERSHIP AND DEMAND FOR SOMA UMENYE INTERVENTIONS? HOW COULD SOMA UMENYE FURTHER PROMOTE OWNERSHIP AND DEMAND? There was clear demand for SU at all levels of the GoR. Informants agreed that the main reason for the GoR’s demand for SU is that it helps them improve early grade literacy – a central feature of the ESSP. As a member of the World Bank explained, “I think the demand is at all levels but mostly at the central level. This is because the central government primarily feels the pressure to improve the early￾grade reading conditions as contained in the policy documents. Then the demand goes up to the school level through the district and the sector.” In sum, the responsibility of the government to improve early grade literacy generates demand for SU as a promising intervention toward accomplishing this goal and task. One clear demonstration of demand for SU was the close cooperation and collaboration of REB staff and SU at each stage of the material development process. Both IP and GoR staff cited this process as valuable for creating REB ownership. This was strengthened by all SU materials being owned and branded by REB, thereby creating a greater sense of ownership and investment from GoR partners. As an SEI stated, “SU textbooks are jointly co-produced with REB and are officially used in lower primary nationwide. This is an illustration that SU materials are official MINEDUC and REB materials.” 48 Cooperation between SU and the GoR was mirrored at the district level, where the evaluation team found numerous instances of sector and district-level GoR officials intervening to ensure the successful implementation of SU interventions. One DA attributed this to the educational priorities shared by SU and district-level government, leading to mutual cooperation and buy-in for SU’s interventions. Various interviewees gave examples of GoR involvement with and support for SU activities. As a PA explained, “The Deputy Director General from REB is visiting a training site at the district level. Also, the district Vice Mayors and even Mayors visit Soma Umenye activities and meet participants.” Similarly, a DDE explained that the Mayor wrote to the SEIs about key activities, and came to the opening of trainings and other events. Recognition of the importance of EGR in Kinyarwanda represented an improvement from previous years. GoR, IP, and partner respondents recognized the value that SU has instilled by establishing the importance of EGR in the Rwandan context. IP respondents indicated that while Kinyarwanda education, particularly in the early grades, was previously viewed as a lesser subject (relative to others), the training and support provided by SU has helped to elevate the subject and draw attention to its importance. Kinyarwanda teachers supported this view, showing appreciation in FGDs for the success that the trainings have had in establishing the importance of developing a strong foundation in Kinyarwanda in early grades. As one IP staff explained, “Before SU, P4-P6 bad teachers were ‘punished’ by being sent to P1. Attitudes seem to have changed now in recognizing that P1-3 are important.” Furthermore, there is evidence that this change is having an impact at the school level. Head teachers are placing emphasis on lower primary grades, which historically had been de-emphasized relative to higher grades. One PA for SU noted that they have seen behavior change at the level of head teachers, in that they recognize the value of EGR and are placing more value on teachers at this level. EQ3.C. WHAT IS THE CURRENT LEVEL OF SKILLS AND CAPACITY AMONG STAKEHOLDERS TO SUSTAIN SOMA UMENYE INTERVENTIONS? HOW COULD SOMA UMENYE FURTHER BUILD UP SKILLS AND CAPACITY AMONG STAKEHOLDERS? There has been a clear improvement in technical capacity at REB. IP and REB staff from different departments explained that the capacity of REB has improved in several critical aspects. First, REB staff gained capacity in textbook development and verification through the joint development process undertaken with SU. REB respondents agreed that their capacity had been directly improved in this aspect, and showed appreciation for the co-creation process in which the textbooks were developed. As an IP staff member commented, “We want to see REB taking on bits and bits so that by the time SU is over, REB can manage it. REB is taking on technical activities, as the financial bits must come through budgeting.” Furthermore, two REB officials noted that, although still in early stages, the capacity of their staff to conduct comprehensive assessment has also improved. Currently SU is working with REB to finalize and deliver the protocol for a nationwide comprehensive assessment in EGR, working directly with REB to finalize the tools and protocols to undertake this effort. In addition to direct positive feedback, officials also demonstrated strong knowledge of the SU materials, training, and theory of change. Looking to the future, both IP and REB staff agreed that REB resources and staff capacity will be essential to the continued sustainability of SU interventions. Lastly, SU has also provided direct capacity building support to the MINEDUC in addition to REB staff. A MINEDUC official interviewed by the team noted that this training has had a positive impact, accelerating the approval process for SU materials by improving the MINEDUC officials’ involvement in review and verification. However, IP staff expressed doubts about the feasibility of REB resources 49 covering aspects of SU without supplemental funding, such as the printing of SU textbooks in the future. This view was echoed by a REB official who expressed similar doubts about the ability of REB to fund materials after SU ends. DAs have proven critical to building capacity of district and sector level government partners. DAs are embedded in district education offices, working directly alongside DEOs and SEIs to implement SU and work toward joint EGR priorities. Both DAs and DEOs agreed that this arrangement is mutually beneficial. DAs also conduct school visits alongside DEOs and SEIs, improving collaboration and further strengthening the capacity of district and sector level education officials. One DA explained, “We try to show stakeholders that they need to implement these changes because they are for their own interest, regardless of whether SU remains or not.” According to two DAs, these joint visits have helped increase the capacity of DEOs and SEIs to sustain SU’s interventions. They noted that their observations from monitoring and coaching show promise for sustainability, with DEOs and SEIs showing increased expertise and understanding of the importance of SU interventions. Because of this, DAs were confident that much of the school-level follow-up work will continue even without their direct support. EQ3.D. GIVEN AVAILABLE RESOURCES OF GOR AND LOCAL ORGANIZATIONS, WHICH SOMA UMENYE INTERVENTIONS ARE MOST SUSTAINABLE? At the national level, TTCs were seen as a way to integrate SU into all schools going forward. IP staff pointed to TTCs as a primary means by which SU will be made sustainable. By integrating SU curriculum into these colleges, teachers will be equipped to teach Kinyarwanda through the SU approach, without having received training directly through SU. One IP respondent posited that by conducting a ToT for TTC tutors, the tutors will adopt the SU approach, which will then extend to their graduates. This respondent also explained, “The focus is primarily on pre-service, but we also want to build capacity in-service. They are also planning a training for tutors on the new TTC curriculum to build the capacity of tutors to deliver the new curriculum.” As this approach is still underway, the evaluation team was unable to assess any outcomes to date on successfully integrating SU into in-service TTC curriculum and training practices. At local levels, stakeholders focused on the potential of COPs for sustainability and lasting impact of the training. Head teachers, district, and sector level officers cited the COPs as providing a means toward ensuring sustainability of SU trainings. While still in nascent stages, these respondents observed that teachers and head teachers have already shown enthusiasm for sharing best practices, even though informal meetings and exchanges were not directly supported by SU. Even if the trainings end, these COPs could provide a way for teachers to continue to build upon the knowledge base, and share challenges and/or lessons learned with others engaged in EGR education. One DA explained that, “the coaching system will also help as well as the COP. COP is organized by the head teachers without Soma Umenye, so this can be carried on after SU. When they meet, they discuss challenges and can share best practices/train each other.” As a PA noted, giving guidance to parents, children, and teachers is sustainable because it does not cost money: “If SU were to end, telling children to use the books and take care of them doesn’t require funds, telling teachers to use time well, establishing a day of reading, etc. Essentially, it is the good practices and behavior change like this that are sustainable because they don’t cost money.” At the national level, IP and GoR staff members were confident in the materials’ utility as a means to sustainability. GoR respondents at the national level expressed confidence that they will be able to provide textbooks in the future (i.e., after the period of performance for SU). These respondents stated that they view these materials as crucial to sustainability, citing the coordination that 50 took place between SU and the GoR in developing and distributing the materials. This view was supported by four respondents at the province and district levels who agreed that the textbooks will ultimately be sustainable, and will be maintained by schools. The respondents were clear that the materials will need to be reprinted, and were confident that they will be, although they were not clear about what funding will be used for reprinting the materials when necessary. Furthermore, IP staff noted that SU and REB are working to digitize the materials, making soft copies available for use or reprint as needed. This will help to ensure sustainability in instances where physical books are lost or damaged, or when there is demand from private schools that have not received any books to date. If the GoR is unable to fund the reprinting of the materials, there may be possibilities of funding mechanisms via other donors. For example, a World Bank staff member, not speaking officially on behalf of the organization, explained that the Bank had allocated grants to the GoR for specific activities, but noted that funds could possibly be re-directed based on the needs of the GoR. As stated, “Although the World Bank has a focus already, if the GoR suggested that a portion of the funds needed to be allocated to another activity, I think that would be a possibility...The challenge is that the areas where World Bank is currently intervening are equally important. So, it would depend on where the GoR thinks there is critical need.” Establishment of a comprehensive assessment protocol showed promise in ensuring SU’s sustainability in the future. IP staff cited the importance of comprehensive assessment, with one IP staff member arguing that establishing and facilitating the roll-out of an assessment protocol in all schools would be essential to ensuring the sustainability of SU’s gains. This staff member stated, “At the school level, sustainability can be achieved through focusing on comprehensive assessment and facilitating it in every school through the LEGRA….REB appreciated that LEGRA is built around each school’s and the system’s capacity. The challenge with this is teachers doing the assessment with guidance from head teachers. Training for how to do the assessment is about to happen.” However, as the LEGRA pilot was not yet underway at the time of the evaluation, the team could not determine direct implications of comprehensive assessment for sustainability. At the school level, teachers and head teachers viewed SU training as sustainable through knowledge sharing. The majority of the head teachers interviewed agreed that SU training is sustainable through knowledge and techniques gained by their teachers. However, both teachers and head teachers generally did not believe that the training will prove sustainable on its own. At least three head teachers cited the importance of the COPs in providing a means to share and disseminate best practices and lessons learned. These head teachers were confident in the promising results of COPs and other informal means of knowledge sharing. They also expressed optimism that monthly COPs would continue, despite the discontinuation of SU. As one head teacher explained, “I personally think that this program of forum was initiated in the schools with the target that in case the project is no longer operating, people keep on coming together and discuss. Because this program of meeting session occurs every month, whereby people evaluate the achievements and the performance level and academic improvement of students on monthly basis, I believe that this is a program that will remain permanent in spite of the project expiry.” Another teacher commented that he thinks COPs will remain “because they will help in sharing skills and experience in teaching and learning.” Yet another head teacher said he perceived the COPs as a strategy of SU for evaluative checking and follow-up, stating, “at the sector level, as coaches and school 51 leaders will meet when Soma Umenye is no longer operating with us, particularly to ensure continuity of its activities left in our hands.” In addition to the COPs, two head teachers also cited associations that would soon begin at the sector level. These associations would bring together head teachers and coaches to share experiences and address challenges. One head teacher mentioned that he participates monthly in a SU COP, as well as a BLF COP, and that they are both helpful and complementary. These head teachers see these associations as an important means of sustaining the gains their schools have made to date. Furthermore, head teachers cited the importance of engaging parents with the SU approach so that they can support and sustain their children reading and utilizing SU materials. Some stakeholders noted a desire to mainstream SU EGR best practices into performance contracts (Imihigo) at a district and school level, which could contribute to sustainability. Teachers and head teachers expressed interest in mainstreaming SU EGR best practices into their performance contracts. Additionally, SU staff members identified opportunities to build SU objectives, including improved teacher and student performance in EGR, into Imihigos at the district and sector levels. According to one head teacher, “We did not have detailed performance criteria during the development of Imihigo…we just had percentages and other quantifiable indicators without attention to details. But we want to include it in Imihigo- especially now that students have textbooks.” Such changes could ensure that district and school-level stakeholders continue working to achieve SU performance standards in the future. EQ4: COLLABORATION, LEARNING, AND ADAPTATION: WHAT EVIDENCE, IF ANY, INDICATES THAT THE ACTIVITY HAS EMPLOYED ADAPTIVE MANAGEMENT APPROACHES? DOES EXISTING EVIDENCE INDICATE THE EFFECTIVENESS OF ANY CHANGES MADE? There is strong coordination between SU and GoR at all levels. As discussed previously under EQ1, SU has active and strong working relationships with the GoR, particularly REB. REB officials appreciated the assistance and capacity-building provided by SU. Both IP and REB respondents agreed that this strong collaboration improves the success of SU activities, including materials development and assessment facilitation. As a REB official explained, “We work together every time. We currently have this workshop to finalize this tool. At each stage, we work with Soma Umenye staff and even leaders and they present the project, discuss how we prepare, organize and rearrange workshop at each stage.” As one REB official noted, SU and REB work in tandem, and activities are not initiated without the involvement and support of both entities. While there is some collaboration with other donors, it is often limited to coordination and ensuring non-interference of activities. One example of successful donor collaboration is between DFID’s Building Learning Foundations (BLF) project and SU. Respondents agreed that SU and BLF coordinated successfully and avoided duplications in activities. At the school level, teachers find value in the support received from both BLF and SU, viewing the two activities as complementary rather than competitive. Because the two activities focus upon different subjects in the classroom, teachers are able to apply what they have learned in BLF and SU trainings broadly across their teaching practices, which they find improves overall learning among students. However, actors at both the district and school levels discussed the coordination between BLF and SU as generally centered on ensuring that trainings or classroom observations do not overlap. They did not cite any major instances of systematic or proactive collaboration between SU and other projects operating in their district. Another example of technical collaboration includes SU’s work with UNICEF to develop pre-primary teacher guides for Kinyarwanda instruction. UNICEF had identified gaps between their pre-primary 52 guides and SU’s P1 guide. SU volunteered technical experts to work jointly with UNICEF to revise the pre-primary guide. However, it was evident from discussions with several officials from donor organizations that these cases of collaboration are few, and that most effort goes to avoiding duplication. As an informant from a donor organization stated, “It’s a very crowded stage in Rwanda, and we have never spent as much time on coordination as I have here. But considering this, I think it works pretty well here. On the whole, glaring duplications are avoided.” Most respondents at the central, district, and school levels echoed this sentiment. Internally to USAID, coordination with Mureke Dusome was successful, but additional opportunities remain. SU and their internal partners, Mureke Dusome, coordinated on several joint initiatives. The two activities convene monthly coordination meetings and participate in joint trainings and workshops to solicit ideas and share lessons learned. Given that Mureke Dusome has a component of developing books with publishers, SU invited their staff members to review educational materials. Additionally, Mureke Dusome staff members appreciated that SU has both provided books to every student, and relied on students to bring these books to Mureke Dusome-sponsored reading clubs. Mureke Dusome staff has received positive feedback from parents on both SU and Mureke Dusome, saying that they appreciated the complementarity. Lastly, SU and Mureke Dusome have worked together on advocacy initiatives, including a joint campaign around National Literacy Month and an International Mother Tongue Language Day. However, other than these examples, much of the coordination between SU and Mureke Dusome has centered on avoiding interference of activities. The team did not identify local level awareness of coordination between Mureke Dusome and SU. District and sector level staff, as well as head teachers, teachers, and parents were not able to identify any instances of such coordination, even though most were aware of Mureke Dusome. Mureke Dusome staff cited additional opportunities to collaborate on shared objectives, including parent training, community engagement, and remediation. There is strong coordination at the district level, with DAs highly integrated into district offices. All DAs have WhatsApp groups wherein they can communicate directly with DEOs, SEIs, and head teachers, and directly address pressing issues. Both DAs and their district and sector partners viewed this communication as highly effective, building a strong working relationship, and allowing for regular interaction between all groups. As one DDE explained, “The DA is also on a group WhatsApp. This means that if there is an emergency message or urgent message that he wants to communicate he can even communicate directly to SEIs without waiting for me to transfer the message to the group. The DA is also allowed to provide the message to SEI.” DEOs also noted the advantage of having DAs directly in the District Education Office, allowing them to easily collaborate on trainings, school visits, and strategies for improving their support to schools. One DEO explained that this coordination with DAs has helped them to better tailor their support to teachers, as they jointly observe classrooms, identify strengths and challenges for teachers together, and then collaborate on developing coaching strategies to improve teaching practices in the school. EQ.4.B. HOW IS SOMA UMENYE USING ASSESSMENTS (BOTH FORMATIVE AND SUMMATIVE) TO SPUR ADAPTATION? SU has demonstrated intentionality about utilizing research and analysis results to inform programming. Overall, SU has shown a strong willingness to conduct research and analysis to better understand the operating context of EGR, and improve the activity’s implementation. One prominent example has been the Instructional Time Study, commissioned by SU to better understand how teachers were using their time in the classroom (including teacher and student tardiness and absenteeism), and the resulting impact on student outcomes. As an IP staff member explained, “the process of research 53 was very collaborative. The design of the study was done with a reference group including REB, URCE, and more.” Based upon these findings, SU directly adapted their work plan to address challenges highlighted in the study. They shared findings internally, and conducted a workshop to determine how to respond to the study’s findings. Subsequently, SU undertook concrete actions, including amending the P1 refresher training to incorporate a theory of change that includes time management. According to IP staff, they continue to monitor items raised in the study, including issues of teacher punctuality and time. Additionally, SU has made efforts to improve knowledge management and data sharing, including the creation of a national data dashboard, which will incorporate education data from over 30 development partners into a unified, publicly available database. SU is currently facilitating this activity in partnership with the GoR. SU and its beneficiaries have made efforts to ensure greater learning in the final years of the activity. SU is currently planning a Learning Agenda event that will bring partners together to discuss challenges and lessons learned to date, and build collaborative strategies for the final years of project implementation. The event will include a review of the 2018 EGRA, the Instructional Time Study, and data and reports from five different pilot LEGRAs. Based upon the findings of this event, participants will produce summary points and recommendations for SU to adapt their programming in the remaining work plans. To date, information has not been collected and documented systematically to inform adaptation and learning. In interviews, IP staff acknowledged challenges in systematic knowledge management that informs adaptation and learning. One IP staff member noted that, “Everyone records things, but not in a systematic way that can be used as learnings for the future.” This interviewee explained that IP staff are so busy with implementation that they often do not have time to record experiences and lessons learned in a methodical way. Thus, SU staff members have cited the need for an in-house CLA specialist who could oversee internal knowledge management and learning and adaptation activities to inform both SU’s programming, and donors and projects operating in similar contexts in the future. SU currently collects data on a range of school-level indicators for monitoring and evaluation purposes. DAs, who are responsible for collecting much of these data, only conduct five school monitoring visits per month. While these data are useful, given limited coverage, the data are not sufficient to provide a national representation that could inform adaptation and learning. Furthermore, there is not extensive data-sharing between SEIs and DAs from school visits. While both SEIs and DAs conduct regular monitoring visits to schools to collect data on teaching practices, material distribution, and other key indicators, they are not currently aggregating or sharing these data. EQ.4.C. WHAT OPPORTUNITIES EXIST TO STRENGTHEN COLLABORATING, LEARNING, AND ADAPTING (CLA) PRACTICES IN THE ACTIVITY? Numerous opportunities exist to strengthen collaboration with partners. Stakeholders recognized multiple opportunities for greater collaboration with Mureke Dusome. One Mureke Dusome staff member proposed that SU and Mureke Dusome work together to provide remediation programming, pointing to Mureke Dusome’s reading clubs as a pre-existing venue in which SU could implement remediation workshops or camps. Likewise, Mureke Dusome staff cited the need to continue collaboration on the provision of books, in light of the high demand. Various stakeholders highlighted opportunities for collaboration in cultivating a culture of reading in communities. For instance, a Mureke Dusome staff member emphasized that both SU and Mureke 54 Dusome need to do more to sensitize parents to SU, and help them encourage good reading practices with their children. When discussing sustainability, a World Bank member emphasized a similar point: “There is no intervention that can be sustained without the role of the parents and the community in general. Parents need to be very close partners with Soma Umenye but I don’t think Soma Umenye has fully realized this. If they have, I think they have not fully explored the various options that would make parents fully own Soma Umenye activities. Soma Umenye is already so spread but even then, they would have had very close partnership with Mureke Dusome, which already has presence in the community. When Soma activity stops at the school level, the chain that would sustain their activity is not complete.” BLF staff members have expressed interest in greater collaboration with SU, particularly in nationwide assessment. Both a BLF staff member and a member of the IP cited the need for more regular meetings of SU and BLF, wherein they can identify opportunities for ongoing collaboration and develop joint strategies. There were opportunities for wider data sharing at the district level to improve and inform implementation. A wide variety of data is currently collected at the district level. For example, SU collects data for monthly MEL reports on student performance and teacher practices, head teachers record data from coaching teachers, and SEIs and DEOs collect data during their school visits. Additionally, data from the LEGRA will be collected every term. Using this information, SU (through each of the DAs) could convene a meeting at a district level at the end of each term to share information with district and sector staff, and head teachers, and discuss challenges and strategies for improvement. As a head teacher commented, “There are opportunities to share. Yes. And refresher trainings which can focus on reporting from head teachers, DOSs and teachers. And suppose that there is, the training is about one day or two days…three over four of trainees can do the report, how teachers did, how head teachers did, how DOS did…how those teachers, those head teachers, DOSs have implemented the program.” 55 CONCLUSIONS IMPLEMENTING PARTNER: PROGRESS TOWARD GOALS OF CONTRACT IR 1. CLASSROOM INSTRUCTION IN EARLY GRADE READING IMPROVED 1. Sub-IR 1.1. Materials: SU’s most appreciated accomplishment is material production, especially student textbooks, with aims to provide each student with their own book to use. This is a first in Rwanda. The positive effect of this can be seen in increased student engagement. However, parents and teachers are concerned that books will be lost or damaged, and it is unclear if and how often each student has their own book, and there is a lack of clarity regarding ownership and responsibility for care of the books. 2. Sub-IR 1.1. Materials: Overall, teachers liked and were becoming accustomed to using the teacher guide. However, many teachers were unable to carry out the lessons as intended due to the lack of supplementary materials; some teachers were uncertain how to use these materials when they received them. 3. Sub-IR 1.2. Instructional Practices: Most stakeholders believed that teaching practices have improved through SU, and that teachers are utilizing the intended EGR instructional practices. However, teachers tended to overuse the “I do, we do, you do” strategy beyond scaffolding, and allocated much more time to the “I do” than the “you do”, leaving too little time for students to engage in learning tasks, such as independent writing. 4. Sub-IR 1.2. Instructional Practices: Although teachers could follow the teacher guide, they continued to rely on non-evidence-based instructional practices that are ineffective for improving literacy, such as recall questions, chorus answers, and excessive recitation and repetition. Teachers and head teachers tended to be unaware that these practices are not conducive to student learning. 5. Sub-IR 1.3. Coaching: Head teachers and DOSs seemed to be fulfilling their roles as coaches, although they are not all equipped to recognize evidence-based versus non-evidence-based practices. 6. Sub-IR 1.3. Coaching: District, sector, and school-level actors valued COPs and the opportunity they provided to discuss successes and challenges; teachers, in particular, stressed the importance of regularly reflecting on the EGR instructional practices with their peers and coach. However, the role of the SEIs in this, as well as other SU interventions, was not clear. 7. Sub-IR 1.3. Coaching: There is not a system in place to help ensure coaching follow-up across different actors, and that identifies areas for improvement, recommended subsequent actions, and advances. 8. Sub-IR 1.4. Student Assessment: Formative assessment seemed to be a nascent practice that teachers are not doing systematically and deliberately, and they share different understandings of its function and execution. Furthermore, it was not apparent that this information is recorded and used to inform subsequent student support, shared with school leaders, and used to inform teaching and student support. IR 2: SYSTEMIC CAPACITY FOR EARLY GRADE READING INSTRUCTION IMPROVED 9. Sub-IR 2.1. Advocacy: SU has given voice to EGR messages nationally through its leading roles in advocacy fora, disseminated EGR messages to head teachers, teachers, and parents to strengthen EGR practices, and supported literacy promoting initiatives such as Andika Rwanda. However, there 56 are yet untapped opportunities for EGR advocacy to be conducted and sustained, such as Umuganda. 10. Sub-IR 2.2. Standards and Benchmarks: The REB deems the Kinyarwanda student literacy benchmarks as important for instruction, assessment, and helping to measure progress toward reading outcomes. However, these benchmarks have yet to be shared with teachers and head teachers. 11. Sub-IR 2.3. Research-based Policies: SU’s collaboration with URCE on research, such as the Instructional Time Study, has the potential to positively affect EGR practices. Broad dissemination of the results oriented toward informing practice will be key for creating systemic change in EGR teaching and learning. 12. Sub-IR 2.4. Assessment Systems: SU’s support of LEGRA and the REB’s comprehensive assessment initiative have helped strengthen the REB’s EGR assessment systems. However, district and school actors are not yet equipped to analyze and use the results, and school and district actors are not accustomed to sharing results, which may foster concerns about results being used punitively. 13. Sub-IR 2.5. TTCs: SU’s collaboration with TTCs has helped to equip new teachers with EGR best practices, and has assisted in aligning the TTC curriculum with primary school curriculum. These are key for mainstreaming EGR practices in pre-service teacher training. However, there is not an MOU, or similar agreement, in place between SU and URCE to help guide their future partnership. CONTRIBUTION TO GOR PRIORITIES AND LEARNING OUTCOMES 14. The SU activity is directly supporting GoR priorities as defined in the ESSP and GoR policy documents, such as Kinyarwanda literacy in lower primary, assessment, teacher training, teaching and learning materials, and implementation of the new curriculum. There is a strong working relationship between SU and REB to execute these tasks, which has led to capacity building at the central and district levels. INEQUITIES BASED ON GENDER AND DISABILITIES 15. SU has incorporated principles of gender sensitivity in its materials and trainings. Teachers have begun to implement some of the strategies learned in trainings to promote equal participation and opportunities in the classroom. 16. Teachers have begun implementing some practices to support learners with special needs; however, they lack the ability to identify and support students with special needs. ORGANIZATIONAL STRUCTURE AND STAFF 17. SU has thus far been highly centralized in Kigali, focusing primarily on establishing strong coordination with REB, and drawing heavily from international STTA to contribute technical expertise to material, curriculum, and training development. 18. DAs are overburdened, working directly with a large number of schools, as they are expected to collect data, coach, and support trainings within their district. They lack sufficient time to provide adequate follow-up and support to head teachers and teachers, and to establish better mechanisms of coordination with SEIs. 57 IMPLEMENTATION CHALLENGES 19. The most persistent challenges in implementation are delays and inconsistencies in distribution of classroom materials, and many school staff are unaware of how to follow up on the delivery of the materials. 20. Large classroom sizes (i.e., number of students) present a challenge for teachers in carrying out the EGR instructional practices, particularly the components of the curriculum that require individualized student support from the teacher. 21. Teachers were concerned about having adequate time to complete lessons and ensure that content is not rushed. In particular, teachers were concerned that combining phonics and phonological awareness in one lesson does not afford students enough depth to master each skill separately. 22. Not all teachers and school leaders have been available for trainings, given that trainings are held on weekends and some attendees have religious and other commitments. SU has organized “catch-up” trainings for these attendees, but some participants found that these trainings were rushed. 23. There were widespread concerns about the time required to complete SU’s coaching component. SEIs and DAs make a limited number of school visits per month, and are concerned that they will be unable to follow up on feedback given during coaching sessions. Head teachers are also concerned about finding time to coach teachers regularly, given the multitude of their other responsibilities. 24. Engaging parents was seen as a critical step to ensuring that SU materials are well-maintained, and students have support in developing literacy skills. However, parents often have little knowledge of SU interventions, and are lacking strategies for supporting their children’s literacy learning. 25. Teacher transfers between subjects, grade levels, and schools can result in some lower primary teachers not being prepared to implement the EGR instructional practices. Some stakeholders have implemented their own solutions to ensure teachers trained by SU continue to teach Kinyarwanda in the early grades, but teacher transfers remain a challenge in many regions. SUSTAINABILITY 26. There is not yet a common definition or cohesive vision of sustainability among stakeholders. However many stakeholders, including those from REB and the IP, believe that sustainability will be achieved through lasting behavior change, and that the COPs are a way to establish and sustain change. Additionally, there was widespread stakeholder agreement that a key way to ensure sustainability is to incorporate activities and goals for SU-supported interventions into the Imihigos at central, district, and school levels. 27. To date, the largest behavior change with regard to sustainability has been among national and district-level GoR officials. REB staff at a central level have improved capacity to develop and disseminate textbooks and other materials, and have enhanced their technical knowledge of the SU curriculum. District stakeholders have demonstrated SU buy-in by supporting SU implementation. 28. The most widespread concern among all stakeholders was the sustainability of the materials, particularly reprinting textbooks, if and when necessary/required. COLLABORATION, LEARNING, AND ADAPTING 29. SU’s successes are attributable, in large part, to the strong collaboration between SU and their GoR counterparts in REB, and at district offices. This strong working relationship has improved the quality of SU’s activities, particularly at the school level, and has helped foster ownership among Rwandan stakeholders. 58 30. Internally, SU has proven to be adaptive in responding to changes in context or operational challenges. However, these efforts have been hampered by a lack of both comprehensive knowledge management, and systematization of learning within the organization. 31. Externally, SU has numerous opportunities to build upon its current collaboration with partners, including BLF and Mureke Dusome. While coordination with donors and partners has proven successful in navigating (a crowded field of) donor activities, not enough has been done to date to identify areas of common interest, or to develop comprehensive plans to collaborate on shared priorities and objectives. 59 RECOMMENDATIONS IMPLEMENTING PARTNER’S PROGRESS TOWARD GOALS IR 1. CLASSROOM INSTRUCTION IN EARLY GRADE READING IMPROVED 1. Sub-IR 1.1. Materials: DAs should work through SEIs and the School General Assembly Committees (SGACs) to clarify student and parent responsibilities around materials ownership and to encourage their maintenance. 2. Sub-IR 1.1. Materials: Rapid distribution of all of the remaining materials should be prioritized, including supplementary materials, and DAs should work with SEIs to ensure every school has all materials. After full distribution and verification, a refresher training should be provided so that teachers can effectively use the materials and reinforce the message that each student should have a textbook in class and at home. 3. Sub-IR 1.2. Instructional Practices: Refresher training, or a session in each training, should be provided that addresses the overuse of the “I do, we do, you do” strategy; clarifies its scaffolding utility, and stresses the importance of leaving adequate time for the “you do” component; including time for students to both read and write independently. The training could also include measures for effective time use that were identified in the Instructional Time Study, particularly for large classes. 4. Sub-IR 1.2. Instructional Practices: In trainings, coaching, and COPs with teachers and head teachers, SU should address the practices that teachers use that are not conducive to student learning and literacy development (such as recall questions, chorus responses, and excessive repetition and recitation). Teachers should be allowed to anonymously express their concerns about such approaches, and discuss each one, enabling critical reflection and dialogue. It should be ensured that head teachers/DOSs are equipped to guide teachers away from impeding practices through further training, and strategies for identifying and recording these practices in their observational and coaching tools should be developed and included. 5. Sub-IR 1.3. Coaching: In order to address teachers’ needs for frequent and high-quality coaching and guidance, SU may want to explore the possibility of providing tablets for head teachers and teachers. The tablets could have the six videos of model literacy instruction, software for continuous assessment, and other supporting EGR-related software that is adapted to the Rwandan context. 14 In addition to supporting teachers to effectively teach and assess students, this tablet can be utilized to perform the following functions: 1) recording formative assessment results of students, 2) analyzing student marks (correct and incorrect responses) 3) documenting suggestions and guidance from coaching received, 4) recording teaching by the head teacher or another teacher in order to analyze her/his own teaching (as well as sending to the DA 14. Tablets were provided to second grade teachers in as part of a randomized control trial study in Kenya that contained software for continuous assessment called “Tangerine Class” as well as other literacy instructional support programs, such as Papaya that has audio capability to practice letter sounds. The result was a statistically significant effect on student learning for the teachers with tablets. Additionally, the initiative was relatively cost effective: every $100 invested in the teacher tablet treatment group increased the number of students able to read fluently in English and Kiswahili by an average of 4.3 and 4.2 students, respectively. See Piper, B., Jepkemei, E., Kwayumba, D., & Kibukho, K., “Kenya’s ICT Policy in Practice: The Effectiveness of Tablets and E-readers in Improving Student Outcomes,” FIRE vol. 2 issue 1(2015): http://dx.doi.org/10.18275/fire201502011025. 60 for feedback), and 5) documenting challenges and successes experienced teaching. 6. Sub-IR 1.3. Coaching: SU should support REB to further define the role of SEIs and their responsibilities, both in general, and as related to SU implementation. SU, through the DAs and/or or at training sessions, could assist in developing a plan for scheduling coaching visits and COPs. Additionally, SEIs’ shifting role from inspection to support should be a point of reflection and discussion in COPs and trainings. 7. Sub-IR 1.3. Coaching: DEOs, DAs, and SEIs should guide head teachers to conduct regular COPs with teachers, and encourage peer mentoring among teachers; these should be documented to share with the SEI, DEO, and DA, such that subsequent COPs build upon each other. 8. Sub-IR 1.3. Coaching: SU, as part of its MEL practices, should ensure that coaching provided by head teachers, SEIs, DEOs, and DAs is systematically documented and shared at the sector/district/provincial levels in order to inform both subsequent coaching by the same or different actors, and future training, coaching, and instructional practices. 9. Sub-IR 1.4. Student Assessment: Refresher training to DEOs, SEIs, head teachers, and teachers should be provided to guide how to do formative assessment deliberately and systematically. This training should address recording results, how to analyze results to inform instruction, and for coaches, how to provide guidance to teachers that informs instruction. Additionally, assessment results could be used to identify struggling schools in order to provide more targeted, extensive, and effective coaching attention. IR 2: SYSTEMIC CAPACITY FOR EARLY GRADE READING INSTRUCTION IMPROVED 10. Sub-IR 2.1. Advocacy: SU should establish a partnership with MINALOC so that important EGR messages are incorporated into the monthly national topics that are communicated during Umuganda. These may include advice to parents to adopt EGR best practices at home and in the community. SU should also explore collaboration with Mureke Dusome’s Umuganda literacy activities. 11. Sub-IR 2.2. Standards and Benchmarks: Once benchmarks have been applied to schools to categorize the students based on their varied Kinyarwanda literacy abilities, SU should ensure that DAs, DEOs, SEIs, and head teachers systematically monitor the quality of remediation teachers provide to struggling learners. SU should ensure that teachers are equipped to provide this support (including through differentiated instruction). 12. Sub-IR 2.3. Research-Based Policies: SU should support URCE to document evidence-based EGR best practices, and should develop policy briefs targeting a wide audience at national, district, and school levels to influence policy and practice. 13. Sub-IR 2.4. Assessment Systems: SU should carry out sensitization campaigns to inform district, sector, and school-based education stakeholders about the importance of benchmarks, of LEGRA as a tool to systematically monitor student literacy performance, and on the use of results to inform subsequent practices (rather than as a punitive measure). 14. Sub-IR 2.5. TTCs: SU should establish an MOU or similar agreement with URCE and TTCs so that involved parties formally commit to their tasks to mainstream EGR activities into pre-service teacher education curriculum. SU should develop systematic collaboration with URCE and be present during review of TTC curriculum in order to incorporate and institutionalize EGR best practices. 61 CONTRIBUTION TO GOR PRIORITIES AND LEARNING OUTCOMES 15. SU should document the approach taken to both directly support GoR priorities and to establish a close working relationship with REB, while also building GoR capacity. These best practices should be shared nationally and internationally through policy briefs and/or other means to inform other donor and government initiatives. INEQUITIES BASED ON GENDER AND DISABILITIES 16. SU should produce or adapt specific materials for addressing special needs in the classroom, and go beyond merely creating awareness. These materials should provide teachers and school staff with strategies for supporting learning for students with special needs. The materials can also include information on how teachers can request accessible materials for children with specific disabilities (e.g. Braille reading materials). 17. SU should explore forming partnerships with Department of Special Needs Education at URCE, coordinating with BLF (if feasible), since BLF has SEN coordinators who graduated from URCE and work in the BLF pilot districts. ORGANIZATIONAL STRUCTURE AND STAFF 18. For the remainder of the activity, SU should aim to ensure that DEOs, SEIs, and head teachers are equipped to sustain SU interventions. To do this, SU should consider providing staff to support DAs in capacity building of DEOs and SEIs to conduct coaching visits, hold COPs, and carry out Monitoring and Evaluation (M&E) data collection within the district. 19. SU should hire a MEL/CLA specialist for each province to systematically capture and document M&E data from schools, sectors, and districts, monitor coaching and COP reports, LEGRA results, and formative assessment results. The specialist would be responsible for synthesizing province￾wide data/information from schools, sectors, and districts at the end of each term. IMPLEMENTATION CHALLENGES 20. To help ensure that the SU materials and approach are effectively delivered, specific training and guidelines should be provided to head teachers and teachers on how to engage students despite large class size (e.g. student group work, student peer teaching, students writing part of a story, independent work). 21. Teacher trainings should include time for teachers to discuss their concerns about instructional strategies (such as covering all of the content in 40 minutes). 22. A condensed training on Sunday for the Seventh Day Adventist head teachers, teachers DEOs and SEIs should be provided. 23. DAs should support SEIs and head teachers to provide information for parents through the SGACs, including a guidance pamphlet on how to support their child’s literacy development. The pamphlet can suggest simple things for parents to do with their children to help them develop literacy skills. It should include guidance on how to care for student textbooks and equipping students with materials (e.g., with book bags, notebooks, and pencils/pens). Partnering with Mureke Dusome should be considered for this activity. 24. SU staff should draft templates for letters and/or memos that DEOs can use to discourage transfers of Kinyarwanda teachers trained by SU. 62 SUSTAINABILITY 25. SU should work to advance its goals and activities, including early grade literacy rates and COPs, into the strategic plans and performance contracts (Imihigos) of the sectors and districts, and mainstreamed into the DEO, SEI, and head teacher Imihigo requirements. DAs should ensure that the head teachers, DOSs and teachers are made aware of these requirements. 26. SU should prioritize COPs as a means of making SU interventions sustainable, thus ensuring capacity to identify evidence-based EGR practices, document and analyze shortcomings, and use M&E and assessment data to inform subsequent action. Furthermore, after all the COPs have been established and are operating, SU should identify strengths and challenges of COPs and adapt them accordingly to increase sustainability. 27. A year before the SU activity ends, SU should support the REB in planning for the reprint and redistribution of the P1-P3 teaching and learning materials, assuring that sufficient resources will be developed and provided, as needed. This planning should also include exploring varied avenues of support (e.g., the World Bank and other donors). COLLABORATION, LEARNING, AND ADAPTING 28. USAID and SU should maximize potential opportunities with Mureke Dusome for joint advocacy on common causes, such as enhanced public awareness of inclusive education and increasing parental engagement in children’s literacy development, and care of textbooks and other educational resources. 29. SU should identify specific points of convergence with BLF such as 1) comprehensive assessment, 2) COPs, 3) school leadership, and 4) SEI’s roles and responsibilities. Further, SU should establish action plans for joint interventions and collaboration. 30. SU should hire a CLA and/or knowledge management specialist/advisor at a central level to improve collection and management of internal information, consolidate and synthesize information from all levels (central, provincial, district, sector, school), document lessons learned to inform adaptation, and conduct regular learning reviews. 31. To inform adaptation and build capacity to use data for informing practice, the PA or a provincial CLA specialist should lead provincial and district-level CLA meetings at the end of each term to analyze M&E data, including information from coaching and COP reports with district and sector staff and head teachers. 63 REFERENCES Piper, B., Jepkemei, E., Kwayumba, D., & Kibukho, K. “Kenya’s ICT Policy in Practice: The Effectiveness of Tablets and E-readers in Improving Student Outcomes.” FIRE: Forum for International Research in Education 2, issue 1 (2015): http://dx.doi.org/10.18275/fire201502011025. Republic of Rwanda, Ministry of Finance and Economic Planning. Rwanda Vision 2020. Kigali: Ministry of Finance and Economic Planning, (2000): https://www.sida.se/globalassets/global/countries-and￾regions/africa/rwanda/d402331a.pdf. Room to Read. A Study on Kinyarwanda Instructional Time in Lower Primary. Draft Version; July 19, 2019. Rwanda Ministry of Education. Education Sector Strategic Plan (ESSP). Kigali: Rwanda Ministry of Education, (2013): https://www.globalpartnership.org/content/rwanda-education-sector-strategic-plan-2010- 2015. UNESCO Institute for Statistics. Education: Net enrolment rate by level of education. Accessed May 9, 2018. http://data.uis.unesco.org/#. USAID Rwanda. Country Development Cooperation Strategy. Kigali: USAID, (2015): https://www.usaid.gov/si tes/default/files/documents/1860/Rwanda_CDCS_2015-2020.pdf. World Bank. Global Monitoring Report 2015/2016: Development Goals in an Era of Demographic Change. Washington, DC: World Bank Group, (2016): https://www.worldbank.org/en/publication/global￾monitoring-report. World Bank. The Human Capital Project. Washington, DC: World Bank Group, (2018): https://openknowl edge.worldbank.org/handle/10986/30498. 64 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK The Soma Umenye activity, under the LEARN project, is a $72.4 million, five-year initiative (7/2016- 7/2021) of USAID and the Rwanda Education Board (REB) that aims to improve the Kinyarwanda language reading skills of at least one million children nationwide in public and government-funded schools by the end of their third year of primary school. Specifically, Soma Umenye seeks to ensure that at least 70 percent of Primary Grade 1(P1), Primary Grade 2 (P2), and Primary Grade 3 (P3) students are able to read grade-level text with fluency and comprehension. The Government of Rwanda's (GoR) ambitious development agenda, which aims at becoming a middle￾income, knowledge-based economy by 2020, emphasizes the development of human capital to spur economic growth. The Vision emphasizes strengthening human capital formation as one of the six strategy pillars, particularly through enhanced education service delivery. Efforts to improve the education system in Rwanda are focused on ensuring universal education for all, improving education quality, and expanding technical and vocational training. Soma Umenye strives for close collaboration with GoR counterparts, including through alignment of Soma Umenye activities with the GoR’s Education Sector Strategic Plan (ESSP) 3, as well as with district education plans. To improve early grade reading, Soma Umenye promotes changes to the “5Ts” of reading: time, text, teaching technique, test, and tongue. Specifically, Soma Umenye aims to improve student reading by: • Improving quality of classroom instruction at the primary level through the application of teacher and student performance standards, and the implementation of an evidence-based in- service training and coaching program • Strengthening the pre-service teacher training program • Increasing availability and use of evidence-based and gender-sensitive early grade reading materials • Strengthening the capacity of the Ministry of Education and University of Rwanda to conduct and apply research, and lead the reading reform process Soma Umenye is designed to build the capacity of people and systems in Rwanda to adopt and sustain improvements in reading, including providing equitable opportunities for professional development and advancement for female teachers and relevant stakeholders. USAID determined a performance evaluation of Soma Umenye to be an important tool to assess challenges and barriers to activity success. Specifically, the performance evaluation will examine the effectiveness of the recent adjustments to activity implementation to better reflect the current orientation and plans for Soma Umenye in Years 2- 3. This includes assessing the performance of the implementing partner and gathering stakeholder perceptions, in order to identify areas for improvement or adjustment. 65 ANNEX II: EVALUATION METHODS AND LIMITATIONS GETTING TO ANSWERS MATRIX (G2A) Evaluation Question Source(s) Methods Analysis 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? a. What progress has Soma Umenye made toward the stated goals and objectives of the contract? i. What progress has been made towards each of the sub-IRs (as shown in the table above)? ii. Is Soma Umenye viewed as positively contributing to GoR priorities and learning outcomes among GoR stakeholders? iii. To what extent has Soma Umenye addressed inequities based on gender and students and teachers with disabilities? iv. Is Soma Umenye making effective use of its organizational structure and staff? Soma Umenye contract and activity documents, including monitoring data USAID staff, IP personnel (both upper level management and district advisors), local partners, GoR officials, local/district officials, head teachers, teachers Desk review, KIIs, FGDs, classroom observation Content analysis, thematic coding, Convergence/ divergence determinations b. Have teachers incorporated best practices from Soma Umenye’s trainings into their classrooms? If not, what have been the key impediments to doing so? How has Soma Umenye responded to those challenges? i. How do teachers perceive Soma Umenye interventions? Do teachers perceive that the Soma Umenye interventions have had an impact on their teaching techniques and their students’ learning? Training materials, sub- grantees, teachers, activity performance monitoring data Teachers, head teachers, local/district officials, , partners, IP staff (including district advisors), Soma Umenye Instructional Time Study Desk review, KIIs, FGDs, classroom observation Content analysis, thematic coding, Convergence/ divergence determinations 66 Evaluation Question Source(s) Methods Analysis 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? a. National level challenges i. Has Soma Umenye addressed barriers to greater GoR uptake of evidence-based approaches? b. District, sector, school-level challenges Soma Umenye activity documents, including quarterly and annual reports USAID staff, IP personnel (both upper level management and district advisors), partners, GoR officials, local/district officials, head teachers, teachers Desk review, KIIs, FGDs, classroom observation Content analysis, thematic coding, Convergence/ divergence determinations 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? a. How does Soma Umenye define sustainability? b. Is there currently demonstrable ownership and demand for Soma Umenye interventions? How could Soma Umenye further promote ownership and demand? (covering the following stakeholders) i. GoR officials ii. Local administrators iii. Teachers iv. Local NGO partners c. What is the current level of skills and capacity among stakeholders to sustain Soma Umenye interventions? How could Soma Umenye further build up skills and capacity among stakeholders? (covering the following stakeholders) i. GoR officials ii. Local administrators • What role are Soma Umenye district advisors playing in developing the skills and capacity of local administrators? Soma Umenye activity documents, including quarterly and annual reports USAID staff, IP personnel (both upper level management and district advisors), partners, GoR officials, local/district officials, head teachers, teachers Desk review, KIIs, FGDs Content analysis, thematic coding, Convergence/ divergence determinations 67 Evaluation Question Source(s) Methods Analysis iii. Teachers iv. Local NGO partners d. Given available resources of GoR and local organizations, which Soma Umenye interventions are most sustainable? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Does existing evidence indicate the effectiveness of any changes made? a. Is Soma Umenye effectively collaborating with USAID, GoR, and other development partners in Rwanda? How could their collaboration be improved? b. How is Soma Umenye using assessments (both formative and summative) to spur adaptation? c. What opportunities exist to strengthen collaborating, learning, and adapting (CLA) practices in the activity? Soma Umenye activity documents, including quarterly and annual reports USAID staff, IP personnel (both upper level management and district advisors), partners, GoR officials, local/district officials Desk review, KIIs, FGDs Content analysis, thematic coding, Convergence/ divergence determinations 68 ANNEX III: DATA COLLECTION INSTRUMENTS Soma Umenye Performance Evaluation Classroom Observation Protocol Date_________ Name of facilitator(s)__________________________________________ School name____________________ District_______________ Sector____________ Instructions for the observer: ● Introduce yourself and tell the teacher you would like to watch her/his literacy lesson in order to understand more about how the SU activity is carried out in the classroom. Tell her that it is anonymous and the name of the teacher will not be noted anywhere. Ask if she/he gives you permission to do the observation. If granted permission, ask the teacher where in the classroom would be the least distracting for the class. Ask the teacher how many students are absent then count how many are present and note the number. ● Complete the observation form during the lesson, except for the final box for other observations, impressions and notes where you write down anything noteworthy that was not captured by the instrument. Observe for at least 30 minutes, or the duration of the lesson. School Information Name: District: Sector: Level (circle): P1 P2 P3 Shift (circle): AM PM Class Observed Students: # Girls present_______ # Boys present_______ # Students absent_______ Unit: Lesson number: Did you speak to the teacher prior to the lesson? ___Yes ___ No For how long?_______ Time lesson started:______ Time lesson ended:_________ # Minutes late:________ Reason lesson started late (if applicable): Lesson Preparation Did the lesson have a clear learning objective? __ Yes __ No If yes, did the teacher share the learning objective with students? __ Yes __ No 69 Lesson Preparation How much of the lesson did the teacher use the teacher guide? __ The entire lesson __ Most of the lesson ___ Half of the lesson ___Not much of the lesson __ Not at all When/for what didn’t she/he? When/for what did she/he? Methodology All students participated in the lesson activities __ Yes ___ No Notes: The teacher equally encouraged participation of boys and girls ___ Yes ___ No Notes: The teacher equally encouraged participation of students with special needs ___ Yes ___ No __ N/A Notes: The teacher used gender and special needs inclusive language __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: Teaching and learning aids were used in the lesson __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: 70 Methodology The teacher used the “I do, We do, You do!” methodology __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: The teacher followed the lesson methodology and times as written in the teacher guide __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: All learners were given the opportunity to use books during the lesson __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: The teacher gave positive verbal reinforcement/appreciation to learners __0 times ___ 1-2 times __ 3-4 times ___ 5-6 times ___ More than 6 times Notes: Literacy Skills Which core literacy skills were taught in the lesson? __ Phonological awareness ___ Phonics ___ Vocabulary __ Reading fluency ___ Comprehension For teaching the first literacy skill which was __________________ the following took place: For teaching the second literacy skill which was_________________ the following took place: 71 Literacy Skills For teaching the third literacy skill which was________________ the following took place: Total minutes that students spent independently reading during the lesson ____ What were they instructed to read? __________________________________ What were they instructed to write?___________________________________ Student Engagement How many students are engaged with learning (paying attention to teacher or learning task) after lesson has been in progress for: 5 minutes # of students____ 10 minutes # of students____ 15 minutes # of students____ 20 minutes # of students____ 25 minutes # of students___ Number of times during the lesson that teacher directed students to use SU materials_____ Does every student have his/her own book? Yes___ No____ Explain (which book and how many students share): If some students are not engaged during the lesson, what were they doing? Did the teacher do something to get them to engage with the lesson? __ Yes __ No If yes, what? Number of students actively engaging with SU materials when directed to do so (first time): ____ Number of students actively engaging with SU materials when directed to do so (second time): ____ Number of students actively engaging with SU materials when directed to do so (third time): ____ Note: Active engagement with materials means students opening their books to the correct page, following teacher as she/he reads, and/or doing the task in the materials they are told to). 72 Student Engagement If some students are not engaged with the materials, what were they doing? Did the teacher do something to get them to engage with the materials? __ Yes __ No If yes, what? Assessment/Evaluation of Learners The teacher checked to see if learners understood what they were learning __ Very frequently ___ Frequently __ Some ___ Moderately ___ Not at all First literacy skill ______________ was assessed in the following way: Second literacy skill ______________ was assessed in the following way: Third literacy skill ______________ was assessed in the following way: Assessment was inclusive of both genders and special needs students ___ Yes ___ No If yes, how? Learners were given homework ___ Yes ___ No Learners were encouraged to practice what they had learned at home ___ Yes ___ No 73 Teaching and Learning Materials Teaching and learning materials that were visible in the classroom: __Chalkboard __ SU/REB read aloud story books __ SU/REB alphabet chart __ SU/REB Decodable readers __ SU/REB Pocket board __ SU/REB supplementary leveled readers __ SU/REB pupil textbooks __ SU/REB Andika Rwanda readers __ Class library __ Class diary __ Bookshelf __ Other(s): ________________ Notes: Teaching and learning materials that were used during the lesson: __ Chalkboard __ SU/REB read aloud story books __ SU/REB alphabet chart (__Teacher __ Student) __ SU/REB Decodable readers __ SU/REB Pocket board __ SU/REB supplementary leveled readers __ SU/REB pupil textbooks __ SU/REB Andika Rwanda readers __ Class library __ Class diary __ Other(s): ______________________________ Notes: Learners were encouraged to take a book or other learning material home __ Yes __ No Notes: Student Work Based on looking at notebooks of 4 different students (2 boys, 2 girls), how many of them appear to have correctly understood the lesson? __All 4 __3 __ 2 __1 __ None of them Provide evidence/explanation for categorization: 74 Student Work Based on looking at notebooks of 4 different students, how many texts had they written?___ How many of these were original texts (not copying from chalkboard or book)?___ Provide evidence/explanation for categorization: Was the writing ability at the expected level for the grade? __ Higher than expected __ As expected __Lower than expected Provide evidence/explanation for categorization: Other Observations, Impressions and Notes (Note anything relevant not captured in the protocol, such as head teacher supervision or presence, anything unexpected that occurred, anything that struck you as interesting or puzzling, and overall general impressions of the lesson, teaching, and learning) 75 Soma Umenye Performance Evaluation Teacher Questionnaire Date__________ District_________________ Sector_______________ Instruction: Thank you for taking the time to fill out this questionnaire. We are part of a team working with Dexis Consulting Group to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and document lessons learned to improve future programming. Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. Teacher Information Grade 1 ___ Grade 2 ___ Grade 3 ___ Female ____ Male _____ Years of teaching experience: _____ Number of students you teach: _____ # of girls _____ # of boys _____ # of children with special education needs _____ Deaf or hard of hearing _____ Blind or difficulty seeing _____ Learning disability _____ Down syndrome _____ Autism _____ Repeaters _____ Other(s)_______________________________ Training and Coaching 1. How many trainings on early grade literacy have you attended this school year? _____ 2. How many trainings on early grade literacy did you attend last school year? _____ 3. Have you received training in how to use the teacher guide? Yes__ No__ Not sure__ If yes, has it been sufficient to use it effectively? Yes__ No__ Not sure__ 76 Training and Coaching 4. Have you received training in how to use the read aloud books? Yes__ No__ Not sure__ If yes, has it been sufficient to use them effectively? Yes__ No__ Not sure__ 5. Have you received training in how to use the student textbooks? Yes__ No__ Not sure__ If yes, has it been sufficient to use them effectively? Yes__ No__ Not sure__ 6. Have you received training in how to use the pocket board? Yes__ No__ Not sure__ If yes, has it been sufficient to use them effectively? Yes__ No__ Not sure__ 7. For any of the areas of training on materials mentioned above that you have had insufficient training, please explain/elaborate: 8. Have you received training on the importance of teaching each core literacy skill? Explain/example. Yes__ No__ Not sure__ If yes, has it been sufficient to do this effectively? Yes__ No__ Not sure__ 9. Have you received training on literacy instructional practices? Explain/example. Yes__ No__ Not sure__ If yes, has it been sufficient to do them effectively? Yes__ No__ Not sure__ 10. Have you received training to do formative assessment of students? Explain/example. Yes__ No__ Not sure__ If yes, has it been sufficient to do this effectively? Yes__ No__ Not sure__ 11. Have you received training on teaching with gender sensitivity? Explain/example. Yes__ No__ Not sure__ If yes, has it been sufficient to do this effectively? Yes__ No__ Not sure__ 12. Have you received training on inclusive practices for special education needs? Yes__ No__ Not sure__ If yes, has it been sufficient to do this effectively? Yes__ No__ Not sure__ 77 Training and Coaching 13. For any of the areas of training on instructional practices mentioned above that you have had insufficient training, please explain/elaborate: Literacy Instructional Practices 14. How many times per week do you read a story to your students? 5 times__ 4 times__ 3 times__ 2 times__ Never__ If never, explain __________________ When you do, do you ask your students questions about the story? Always__ Sometimes__ Never___ If never, explain_______________________________________________________________ 15. How many times per week do you ask your students to read on their own in class? 5 times__ 4 times__ 3 times__ 2 times__ Never__ If never, explain __________________ 16. How many times per week do you ask students to write on their own in class? 5 times__ 4 times__ 3 times__ 2 times__ Never__ If never, explain __________________ 17. How many times do you give reading homework? 5__ 4__ 3__ 2__ 1__ Never__ If never, explain________________________ 18. How many times do you give writing homework? 5__ 4__ 3__ 2__ 1__ Never__ If never, explain________________________ Materials 19. How often do you and your students use the following materials during literacy lessons? ELL= Every literacy lesson SLL= Some literacy lessons NLL= No literacy lessons Teacher guide ELL ___ SLL___ NLL___ Pupil text books ELL ___ SLL___ NLL___ Pocket board ELL ___ SLL___ NLL___ Alphabet chart: ELL ___ SLL___ NLL___ Read Aloud storybooks: ELL ___ SLL___ NLL___ Decodable readers ELL ___ SLL___ NLL___ 78 Materials Class library ELL ___ SLL___ NLL___ Read aloud story book ELL ___ SLL___ NLL___ Other___________________ ELL ___ SLL___ NLL___ 20. Which of these Soma Umenye supported materials do you currently have in your classroom? Teacher guide___ Pupil text book___ Read aloud books___ Pocket board___ Class library__ 23. Which of the materials that you have do you use the most frequently? Why? 24. Do you have sufficient materials? (teacher guides, pupil textbooks, read aloud storybooks, etc.)? Yes____ No____ Explain/specify: 25. Which of the Soma Umenye materials will you continue to use even if other support from SU doesn’t continue? Teacher guide___ Pupil text book___ Read aloud books___ Pocket board___ Class library__ Explain: 79 Coaching 26. How often is your class observed by the Head Teacher or Dean of Studies (DOS)? Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 27. Who observes your class? Head Teacher____ DOS____ SEO____ Other ____ Specify other: 28. How often do you receive coaching? Twice a month___ Once a month___ 2-3 times per year___ Once per year___ Not at all___ Other___ Specify other: 29. Who provides coaching for you? Head Teacher___ DOS___ Other____ Specify other: 30. Is the amount of coaching you receive sufficient for you to carry out effective instructional practices? Yes____ No____ Explain/elaborate: 31. How often do you receive refresher trainings by your coach or someone else (if someone else, specify_______________): Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 33. If yes, how often do you meet to discuss teaching and learning and or related topics? Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 80 Coaching 34. How helpful is the COP for improving your literacy instructional practices? Very helpful___ Helpful___ Somewhat helpful____ Not very helpful____ Not at all helpful____ Explain: Assessment 35. When was the last time you carried out a formative assessment of your students? _____ 36. How often do you do formative assessment of your students? Once a week__ Twice a month___ Once a month__ Every two months___ Twice a year____ Other___ Specify other: 81 Soma Umenye Performance Evaluation Teacher Focus Group Guide Date_________ Name of facilitator(s)__________________________________________ School name____________________ District_______________ Sector____________ Instructions for Facilitator: Sit in a place where they can talk freely without being heard by the Head Teacher or students. Explain the following: ● The purpose of the evaluation is to better understand how the new teaching materials and practices are being put into practice as well as successes and challenges thus far. ● The focus group discussion should take no more than an hour. ● All of the information is confidential and no names will be shared. ● We would like to know if they are willing to participate. ● We would like to record the discussion for the purpose of assuring we capture everything and that it is only for the evaluation team and we request your permission to do so. Use the protocol to note everything by teacher. For open ended response questions, differentiate response by teacher by noting next to the response “T1”, “T2”, “T3”, etc. Teacher information # of teachers ____ # Female ____ # Male ____ # Grade 1 teachers ____ # Grade 2 teachers ____ # Grade 3 teachers ____ Years teaching T1 ____ T2 ____ T3 ____ T4 ____ T5 ____ T6 ____ Training and Coaching 1. How many trainings on EGR have you attended this school year?______ T1 ____ T2 ____ T3 ____ T4 ____ T5 ____ T6 ____ 2. How many trainings on EGR did you attend last school year? T1 ____ T2 ____ T3 ____ T4 ____ T5 ____ T6 ____ 3. Have you received training in how to use the teacher guide? (# of teacher who indicate yes/no) Yes___ No___ If yes, has it been sufficient to use it effectively? Yes___ No___ 82 Training and Coaching 4. Have you received training in how to use the read aloud books? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to use them effectively? Yes___ No___ 5. Have you received training in how to use the student textbooks? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to use them effectively? Yes___ No___ 6. Have you received training in how to use the leveled readers? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to use them effectively? Yes___ No____ 7. Have you received training in how to use the pocket board? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to use them effectively? Yes___ No____ 8. Have you received training in how to use the class diaries? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to use them effectively? Yes___ No____ 9. For any of the areas of training mentioned above that you have had insufficient training, please explain/elaborate: 10. Have you received training on the core literacy skills? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to do this effectively? Yes___ No____ 11. Have you received training on literacy instructional practices? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to do them effectively? Yes___ No____ 12. Have you received training to do formative assessment of pupils? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient do this them effectively? Yes___ No____ 83 Training and Coaching 13. Have you received training on teaching with gender sensitivity? (# of teacher who indicate yes/no) Yes___ No____ If yes, has it been sufficient to do this effectively? Yes___ No____ 14. Have you received training on inclusive practices for special education needs? (# of teacher y/n) Yes___ No____ If yes, has it been sufficient to do this effectively? Yes___ No___ 15. For any of the areas of training mentioned above that you have had insufficient training, please explain/elaborate: Literacy Instructional Practices 16. Which new literacy instruction methods/practices have been the easiest to put into practice? Why? 84 Literacy Instructional Practices 17. Which new literacy instruction methods/practices have been the most challenging to put into practice? Why? 18. Which new literacy instruction methods/practices seem to be the most effective for your pupils to learn to read and write? Gender and Inclusion 19. What do you do to address gender equity with your pupils? (# teachers who give each response) Ensure boys and girls equally participate ____ Talk about gender equity with pupils ____ Use gender sensitive materials ____ Use gender sensitive language ____ Other____ (please specify) 20. What do you do to ensure equal learning opportunities for special educational needs (SEN) pupils? (# teachers who give each response) Include SEN pupils in all activities ____ Equally encourage participation of SEN pupils ____ Use language that is SEN sensitive ____ Use materials that are SEN sensitive ____ Other____ (please specify) 85 Materials 21. How often do you and your students use the following materials during literacy lessons? ELL= Every literacy lesson SLL= Some literacy lessons NLL= No literacy lessons * Next the corresponding letter write the number of teachers who gave that response Teacher guide ELL ___ SLL___ NLL___ Pupil text books ELL ___ SLL___ NLL___ Pocket board ELL ___ SLL___ NLL___ Alphabet chart: ELL ___ SLL___ NLL___ Read Aloud Books: ELL ___ SLL___ NLL___ Decodable readers ELL ___ SLL___ NLL___ Supplementary leveled readers ELL ___ SLL___ NLL___ Andika Rwanda readers ELL ___ SLL___ NLL___ Class diary ELL ___ SLL___ NLL___ Class library ELL ___ SLL___ NLL___ Other___________________ ELL ___ SLL___ NLL___ Other___________________ ELL ___ SLL___ NLL___ 22. Why do you use the frequently used materials the most? 23. Why do you use the least frequently used ones rarely or never? 86 Materials 24. The materials that are the most difficult to use: Teacher guide___ Pupil text book___ Read aloud books___ Decodable readers___ Supplementary readers___ Andika Rwanda readers___ Pocket board___ Class library__ 25. The materials that are the easiest to use: Teacher guide___ Pupil text book___ Read aloud books___ Decodable readers___ Supplementary readers___ Andika Rwanda readers___ Pocket board___ Class library__ 26. The materials that are the most helpful for teaching and learning: Teacher guide___ Pupil text book___ Read aloud books___ Decodable readers___ Supplementary readers___ Andika Rwanda readers___ Pocket board___ Class library__ 27. What would help you to use the materials more frequently and effectively? 28. If you could change something about any of the materials what would it be? 29. Do you have sufficient materials? (teacher guides, pupil textbooks, readers, etc.)? Yes____ No____ Explain/specify: Coaching and Community of Practices (COPs) 30. How often is your class observed by the Head Teacher or Dean of Studies (DOS)? Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 87 Coaching and Community of Practices (COPs) 31. Who observes your class? Head Teacher____ DOS____ Other____ Specify other: 32. How often do you receive coaching: Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 33. Who provides coaching for you? Head Teacher___ DOS___ Other____ Specify other: 34. Is the amount of coaching you receive sufficient for you to carry out effective instructional practices? Yes____ No____ Explain/elaborate: 35. What are the most helpful aspects of the coaching you receive? 36. What are the least helpful aspects of the coaching you receive? 37. Ways to improve coaching and other comments: 38. How often do you receive refresher trainings by your coach or someone else (if someone else, specify_______________): Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 39. Are you part of a Community of Practice (COP) with other teachers? Yes____ No____ 88 Coaching and Community of Practices (COPs) 39. If yes, how often do you meet to discuss teaching and learning and or related topics? Twice a month___ Once a month___ 2-3 times per year__ Once per year__ Not at all__ Other___ Specify other: 40. How helpful is the COP for improving your literacy instructional practices? Very helpful___ Helpful___ Somewhat helpful____ Not very helpful____ Not at all helpful____ Explain: 41. How could the COP be more helpful for improving your literacy instructional practices? Assessment 42. How do you know if your students have learned what you have taught them that lesson? Ask pupils to identify sounds/letters____ Ask pupils to answer questions____ Ask pupils to read sounds/words/sentences___ Ask pupils to write letters/words/sentences___ Have pupils repeat what they learned____ Have pupils explain what they learned____ Other____ Specify other: 43. When was the last time you carried out a formative assessment of your pupils? 44. How often do you do formative assessment of your pupils? Once a week__ Twice a month___ Once a month__ Every two months___ Twice a year____ Other___ Specify other: 45. How do you use the results of the assessments? 46. Challenges with assessment (ongoing, formative, summative) and other notes: 89 General comments of focus group facilitator(s) 90 Soma Umenye Performance Evaluation District Education Officer Interview Guide Date_________ District_________ Name of interviewer____________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? [ALLOW RESPONDENT TO CONFIRM PARTICIPATION] EQ1: Implementing Partner 1. Which of the Soma Umenye interventions are you familiar with, and how do they support the achievement of your District education priorities? 2. Have you worked with Soma Umenye to implement interventions in your District? If so, how? Probe: What activities have you been able to implement in your District to support Soma Umenye interventions? 3. How have Soma’s teacher trainings supported teachers in your District to implement improved EGR practices? Probe: Which desired practices do you find are easier for Head Teachers to master during your coaching sessions with them, and why? Probe: Which desired practices have you found to be the most difficult for Head Teachers to master during your coaching sessions with them, and why? 4. How has Soma Umenye supported your District in terms of providing teaching and learning materials? Which materials has Soma Umenye provided, and what has been their impact on improving literacy? Probe: Which materials have been the most helpful? The least helpful? 91 Probe: Have teachers had challenges in using any of the materials? Please explain. 5. How does Soma Umenye incorporate and address issues of gender and disability in their/your activities? Probe: What has worked best to address these issues? Probe: What, if any, challenges have there been in addressing these issues? How have they been addressed? EQ2: Implementation Challenges 6. What challenges (if any) has Soma Umenye experienced while implementing their interventions in your District? Probe: What challenges (If any) have you experienced in serving as a Soma Umenye Coach in your District, and what do you suggest should be done to overcome such challenges? Probe: Have you been able to implement coaching as per the frequency stipulated by Soma Umenye in view of other responsibilities you have as a DEO? 7. What have been the main challenges to the improvement of EGR in your District? Probe: Are these challenges District -specific or general? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? EQ 3: Sustainability 8. What efforts are being made within Soma Umenye to ensure sustainability? Probe: How has Soma Umenye built collaborative and meaningful relationships with your District over time? 9. Do you feel that your District has shown interest and ownership of Soma Umenye interventions? Probe: Can you provide specific examples of things you have done (if any) that demonstrate ownership and demand for Soma Umenye interventions at the District level? Probe: How could Soma Umenye further promote ownership and demand for its interventions at the District level? 10. What is the current level of skills and capacity of DEOs to sustain Soma Umenye interventions? Probe: Which stakeholders are most capable of sustaining Soma interventions? Why is this? Probe: How could Soma Umenye further build up skills and capacity among stakeholders at the District level? 92 11. Considering budget constraints at the District level, which Soma Umenye interventions are most sustainable and why? Which ones are less sustainable and why? EQ4: Collaboration, Learning, and Adaptation 12. How has Soma Umenye’s emphasis on formative assessment impacted EGR instruction and results in your sector? Probe: How have teachers in your District adjusted to an assessment approach that is formative? Probe: Have there been any challenges in doing formative assessment and/or using the results to inform teaching and learning? 13. Do schools in your District utilize reading benchmarks to gauge students’ performance in EGR over time? Do you think such benchmarks are important? Why? Probe: Are you aware of the recent EGR reading benchmarks that were developed by Soma Umenye? If yes, are you using them in schools in your district? If not, why? 14. How effectively has Soma Umenye collaborated with DEOs to promote improved EGR instruction at the District level? Probe: How could their collaboration be improved? 15. Do you belong to Communities of Practice with other DEOs in this sector? Probe: What is the role of COPs, and how do you think they have been used to improve EGR practices? Probe: What could make them more successful in improving EGR instruction? Recommendations 16. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye intervention overall? 93 Soma Umenye Performance Evaluation Donor/Partner Organization Interview Guide Date_________ Name of interviewer____________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? Do you consent to being recorded? [ALLOW RESPONDENT TO CONSENT TO PARTICIPATION AND RECORDING] 1. To begin, please tell me a bit about your role and your work with Soma Umenye EQ1: Implementing Partner 1. With regards to EGR, what would you say are your organizations top priorities that you are working to establish? 2. How does Soma Umenye work with you on implementing these priorities? Probe: How would you characterize your organizations working relationship with SU overall? 3. Have you worked with SU on developing materials and promoting their use in the classroom? If 94 so, how? Probe: Have there been any challenges in material development and/or their use by teachers and students? How have they been addressed? 4. Have you worked with SU on teacher trainings? If so, how effective would you say these trainings have been at promoting best practices in the classroom? Probe: Have there been any challenges or impediments to implementing these practices? How have they been addressed? 5. At the national and local levels, would you say that systemic capacity for improving EGR instruction has improved? Please give examples for each level. Probe: Have you worked to establish national advocacy mechanisms for EGR? What have been the successes and challenges of these efforts? Probe: Have performance standards and benchmarks for EGR been applied for teachers and students nationwide? How has this been done? Probe: Have you seen an improvement in strengthening EGR assessment systems? What have been the successes and challenges of this? Probe: Will the EGRA be used for improving EGR? How does this complement national standardized testing? 6. Do you incorporate and address issues of gender and disability in your work with EGR? Probe: Is this something that you have worked with SU on? EQ2: Implementation Challenges 7. What have been the main challenges or obstacles that you’ve faced with respect to EGR improvement? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? 95 Probe: Has SU had a role in addressing these challenges? 8. Have you seen greater uptake of evidence and evidence-based approaches with respect to EGR? Has this improved over time? Probe: Has SU contributed to efforts to promote the uptake of evidence-based approaches? EQ 3: Sustainability 9. Do you think that there is demonstrable demand for SU interventions? Probe: Where do you think this demand is strongest (school-level, district or sector level, central level)? 10. How do you think SU could work more effectively with you to promote sustainability? 11. Given the available resources of the GoR, which specific Soma Umenye interventions are most sustainable and why? EQ4: Collaboration, Learning, and Adaptation 12. How effective is the collaboration between SU, USAID, the GoR, and other development partners in Rwanda? Please give examples. Probe: Where is the collaboration the strongest? Between which actors? Probe: Are there any challenges in your collaboration with USAID, SU, or other donors? Recommendations 13. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye overall? 96 Soma Umenye Performance Evaluation GoR Interview Guide Date_________ Name of interviewer____________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? Do you consent to being recorded? [ALLOW RESPONDENT TO CONSENT TO PARTICIPATION AND RECORDING] 1. To begin, please tell me a bit about your role and your work with Soma Umenye. EQ1: Implementing Partner: 1. With regards to EGR, what would you say are the GoR’s top priorities that you are working to establish/achieve? 2. How does Soma Umenye work with you on implementing these priorities? Probe: How would you characterize Soma’s relationship with the GoR overall, and in: • Training teachers • Training DEOs/SEIs • Providing Textbooks in schools 97 Probe: How (if at all) have Soma Umenye interventions complemented the GoR’s efforts or have there been any duplications with any pre-existing interventions? 3. Have you or any of your staff worked with SU on developing materials and promoting their use in the classroom? If so, how? Probe: Have there been any challenges in material development and/or their use by teachers and students? How have they been addressed? Probe: Have materials been distributed in schools as per your expected timelines? If know, why and how has the delay been addressed 4. Have you worked with SU on teacher trainings? If so, how effective would you say these trainings have been at promoting best practices in the classroom? Probe: How (if at all) are you involved in monitoring the effectiveness of these trainings? Probe: Have there been any challenges or impediments to implementing these practices? How have they been addressed? 5. At the national and local levels, would you say that systemic capacity for improving EGR instruction has improved? Please give examples for each level. Probe: Have you worked to establish national advocacy mechanisms for EGR? What have been the successes and challenges of these efforts? Probe: Have performance standards and benchmarks for EGR been applied in student assessment practices in schools, at District or national level? if so, how has this been done? Probe: Have you seen an improvement in strengthening EGR assessment systems? What have been the successes and challenges of this? Probe: Will the EGRA be used for improving EGR? How does this complement national standardized testing? 98 6. How (if at all) do you incorporate and address issues of gender and disability in your work with EGR? Probe: Is this something that you have worked with SU on? EQ2: Implementation Challenges 7. What have been the main challenges or obstacles that you’ve faced with respect to EGR improvement? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? Probe: Has SU had a role in addressing these challenges? If yes, how? 8. How does the GoR use evidence and evidence-based approaches with respect to EGR? Has this improved over time? Probe: Has SU contributed to efforts to promote the uptake of evidence-based approaches? If so, how? EQ 3: Sustainability 9. Are there efforts being made in the GoR to ensure sustainability of SU’s interventions? Examples? 10. Do you think that there is demonstrable demand for SU interventions? Probe: Where do you think this demand is strongest (school-level, district or sector level, central level)? 11. How do you think you could work more effectively with you to promote sustainability? 12. Given the available resources of the GoR, which specific Soma Umenye interventions are most sustainable and why? 13. What are you doing to ensure that successful interventions are sustained long after the project life time? EQ4: Collaboration, Learning, and Adaptation 99 14. How effective is the collaboration between SU, USAID, the GoR, and other development partners in Rwanda? Please give examples. Probe: Where is the collaboration the strongest? Between which actors? Probe: Are there any challenges in your collaboration with USAID, SU, or other donors? Probe: How are you addressing the challenges ( if any)? 15. Have the Soma Umenye interventions been adapted since they began? If so, please mention some examples. Recommendations 16. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye overall? 100 Soma Umenye Performance Evaluation Head Teacher Interview Guide Date_________ District_________ Name of interviewer____________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? [ALLOW RESPONDENT TO CONFIRM PARTICIPATION] EQ1: Implementing Partner: 1. Which of Soma Umenye interventions are you familiar with, and how do they support the achievement of key learning and teaching priorities in your school? 2. Have you worked with Soma Umenye to implement their interventions in your school? If so, how? Probe: What aspects do you focus on when you are doing teacher supervision and why? Probe: Have your actions included trying to engage parents/community to work towards improved EGR practices? Please give examples. 3. How have Soma Umenye trainings prepared you to become a coach/mentor? Probe: What is your exact role as a coach to teachers in your sector? Probe: How often do you conduct teacher coaching, and how often do you think teachers should ideally be coached? Probe: What challenges (If any) have you experienced in serving as a Soma Umenye Coach in 101 your school? What do you suggest should be done to overcome the stated challenges? Probe: Have you been able to implement coaching as per the frequency stipulated by Soma Umenye? Have there been any challenges associated with this, in view of other responsibilities you have as a coach for teachers? Probe: Do you have any other challenges in providing coaching? If so what are they, and how do you suggest overcoming them? 4. How effective do you think Soma’s teacher trainings have been in supporting teachers in your school to implement improved EGR practices? Probe: What practices have you seen the teachers be able to put into practice the most? Probe: What practices have you seen the teachers struggling to put into practice? 5. How has Soma Umenye supported your sector in terms of providing teaching and learning materials? Which materials has Soma Umenye provided and what has been their impact on improving literacy? Probe: Which materials have been the most helpful? The least helpful? Probe: Have teachers had challenges in using any of the materials? Please explain. 6. How has Soma Umenye supported your sector in terms of providing teaching and learning materials? What materials has Soma Umenye provided and what has been their impact on 7. How (if at all) are EGR priorities reflected in teacher performance pledges (Imihigo)? Probe: What is the process for developing teacher performance pledges (Imihigo)? Probe: What desired practices do you find appropriate to include in Imihigo? 8. How does Soma Umenye incorporate and address issues of gender and disability in their/your activities? Probe: What has worked the best to address these issues? Probe: What, if any, challenges have there been in addressing these issues? How have they been addressed? EQ2: Implementation Challenges: 9. What challenges (if any) has Soma Umenye experienced while implementing their interventions in your school? 10. What have been the main challenges to the improvement of EGR in your school? Probe: Are these challenges sector-specific or general? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? 102 EQ 3: Sustainability 11. What efforts are being made within Soma Umenye to ensure sustainability? Probe: How has Soma Umenye built collaborative and meaningful relationships with your school over time? 12. Do you feel that your school has shown interest and ownership of Soma Umenye interventions? Probe: Can you provide specific examples of things you have done (If any) that demonstrable ownership and demand for Soma Umenye interventions at school level? Probe: How could Soma Umenye further promote ownership and demand for its interventions at school level? 13. What is the current level of skills and capacity of Head Teachers to sustain Soma Umenye interventions? Probe: Which stakeholders are most capable of sustaining Soma interventions? Why is this? Probe: How could Soma Umenye further build up skills and capacity among stakeholders at the school level? 14. Considering budget constraints at the school-level, which Soma Umenye interventions are most sustainable and why? Which ones are less sustainable and why? EQ4: Collaboration, Learning, and Adaptation 15. How has Soma Umenye’s emphasis on formative assessment impacted EGR instruction and results in your school? Probe: How have teachers in your school adjusted to an assessment approach that is formative? Probe: Has your school made significant progress in using EGR results to inform further learning? Probe: Have there been any challenges in doing formative assessment and/or using the results to inform teaching and learning? 16. Do you have reading benchmarks that can be used to gauge students’ performance in EGR over time? Do you think such benchmarks are important? Why? Probe: Are you aware of the recently EGR reading benchmarks that were developed by Soma Umenye? If yes, are you using them in schools in your sector? If not, why? 17. How effectively has Soma Umenye collaborated with Head teachers to promote enhanced EGR instruction at the school level? Give concrete examples. Probe: How could their collaboration be improved? 103 18. Do you belong to Communities of Practice with other Head teachers in this sector? Probe: What is the role of COPs ,and what can make them successful in improving EGR instruction? Probe: How has the COP been helpful for you for working towards improved EGR practices? Recommendations 19. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye intervention overall? 104 Soma Umenye Performance Evaluation Parent Focus Group Guide Date_________ Name of facilitator(s)__________________________________________ School name____________________ District_______________ Sector____________ Instructions for Facilitator: Sit in a place where they can talk freely without being heard by the Head Teacher, teachers, or students. Explain the following: ● The purpose of the evaluation is to better understand how the new teaching materials and practices are being put into practice as well as successes and challenges thus far. ● The focus group discussion should take no more than an hour. ● All of the information is confidential and no names will be shared. ● We would like to know if they are willing to participate. ● We would like to record the discussion for the purpose of assuring we capture everything and that it is only for the evaluation team and we request your permission to do so. For each question, make note of each response and how many parents shared that perspective/agreed. Number of parents/family members present: Women______ Men_____ 1. Has someone explained what the goals and interventions of Soma Umenye’s support are? If so, who? Probe: What is your understanding of these goals? Probe: What is your understanding of how these goals are to be achieved? 2. Has someone talked with you about how parents/family members can support the improvement of the children’s EGR? If so, who? Probe: What is your understanding of your role in supporting children’s EGR? 3. What have you been able to do to support children’s EGR skills? Probe: What challenges have you encountered in supporting children’s EGR skills? 4. Which of the Soma Umenye-supported reading materials are you familiar with? Probe: What do you like about these materials? 105 Probe: What would you recommend regarding other/different materials that are key to improving EGR? 5. Since Soma Umenye began supporting the school, have you seen any changes with your child/children’s reading ability? Please explain. Probe: How about reading practices, such as frequency or content? Probe: If you have, what do you think has been the most helpful for making this happen? 6. Since Soma Umenye began supporting the school, have you seen any changes with your child/children’s writing ability? Please explain. Probe: How about writing practices, such as frequency or content? Probe: If you have, what do you think has been the most helpful for making this happen? 7. Do teachers share with you how your child/children’s learning is progressing? If so, how often and how do they do this? 8. Has your child ever brought home a storybook or reader from the school? If so, how many times more or less? Please specify if storybook, reader, or other. 9. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye overall? 106 Soma Umenye Performance Evaluation Sector Education Officer Interview Guide Date_________ Sector_________ Name of interviewer____________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? [ALLOW RESPONDENT TO CONFIRM PARTICIPATION] \ EQ1: Implementing Partner: 1. Which of the Soma Umenye interventions are you familiar with and how do they support the achievement of your sector education priorities? 2. Have you worked with Some Umenye to implement interventions in your sector? If so, how? Probe: What activities have you been able to implement in your sector to support Soma Umenye interventions? 3. How have Soma’s teacher trainings supported teachers in your sector to implement improved EGR practices? Probe: Which desired practices do you find it easy for Head Teachers to master during your coaching sessions with them and why? Probe: Which desired practices have you found to be the most difficult for Head teachers to master during your coaching sessions with them and why? 4. How has Soma Umenye supported your sector in terms of providing teaching and learning materials? Which materials has Soma Umenye provided and what has been their impact on improving literacy? 107 Probe: Which materials have been the most helpful? The least helpful? Probe: Have teachers had challenges in using any of the materials? Please explain. 5. How does Soma Umenye incorporate and address issues of gender and disability in their/your activities? Probe: What has worked the best to address these issues? Probe: What, if any, challenges have there been in addressing these issues? How have they been addressed? EQ2: Implementation Challenges: 6. What challenges (if any) has Soma Umenye experienced while implementing their interventions in your sector? Probe: What challenges (If any) have you experienced in serving as a Soma Umenye Coach in your sector, and what do you suggest should be done to overcome such challenges? Probe: Have you been able to implement coaching as per the frequency stipulated by soma Umenye in view of other responsibilities you have as a SEO? 7. What have been the main challenges to the improvement of EGR in your sector? Probe: Are these challenges sector-specific or general? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? EQ 3: Sustainability 8. What efforts are being made within Soma Umenye to ensure sustainability? Probe: How has Soma Umenye built collaborative and meaningful relationships with your sector over time? 9. Do you feel that your Sector and District have shown interest and ownership of Soma Umenye interventions? Probe: Can you provide specific examples of things you done ( If any ) that demonstrable ownership and demand for Soma Umenye interventions at sector/district level? Probe: How could Soma Umenye further promote ownership and demand for its interventions at the sector/district level? 10. What is the current level of skills and capacity of sector education inspectors to sustain Soma Umenye interventions? Probe: Which stakeholders are most capable of sustaining Soma interventions? Why is this? 108 Probe: How could Soma Umenye further build up skills and capacity among stakeholders at sector /district level? 11. Considering budget constraints at district/sector level, which Soma Umenye interventions are most sustainable and why? Which ones are less sustainable and why EQ4: Collaboration, Learning, and Adaptation 12. How has Some Umenye’s emphasis on formative assessment impacted EGR instruction and results in your sector? Probe: How have teachers in your sector adjusted to an assessment approach that is formative? Probe: Have there been any challenges in doing formative assessment and/or using the results to inform teaching and learning? 13. Do schools in your sector utilize reading benchmarks to gauge students’ performance in EGR over time? Do you think such benchmarks are important? Why? Probe: Are you aware of the recent EGR reading benchmarks that were developed by Soma Umenye? If yes, are you using them in schools in your sector? If not, why? 14. How effectively has Soma Umenye collaborated with Sector Education Inspectors to promote improved EGR instruction at the sector level? Probe: How could their collaboration be improved? 15. Do you belong to Communities of Practice with other SEOs in this sector? Probe: What is the role of COPs, and how do you think they have been used to improve EGR practices? Probe: What could make them more successful in improving EGR instruction? Recommendations 16. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye intervention overall? 109 Soma Umenye Performance Evaluation USAID/Soma Umenye Interview Guide Date__________ Name of interviewer_________________ Protocol: Thank you for taking the time to speak with us. My name is _____ and I am working with Dexis Consulting to evaluate the USAID-funded Soma Umenye activity. Our team is currently conducting an independent mid-term evaluation, which will be used to document successes, challenges and document lessons learned in order to improve future programming. The evaluation is centered around four central evaluation questions: 1. Implementing Partner: How has the implementing partner and its team performed against the terms of the contract? 2. Implementation Challenges: What are some challenges or obstacles to implementing the activity? How could those challenges be addressed? 3. Sustainability: What efforts are being made within Soma Umenye to ensure sustainability and how could Soma Umenye more effectively work towards sustainability? 4. Collaboration, Learning, and Adaptation: What evidence, if any, indicates that the activity has employed adaptive management approaches? Your answers to our questions will remain completely anonymous; our team will not share any personally identifiable information (PII) when reporting your opinions to USAID. We welcome your open and honest opinion on these issues to help the Soma Umenye activity and others like it be more effective in the future. We expect this conversation will take about one hour. Are you willing to participate? Do you consent to being recorded? [ALLOW RESPONDENT TO CONSENT TO PARTICIPATION AND RECORDING] 1. To start, please briefly explain your role and your work with Soma Umenye. EQ1: Implementing Partner: 2. Of all the interventions that Soma Umenye supports and carries out, which do you think have had the greatest impact on the goal of improving early-grade reading? Probe: How do Soma’s staff and organization contribute to the overall impact of the activity? 3. What successes and challenges has SU faced in improving classroom instruction in EGR? Probe: Has SU made the expected progress on developing materials and promoting their use in the classroom? Why or why not? Probe: How effective do you think Soma’s teacher trainings have been in establishing evidence￾based best practices in the classroom? Why or why not? Probe: Has the capacity of head and mentor teachers to coach and supervise EGR instructors improved? Why or why not? Probe: To what degree are schools and teachers using student assessment results? 110 4. To what degree has the SU activity contributed to building systemic capacity for improving EGR instruction? Probe: Have national advocacy mechanisms for EGR been established? Examples? Probe: Are performance standards and benchmarks being applied for teachers and students? Probe: Have research based EGR policies and/or curricula been implemented as a result of the SU activity? Probe: Have EGR assessment systems been strengthened? Probe: How, if at all, has SU helped strengthen the capacity of TTCs to prepare effective EGR teachers? 5. How does Soma Umenye contribute to the GoR’s priorities overall? 6. How does Soma Umenye incorporate and address issues of gender and disability in their/your activities? Probe: What, if any, challenges have there been in addressing these issues? How have they been addressed? EQ2: Implementation Challenges: 7. What have been the primary challenges or obstacles to implementing the activity? Probe: How have these challenges been addressed? If they haven’t yet, is there a plan to do so? 8. Have you had any challenges with the GoR uptake of evidence-based approaches with relation to EGR? Probe: Have the challenges to uptake been greater at the national or district/local levels? Probe: How have Soma Umenye grade- level reading benchmarks been mainstreamed into schools? EQ 3: Sustainability 9. How do you define sustainability in the context of Soma Umenye? 10. What efforts are being made within Soma Umenye to ensure sustainability of Soma’s interventions? 11. Do you feel that other stakeholders have shown demonstrable ownership and demand for Soma Umenye interventions? Probe: How could Soma Umenye further promote ownership and demand? 12. What is the current level of skills and capacity among stakeholders to sustain Soma Umenye interventions? 111 Probe: How could Soma Umenye further build up skills and capacity among stakeholders? Probe: Which stakeholders are most capable of sustaining Soma interventions? Why is this? 13. Given available resources of GoR and local organizations, which Soma Umenye interventions are most sustainable and why? EQ4: Collaboration, Learning, and Adaptation: 14. How has Soma employed adaptive management approaches? Does existing evidence indicate the effectiveness of any changes made? Probe: How is adaptive management built into SU planning, reporting, and routine operations? Probe: Does Soma Umenye use assessments to encourage adaptation? 15. How effectively does Soma Umenye collaborate with USAID (or vice versa), GoR, and other development partners in Rwanda? Please give examples. Probe: How could their collaboration be improved? 16. How do you think collaborating, learning, and adapting (CLA) practices in the activity could be improved? Recommendations 17. Is there anything you haven’t mentioned before that you would recommend to improve Soma Umenye overall? 112 ANNEX IV: SOURCES OF INFORMATION DOCUMENTS Republic of Rwanda, Ministry of Education. 2017. Education Sector Analysis. Republic of Rwanda, Ministry of Education. 2018. 2018 Education Statistics. USAID and Chemonics International, Inc (Chemonics). 2016a. Capacity Building Plan Soma Umenye Activity. USAID and Chemonics. 2016b. Disability and Inclusion Assessment Soma Umenye Activity. USAID and Chemonics. 2016c. Gender Assessment Soma Umenye Project. USAID and Chemonics. 2016d. Monitoring, Evaluation, and Learning Plan Soma Umenye Activity. USAID and Chemonics. 2016e. Quarterly Program Report Soma Umenye Project Quarter 4, Fiscal Year 2016. USAID and Chemonics. 2016f. Year 1 Work Plan Soma Umenye Activity. USAID and Chemonics. 2017a. Year 2 Work Plan USAID Soma Umenye Project. USAID and Chemonics. 2017b. Monitoring, Evaluation, and Learning Plan USAID Soma Umenye Project. USAID and Chemonics. 2017d. Quarterly Program Report Soma Umenye Activity Quarter 1, Fiscal Year 2017. USAID and Chemonics. 2017e. Quarterly Program Report Soma Umenye Activity Quarter 2, Fiscal Year 2017. USAID and Chemonics. 2017f. Quarterly Program Report Soma Umenye Activity Quarter 3, Fiscal Year 2017. USAID and Chemonics. 2017g. Quarterly Program Report Soma Umenye Activity Quarter 4, Fiscal Year 2017. USAID and Chemonics. 2017h. Soma Umenye Activity Annual Program Report Year 1. USAID and Chemonics. 2018a. Monitoring, Evaluation, and Learning Plan USAID Soma Umenye Project. USAID and Chemonics. 2018b. Quarterly Program Report USAID Soma Umenye Activity Quarter 2, Fiscal Year 2018. USAID and Chemonics. 2018c. Quarterly Program Report USAID Soma Umenye Project Quarter 1, Fiscal Year 2018. USAID and Chemonics. 2018d. Quarterly Program Report USAID Soma Umenye Project Quarter 3, Fiscal Year 2018. USAID and Chemonics. 2018e. Quarterly Program Report USAID Soma Umenye Project Quarter 4, Fiscal Year 2018. USAID and Chemonics. 2018f. Soma Umenye Activity Year 2 Annual Program Report. 113 USAID and Chemonics. 2018g. Year 3 Work Plan USAID Soma Umenye Project. USAID and Chemonics. 2019a. Quarterly Program Report USAID Soma Umenye Project Quarter 1, Fiscal Year 2019. USAID and Chemonics. 2019b. Soma Umenye Activity Quarterly Program Report Quarter 2, Fiscal Year 2019. USAID and International Business and Technical Consultants, Inc (IBTCI). 2018a. Impact Evaluation USAID/Soma Umenye Activity. Presentation Slides. USAID and IBTCI. 2018b. Soma Umenye Impact Evaluation- Year 1 Baseline Report Executive Summary. USAID and IBTCI. 2018c. Soma Umenye Impact Evaluation- Year 1 Baseline Report. USAID. 2016. Request for Task Order Proposal (RFTOP) Number SOL-696-16-000005 for Literacy Enhanced Across Rwanda Now (LEARN). SITES Province District Sector School Kigali Province Kicukiro Gikondo P.S. Kinunga Nyarugenge Nyamirambo EFS Nyamirambo Northern Province Musanze Kabaya G.S. Kabaya Busogo G.S. Rusanze Western Province Nyabihu Mukamira E.P. Kanyove G.S. Rwankeri Southern Province Gisagara Save E.P. Save A Ndora G.S. Ndora Huye Tumba E.P. Tumba Rusatira G.S. Kotana INTERVIEWEES Organization Building Learning Foundations 1 staff member 114 Organization Chemonics/Soma Umenye 5 staff members 1 Provincial Advisor 4 District Advisors DFID 1 staff member District Director of Education 2 District Education Officers 2 Head Teachers/DOSs 10 MINEDUC 1 staff member Parents 6 Publishing Company 1 REB 6 staff members Save the Children/Mureke Dusome 1 staff member Sector Education Inspectors 5 Teachers 39 UNICEF 1 staff member URCE 2 staff members USAID Rwanda 2 staff members Vice Mayor 1 World Bank 1 staff member 115 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Francis Kamanzi Title Evaluator Organization Dexis Consulting Group Evaluation Position? Team Lead X Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming-Africa (REEP-A) contract, award number AID-OAA- TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the followingfacts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employmentwith the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization thatmay be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 05/08/2019 116 Name Eugene MUKESHIMANA Title Research Assistant Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming- Africa (REEP-A) contract, award number AID-OAA-TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August, 5th 2019 117 Name Kathryn Norris Title Research Assistant Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming￾Africa (REEP-A) contract, award number AID-OAA-TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 9/30/19 118 Name Kristin Rosekrans Title Evaluation Team Leader Organization Dexis Consulting Group Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming-Africa (REEP-A) contract, award number AID-OAA-TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 15th, 2019 119 Name Max Shanstrom Title Evaluation Specialist Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming-Africa (REEP-A) contract, award number AID-OAA-TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significantthough indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience orseeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 9/17/19 120 Name Michael TUSIIME Title Local Consultant, Research for Effective Education Programming – Africa Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Evaluation under the Research for Effective Education Programming-Africa (REEP-A) contract, award number AID-OAA￾TO-16-00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Soma Umenye; Implemented by Chemonics International I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 26th 2019 121 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523