WHOLE-OF-PROJECT PERFORMANCE EVALUATION OF THE READING FOR SUCCESS PROJECT – MOROCCO Classroom in Salé for students who are deaf or hard of hearing. Credit: Wafae Khibabi School-to-School Interantional, 2019 March 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Varuni Dayaratna, Erika Keaveney, and Alexandre Monnard, NORC at the University of Chicago and Kristina Solum, Drew Schmenner, Matthew Murray, Mark Lynd, Stephanie Templeton, and Casey McHugh, School-to-School International. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | i WHOLE-OF-PROJECT PERFORMANCE EVALUATION OF THE READING FOR SUCCESS PROJECT – MOROCCO FINAL REPORT TASKING NO. 7617.020.01 PUBLISHED: MARCH 2020 SUBMITTED TO: Ben Sylla, USAID E3/ED Kurt Krausse, USAID/Morocco Amina El Abdellaoui, USAID/Morocco Mariam Britel-Swift, USAID/Morocco PREPARED BY: Erika Keaveney Alexandre Monnard Kristina Solum Stephanie Templeton Matthew Murray Mark Lynd Casey McHugh Drew Schmenner SUBMITTED BY: NORC at the University of Chicago DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | ii ABSTRACT This report evaluates the performance of USAID/Morocco’s Reading for Success (RFS) project and its four related activities: RFS-Small Scale Experimentation, RFS-National Program for Reading, RFS￾Human and Institutional Capacity Development, and RFS-Improving Deaf Children’s Reading Through Technology. It assesses the extent to which RFS contributed to USAID/Morocco’s Education Development Objective of enhanced educational attainment for children at the primary level. Research questions focus on six key topics: project outcomes; project implementation; inclusion; the role of the Moroccan Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE); sustainability; and lessons learned and recommendations. Evaluators from NORC at the University of Chicago and School-to-School International completed an initial desk review of RFS documents, took part in an evaluation design workshop, analyzed relevant quantitative data, and conducted primary qualitative data collection including key informant interviews and focus group discussions in October 2019. Evaluators found that RFS contributed to key elements of reading instruction; fostered a collaborative relationship between partners and the MoE; may not have addressed the specific needs of certain populations of students; and encountered building blocks as well as obstacles in ensuring sustainability. Evaluators also identified lessons and offered corresponding recommendations in five key areas: MoE’s ownership of education reform; implementation; inclusion; research, monitoring, and evaluation; and community engagement. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | iii TABLE OF CONTENTS LIST OF ACRONYMS..........................................................................................................................................IV 1. EXECUTIVE SUMMARY .................................................................................................................................. 1 Evaluation Purpose.......................................................................................................................................... 1 Project Background ........................................................................................................................................ 1 Evaluation Questions...................................................................................................................................... 2 Methodology ..................................................................................................................................................... 2 Findings and Conclusions.............................................................................................................................. 3 Project Outcomes........................................................................................................................................... 3 Implementation ................................................................................................................................................ 4 Inclusion............................................................................................................................................................. 4 Role of MoE...................................................................................................................................................... 5 Sustainability ..................................................................................................................................................... 5 Lessons Learned .............................................................................................................................................. 5 2. EVALUATION PURPOSE ................................................................................................................................ 8 Evaluation Purpose, Audience, and Intended Use ................................................................................... 8 3. PROJECT BACKGROUND ............................................................................................................................ 9 4. EVALUATION METHODS & LIMITATIONS...........................................................................................12 Evaluation Questions....................................................................................................................................12 Methodology Overview...............................................................................................................................13 Sampling, Data Sources, and Collection Methods.................................................................................14 Data Collection And Analysis ....................................................................................................................17 Limitations of the Evaluation Methodology.............................................................................................17 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS.............................................................20 Findings............................................................................................................................................................20 Conclusions....................................................................................................................................................65 Lessons Learned and Recommendations................................................................................................. 68 ANNEXES ..............................................................................................................................................................73 Annex I: Evaluation Statement of Work..................................................................................................73 Annex II: Evaluation Methods and Limitations .......................................................................................91 Annex III: Data Collection Instruments................................................................................................. 114 Annex IV: Sources of Information...........................................................................................................145 Annex V: Details on Cross-Country Evaluation Results Presented in Findings...........................149 Annex VI: RFS Evaluation Summary Matrix ..........................................................................................150 Annex VII: Evaluation Staff Qualifications..............................................................................................156 Annex VIII: Disclosure of Any Conflicts of Interest...........................................................................159 Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | iv LIST OF TABLES Table 1. KII qualitative sample by respondent type, subgroup, and completed number by sex .....15 Table 2. FGD qualitative sample by respondent type, subgroup, number completed, and number of participants by sex.........................................................................................................16 Table 3. EGRA Results in Comparison for RFS-SSE from Midline 2 to Endline..................................29 Table 4. Gain Scores for Passage Reading and Reading Comprehension for RFS-SSE Cohort 1 at Endline ..........................................................................................................................................30 Table 5. Gain Scores for Passage Reading and Reading Comprehension for RFS-SSE Cohort 2 at Endline ..........................................................................................................................................30 Table 6. RFS-SSE Impacts at Endline, by Grade (Cohort) and Subtest..................................................31 Table 7. RFS Performance Evaluation Matrix ..............................................................................................95 Table 8. Respondents by targeted number, tool, subgroup and sampling/selection criteria ........ 106 Table 9. Summary field work schedule ...................................................................................................... 110 Table 10. KII qualitative sample by respondent type, subgroup, and number of respondents by sex....................................................................................................................................................... 145 Table 11. FGD participants by respondent type, subgroup, number completed, and number of participants by sex .......................................................................................................................... 146 LIST OF FIGURES Figure 1. Reading for Success Project Results Framework .........................................................................9 Figure 2. RFS-SSE Evaluation Cohort Design................................................................................................29 Figure 3. Oral Reading Fluency Effect Sizes by Country, as Measured by EGRA ................................32 Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | v LIST OF ACRONYMS AREF Regional Academies of Education and Training CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting CRMEF Regional Teacher Training Center CSO Civil Society Organization DO Development Objective EGRA Early Grade Reading Assessment EGRSLA Early Grade Reading and Sign Language Assessment EGR/W Early Grade Reading and Writing FGD IP Focus Group Discussion Implementing Partner IR Intermediate Result KII Key Informant Interview LMICs Low and Middle Income Countries M&E Monitoring and Evaluation MASSAR Moroccan National Database for Education Statistics MCC Millennium Challenge Corporation MENA Middle East and North Africa MoE Ministry of National Education, Vocational Training, Higher Education, and Scientific Research MOOC Massive Open Online Course MSA Modern Standard Arabic MSL Moroccan Sign Language NGO Non-Governmental Organization PAD Project Appraisal Document PIRLS Progress in International Reading Literacy Study RFS Reading for Success RFS-HICD Reading for Success-Human and Institutional Capacity Development RFS-IDCRT Reading for Success-Improving Deaf Children's Reading Through Technology RFS-NPR Reading for Success-National Program for Reading RFS-SSE Reading for Success-Small Scale Experimentation STS School-to-School International USAID United States Agency for International Development USG United States Government Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 1 1. EXECUTIVE SUMMARY EVALUATION PURPOSE The performance evaluation of USAID/Morocco’s Reading for Success (RFS) project aims to assess the extent to which the project has contributed to the Mission’s Education Development Objective, and informs implementation of ongoing or upcoming activities, as well as the next Country Development Cooperation Strategy (CDCS). NORC at the University of Chicago conducted the evaluation in collaboration with School-to-School International from March 2019 to January 2020. Results will be shared with broader USAID audiences to highlight best practices, lessons learned, and achieved outcomes, as well as Government of Morocco counterparts to further promote improving education in Morocco. PROJECT BACKGROUND USAID/Morocco designed the RFS project to address Development Objective (DO) 3: Enhanced educational development for children at the primary level. RFS was developed in two phases. The first phase focused on building evidence for the implementation of early grade reading activities through research. It also featured a series of workshops and meetings with the Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE) and peers from other countries in the Middle East and North Africa (MENA) region to understand the current educational landscape as part of this evidence-building. This phase included the RFS-Human and Institutional Capacity Development (RFS-HICD) activity, which measured the capacity of the MoE to implement its comprehensive reform plan and proposed performance solutions to address identified gaps. The second phase of RFS comprised three activities:1 1. RFS-Small Scale Experimentation (RFS-SSE), 2015–18: This activity aimed to improve early grade reading outcomes by developing and piloting new Arabic reading lessons and materials for grades 1 and 2, as well as implementing summer enrichment reading programs. 2. RFS-National Program for Reading (RFS-NPR), 2017–22: This activity revised and piloted RFS-SSE’s grade 1 and 2 lessons and materials, developed new curricula and materials for grades 3 and 4, and is expanding reading enrichment activities with the ultimate goal of scaling to a national level. In addition, the RFS-NPR focuses on enhancing reading instruction through better teacher training and improving systems for assessment. 3. RFS-Improving Deaf Children’s Reading through Technology (RFS-IDCRT), 2015–18: This activity aimed to improve the reading skills of students who are deaf or hard of hearing in grades 1-3 through the use of educational software for Moroccan Sign Language (MSL). This evaluation focuses on the three activities implemented during the second phase of RFS and MoE’s implementation of the findings from RFS-HICD. 1 NORC at the University of Chicago, Desk Review for the Whole-of-Project Performance Evaluation of the Reading for Success Project. (2019). https://pdf.usaid.gov/pdf_docs/PA00W4KT.pdf Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 2 EVALUATION QUESTIONS The RFS performance evaluation centered on six evaluation questions and their related sub￾questions, which were developed with input from USAID/Morocco, the MoE, and other partners. Each question addresses a separate theme: 1. Project Outcomes: To what extent have the activities under the RFS project contributed to the project’s overall development objective of enhancing educational attainment for Moroccan children at the primary level? 2. Implementation: How and why did factors impede and/or facilitate achievement of the RFS project's development objective? 3. Inclusion: Which primary-school-aged populations of children are not benefiting from the current RFS project activities, and how can they be better served in future projects/activities? (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) 4. Role of MoE: What was the MoE’s role in leading project activities, and how did this role influence the implementation, management, and results of the activities? 5. Sustainability: What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? 6. Lessons Learned: What key preliminary lessons learned and recommendations can be gleaned from implementation of the project to date? METHODOLOGY The evaluation started in the spring of 2019 with a desk review of existing documents pertaining to RFS that served as a springboard for qualitative fieldwork. Then key representatives from the MoE and USAID/Morocco joined the evaluation team for a one-day evaluation design workshop in May 2019 during which evaluation questions and sub-questions were refined and partners provided local context and built consensus on the evaluation approach. In October 2019, evaluators completed qualitative fieldwork over the course of three weeks in Morocco, including key informant interviews and focus group discussions. Six qualitative tools were developed for each RFS stakeholder respondent group, including MoE staff, ranging from national to provincial officials; USAID/Morocco officials; RFS implementers and partner organizations; school directors; teachers; and parents. Three teams of evaluators conducted interviews not only in each of the eight provinces in the four regions where RFS-SSE and RFS-NPR activities were piloted, but also in four provinces where RFS-NPR has expanded nationally to gain perspective on this scale-up. Evaluators produced detailed field notes for each interview, and all these data were coded by thematic content using NVivo 12 software. Quantitative data, namely existing RFS project data, were reviewed and analyzed as applicable to complement qualitative findings. The combination of qualitative and quantitative research enabled triangulation of data for more accurate analysis. Before finalizing this report, the evaluation team hosted a results validation workshop in Rabat on December 11, 2019 with a follow-up remote meeting on January 22, 2020. MoE officials, implementing partners (IPs), and USAID participants provided additional information, stated whether the findings reflected their experiences, and reviewed and prioritized the recommendations. Outcomes from this validation process have been embedded in the report. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 3 FINDINGS AND CONCLUSIONS PROJECT OUTCOMES EVALUATION QUESTION 1: TO WHAT EXTENT HAVE THE ACTIVITIES UNDER THE RFS PROJECT CONTRIBUTED TO THE PROJECT’S OVERALL DEVELOPMENT OBJECTIVE OF ENHANCING EDUCATIONAL ATTAINMENT FOR MOROCCAN CHILDREN AT THE PRIMARY LEVEL? RFS activities aimed to improve the literacy skills of Moroccan children in primary schools by redesigning early grade reading curriculum and materials focused on learning in Modern Standard Arabic (MSA). As in other countries where USAID supports early grade reading instruction, synthetic phonics—la méthode syllabique—and balanced literacy was used as an approach. Evaluators sought to understand how RFS activities implemented the 5 Ts of reading instruction (time, technique, text, tongue, and testing) and how program participants and beneficiaries perceived their impact. With regard to each of the 5 Ts, evaluators found that: • Time: In general, respondents expressed satisfaction with the increased amount of time allocated to reading at school. • Technique: Respondents across various MoE levels almost unanimously viewed the new approach with dynamized instruction, which uses games and fun activities to help children understand the features of language, as an improvement over the whole word approach that preceded it. • Text: In collaboration with the MoE, personnel from RFS-SSE and RFS-NPR developed teachers’ guides, student textbooks, and storybooks to support the curriculum reform. Respondents from all levels—from regional directors to parents—noted the number of texts available to students as one of the biggest contributors to the success of the new method. Similarly, respondents praised the quality of the materials, not only for their technical merits, but also how they engaged the imagination of students and, thereby, accelerated learning to read. • Tongue: Across the board, respondents endorsed the phonetic approach to teaching Arabic—to communicate orally, to read, and to write. • Testing: RFS’s framework encouraged two types of initiatives that contributed to the testing of children’s reading programs, including 1) measures of impact through administration of the Early Grade Reading Assessment (EGRA), and 2) formative assessment to be conducted by teachers during instruction. However, most MoE officials interviewed for this evaluation reported they had never seen an EGRA report, and reports of formative assessment use were mixed. To measure RFS outcomes, USAID/Morocco measured activities’ performance against selected higher-level indicators. For RFS-SSE, EGRA endline data revealed that although grade 2 students in the experimental group outperformed those in the control group on two subtasks, there were no statistically significant differences in mean scores or gain scores between the two groups in subtasks that measure reading fluency and comprehension. RFS-IDCRT endline data showed that half of the students assessed did not sign a single word correctly in a reading passage, and more than seven out of ten students did not answer a single reading comprehension question correctly. As for the number of students reached, due to the scale-up of RFS-NPR, all grade 1 students nationwide have benefited from the activity since the 2017-18 school year, while grade 2 students have benefited since the 2018-19 school year. Although RFS aimed to increase the provision of reading activities outside of instructional time, the accelerated scale-up of the curriculum used the resources needed to implement all the planned initiatives. As for the number of policies developed or modified to improve reading activities, respondents primarily referenced the legal framework of 2019 and the national strategy as policies that RFS strengthened. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 4 IMPLEMENTATION EVALUATION QUESTION 2: HOW AND WHY DID FACTORS IMPEDE AND/OR FACILITATE ACHIEVEMENT OF THE RFS PROJECT'S DEVELOPMENT OBJECTIVE? Evaluators identified facilitating and impeding factors related to implementing RFS activities. Impeding factors include: • Challenging shifts imposed on the implementers’ work plans and timelines due to fast changing political realities and shifting MOE priorities required the implementers, as well as USAID and the MoE, to adapt quickly and demonstrate a high level of flexibility and responsiveness; • Conflicting views between inspectors and the MoE as to who was responsible for training teachers, which affected RFS-NPR’s teacher training roll-out as to who was responsible for training teachers, which affected RFS-NPR’s teacher training roll-out; • Logistical constraints and challenges regarding the delivery of trainings; • Teachers’ desire for additional supplementary materials at the school level, especially in the scale-up regions of RFS-NPR; • Limited implementation of community and parental engagement interventions; • Constraints limiting the use of the software provided for RFS-IDCRT beneficiaries; and • Broader systemic issues faced by children with disabilities, which impeded RFS-IDCRT’s ability to improve the reading skills of children who are deaf or hard of hearing. Facilitating factors include: • The strong, collaborative relationship between the MoE and USAID/Morocco, which placed MoE in a leadership role; • The implementation of an experimental phase of the program (RFS-SSE), allowing USAID to engage in evidence-based programming and apply lessons learned during experimentation to the development of further RFS activities; • Positive results when parents were reported to be engaged in activities; • Positive engagement from teachers and buy-in at the school level, which local MoE officials observed; and • Engagement from experts and individuals with disabilities as well as increased dialogue, collaboration, and sharing around the education of students who are deaf or hard of hearing. INCLUSION EVALUATION QUESTION 3: WHICH PRIMARY-SCHOOL-AGED POPULATIONS OF CHILDREN ARE NOT BENEFITING FROM THE CURRENT RFS PROJECT ACTIVITIES, AND HOW CAN THEY BE BETTER SERVED IN FUTURE PROJECTS/ACTIVITIES? Students with disabilities, students who spoke languages other than Arabic, as well as those from poor families, divorced families, or nomadic families, were all identified as populations whose specific needs may not be addressed by RFS. The limited or lack of inclusion of people with disabilities was a key, cross-cutting issue identified across all four RFS activities, including RFS-IDCRT. Some respondents within RFS-IDCRT noted there were limits to the improvements that the activity could offer, including support to students who are deaf or hard of hearing who also had additional disabilities. Strengthening parental support and engagement in schools, increasing support for pre￾primary schools, addressing insufficient training and resources for teachers with respect to inclusion, and changing preexisting negative attitudes about students with disabilities were all identified by respondents as measures by which these students could be better served in future projects or activities. Many respondents believed the MoE could do more to support students who have special needs. At the time of this evaluation, USAID is developing a new activity with the MoE to establish a pre-service teacher training curriculum for a certificate program in Deaf education, and RFS-NPR is providing teacher training on differentiation. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 5 ROLE OF MOE EVALUATION QUESTION 4: WHAT WAS THE MOE’S ROLE IN LEADING PROJECT ACTIVITIES, AND HOW DID THIS ROLE INFLUENCE THE IMPLEMENTATION, MANAGEMENT, AND RESULTS OF THE ACTIVITIES? The MoE primarily played a leading role with RFS, but its level of leadership and its roles and influence varied somewhat across the four RFS activities. For RFS-SSE, and even more so for RFS￾NPR, the MoE was in “the driver’s seat,” enabling the projects to work on the Arabic curriculum while leading and taking ownership of key initiatives, namely the accelerated scale-up of the new curriculum. Besides the important technical role that many MoE directorates played, the MoE also mobilized its staff at the regional, provincial, and school levels to participate in both RFS-SSE (where piloted) and especially RFS-NPR. Despite the advantages of the MoE’s leadership, its difficulty at times in coordinating with implementers affected the activities’ implementation and management. Although the MoE requested the RFS-HICD activity, changes in MoE structure, leadership, and focus meant that by the time the study was completed, findings and recommendations were not approved or formally implemented. The MoE’s role in RFS-IDCRT was more limited than in other activities because the Ministry does not have oversight over the private Deaf associations that were educating the beneficiaries of the activity. In addition, the MoE did not help to develop the activity, which originated through the All Children Reading: A Grand Challenge for Development project. USAID/Morocco tried to adapt All Children Reading’s initial design to meet the needs of the MoE and other stakeholders. SUSTAINABILITY EVALUATION QUESTION 5: WHAT ARE THE KEY STRENGTHS AND CHALLENGES THAT HAVE AFFECTED THE RFS PROJECT ACTIVITIES IN CREATING A SUSTAINABLE NATIONAL READING SYSTEM? The sustainability of RFS benefits from the MoE’s high level of ownership within the RFS-SSE and RFS-NPR activities, the extent to which these activities affect the core of the Moroccan education system—the curriculum and the textbooks, the capacity-building MoE staff received, and the strong internal political support for RFS. Impediments to sustainability include challenges with the teacher training cascade design and format, limited support materials and storybooks, and issues with the quality of textbooks designed by the private sector. Valuable relationships were fostered during RFS￾IDCRT that could lead to beneficial partnerships in the future for Deaf education. At the time of this evaluation, USAID was developing a new activity in collaboration with the MOE to develop a pre￾service teacher training curriculum for a teaching certification program for deaf and hard of hearing students. In addition, the RFS-NPR activity provides teacher training on the concept of differentiation. The RFS-HICD activity was designed as an assessment and did not include dedicated support for sustainability. In addition, the MoE has yet to grant formal approval to the activity’s findings. However, there are some reports that some RFS-HICD recommendations are being considered through RFS-NPR, the Millennium Challenge Corporation, and the MoE’s central departments and regional academies. LESSONS LEARNED EVALUATION QUESTION 6: WHAT KEY PRELIMINARY LESSONS LEARNED AND RECOMMENDATIONS CAN BE GLEANED FROM IMPLEMENTATION OF THE PROJECT TO DATE? Lesson 1: MoE ownership: Additional measures should be taken to strengthen MoE’s ownership of the reform, including clarification of roles, improved information sharing, and negotiation of priorities within contractual requirements. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 6 Recommendations 1. As RFS extends to all schools in Morocco through grades 5 and 6, clearly spell out roles to be played by MoE personnel and departments at central and provincial levels in areas of program design, development, and implementation to ensure these roles are performed. 2. Identify, mobilize, and deploy resources required to roll out RFS nationally at each grade level, and the role to be played by each MoE personnel and departments in coordinating training and managing resources. 3. Enable the RFS Steering Committee to provide program design review, monitoring, and quality assurance. 4. Establish a mechanism for sharing information with MoE personnel at central and provincial levels on a timely basis. 5. Ensure that any additional institutional capacity development efforts reflect the evolving systemic nature of the MoE and include provisions for involving the MoE in developing strategies to strengthen these systems. Lesson 2: Implementation: Respondents universally recognized the excellent quality of the RFS reading program; however, as the program expands to grades 5–6 in all schools in Morocco, careful consideration must be given to how it is scaled. Recommendations 1. Define a minimum package of support to be provided to schools participating in each activity, including modalities for teacher and school director support and minimum levels of training and materials. 2. Develop and sustain a dynamic plan for scaling that is updated and complied by all relevant stakeholders; delineates the roles of key actors by grade, year, and province; and identifies resources required for scaling, including transportation, communication, finance, and technical assistance. 3. Establish indicators for donors and the MoE to use to determine the extent to which the objectives of the scale-up are being met. 4. Conduct further research on the reasons underlying the limited impacts of RFS-SSE. 5. Ensure that sustainability is at the center of discussions. Lesson 3: Inclusion: Important strides have been made to ensure all children are included in public education in Morocco. However, vulnerable populations and children with special needs will not have sufficient access to a quality education until a strategy is defined to determine how to address each child’s needs, and a national effort is made to mobilize support for inclusive education at school, in the community, and at home. Recommendations 1. Develop a national strategy to increase the MoE’s capacity to provide inclusive education over time. 2. Develop an inclusive education plan to be shared with development partners ensuring political, financial, and technical support for the rollout of the national strategy. 3. Develop an inclusive education communication strategy to enable the MoE and its partners to implement effective outreach strategies. 4. Strengthen the technical understanding and capacity of the MoE to implement its inclusive education strategy. Lesson 4: Research, monitoring, and evaluation: USAID and the MoE have conducted research on their programs and tracked progress based on results frameworks and national indicators. However, monitoring and evaluation (M&E) systems and research processes must be strengthened to use the information from these efforts to inform decision-making routinely. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 7 Recommendations 1. Develop an information strategy to be used by USAID/Morocco and the MoE to establish a common research, monitoring, and evaluation framework. 2. Where appropriate, support efforts to “collaborate, learn, and adapt” (CLA) among USAID implementers, the MoE, and partners. 3. Strengthen the capacity of the MoE and IPs to conduct research, monitoring, and evaluation. Lesson 5: Community engagement: At both central and provincial levels, respondents stressed the importance of education outside the classroom, including the need for materials and activities to engage parents and communities. However, RFS’s scope limited support for parental and community engagement, raising questions about the importance and effectiveness of such initiatives. Recommendations 1. Conduct a landscape review to identify effective strategies for supporting early grade reading learning outside the classroom, both within and outside Morocco. 2. Investigate the political will of the MoE and potential partners—including parents associations, CSOs, private sector—to support extracurricular activities to support early grade reading. 3. If needed, develop a model in collaboration with the MoE for parent- and community-based support for early grade reading that can be deployed at the provincial level. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 8 2. EVALUATION PURPOSE EVALUATION PURPOSE, AUDIENCE, AND INTENDED USE The performance evaluation of the Reading for Success (RFS) project is intended to support the United States Agency for International Development/Morocco (USAID/Morocco) in gathering information to: • Assess the extent to which RFS project activities contribute to achieving the Education Development Objective (Enhanced Educational Attainment for Children at the Primary Level); • Inform the implementation of ongoing or upcoming activities, including that of the RFS￾National Program for Reading; and • Inform the next Morocco Country Development Cooperation Strategy (CDCS) design, which is scheduled to be drafted in 2020. The evaluation was conducted by NORC at the University of Chicago in collaboration with School￾to-School International (STS) between March 2019 and January 2020. The results of this evaluation will be shared with broader USAID audiences, including USAID/Washington and other Missions, to highlight best practices and lessons learned and to inform upper management of the results achieved by USAID/Morocco in basic education and early grade reading. In addition, relevant results will be shared with the Government of Morocco counterparts and donors to promote improvements to basic education in Morocco through enhanced reading instruction in the early grades. This evaluation report details the study’s six Evaluation Questions as well as the approaches to sampling, data collection, and analysis that the evaluation team employed to answer the questions. Three activities held prior to data collection—including a desk review of existing RFS-related documents, an evaluation design workshop and a stakeholder mapping exercise held in Morocco in May 2019, and an evaluation design report—informed the design of this study. The evaluation comprised six components; a desk review, design workshop, evaluation design report, stakeholder mapping, qualitative fieldwork, and an evaluation report. The desk review helped provide the context, background, and understanding needed for the effective design of qualitative fieldwork and supported the development of a more in-depth and nuanced evaluation. The design workshop enabled the team to ground the evaluation in local context and realities, refine the evaluation questions, identify the most appropriate data sources and methods for answering the questions, build consensus and buy-in on the evaluation approach, and finalize the evaluation work plan. The evaluation design report helped to ensure that evaluators and stakeholders had a shared understanding of the evaluation purpose, methods, and analysis and reporting procedures. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 9 3. PROJECT BACKGROUND USAID/Morocco conceived the RFS project to address Development Objective 3 (DO): Enhanced educational development for children at the primary level. This objective was detailed in the Mission’s most recent CDCS released in 2013.2 Citing persistent inequities in education and lagging literacy levels, USAID/Morocco shifted its focus toward primary education after having targeted other educational domains—including middle schools, teacher training institutes, and job training for disadvantaged youth—during the preceding ten years. As USAID/Morocco laid out in its CDCS, the foundational skill of literacy provides a springboard for students’ future success in the classroom, workforce, and broader civil society. The CDCS emphasized the importance of synergies with the Moroccan government’s education agenda, including the comprehensive reform initiative Vision 2030.3 The education DO would be met through attaining two intermediate results (IRs): “reading skills of primary level students improved” and “learning delivery systems improved.” In late 2013, USAID/Morocco formalized its approach to the educational DO in the CDCS by detailing its vision in a project appraisal document (PAD). The results framework changed slightly in the PAD (Figure 1). The first intermediate result (IR) was the same as in the CDCS. However, the second IR— “learning delivery systems improved”—was replaced with two other IRs—“community support for reading increased” and “policy environment to support reading improved.” Figure 1. Reading for Success Project Results Framework As a whole, RFS aims to address lagging reading outcomes among Moroccan primary grade students. The country’s youth literacy rate of 82 percent is among the lowest in the Middle East and North Africa (MENA), with a notable gap between young men (89 percent) and young women (74 percent).4 While reading achievement has improved in recent years, it remains low. Moroccan students’ reading scores on the Progress in International Reading Literacy Study (PIRLS) were statistically significantly higher in 2016 than in 2011, showing improvement, but Morocco ranked 2 USAID-Morocco, USAID-Morocco Country Development Cooperation Strategy—2013-2017, (2013). 3 Kingdom of Morocco Ministry of National Education, Vocational Training, Higher Education, and Scientific Research, Vision 2030, accessed April 2019, https://www.men.gov.ma/Fr/Documents/ Vision_ strateg_CSEF16004fr.pdf 4 UNICEF, “UNICEF Global Databased—Education: Adult Literacy Rates—Percentage,” accessed April 2019, https://data.unicef.org/wp￾content/uploads/2016/05/education_table-youth-and-adult-literacy-rate-updated-oct.-2015.xlsx. DO #3: Enhanced educational development for children at the primary level IR 1: Reading skills improved Sub-IR 1.1: Reading materials used in primary schools improved Sub-IR 1.2: Reading instruction improved Sub-IR 1.3: Extra￾curricular reading programs expanded IR 2: Community support for reading increased Sub-IR 2.1: Family engagement to support reading increased Sub-IR 2.2: Civil society organization capacity to implement reading initiatives increased IR 3: Policy environment to support reading improved Sub-IR 3.1: Benchmarks for grades 1- 3 established Sub-IR 3.2: Reading instruction time defined Sub-IR 3.3: National assessment protocols for reading established Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 10 48th out of 50 countries in 2016 and scored well below international benchmarks.5 In addition to PIRLS results, prior research supported by USAID underscores the opportunity for improvement. In 2011, an Early Grade Reading Assessment (EGRA) administered in the Doukkala Abda region revealed that only one out of three grade 2 and 3 students could read a second-grade level text well enough to comprehend it.6 Additionally, 33 percent of grade 2 students and 17 percent of grade 3 students could not read any words of a Modern Standard Arabic (MSA) text. Factors cited as contributing to poor performance on the EGRA included a lack of teacher preparation and reading materials, as well as a mismatch between the language of instruction at schools and the predominant languages spoken at home. RFS was developed in two phases. The first phase focused on building evidence for the implementation of early grade reading activities through research. It also featured a series of workshops and meetings with Morocco’s Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE) and peers from other countries in the MENA region. All the knowledge and evidence gained during this first phase informed the second phase of activity design and implementation, including RFS-Small Scale Experimentation (RFS-SSE), RFS-National Program for Reading (RFS-NPR), and RFS-Improving Deaf Children’s Reading Through Technology (RFS-IDCRT).7 The RFS-Human and Institutional Capacity Development (RFS-HICD) assessment of the MoE was conducted during the initial research phase. USAID/Morocco’s early grade reading research began in 2011 with the EGRA administered in the Doukkala Abda region. It then commissioned a series of studies in 2013 on early grade reading in Morocco, including analyses of curricula and texts used in formal and non-formal education; pre￾service teacher training; teachers’ attitudes, beliefs, and practices toward teaching reading in Arabic; and the textbook sector. The research concluded in 2015 with a peer review of the MoE’s Arabic curriculum and the RFS-HICD activity. Only the RFS-HICD activity, which took place from June 2016 to June 2017, has been included as part of this RFS evaluation study. In December 2013, a series of workshops and meetings kicked off with a workshop led by All Children Reading for participants from the MENA region. Country teams convened in Rabat to discuss the theory and practice of early grade reading and sketch out initial ideas for reform in Arabic reading instruction. USAID/Morocco subsequently organized two workshops on reading instruction in Arabic in November 2014 for MoE staff—one for high-level officials and the second for a larger group of practitioners. USAID/Morocco also facilitated a series of visits and discussions between the MoE and their counterparts in Egypt. This allowed Moroccan officials to learn about the Arabic reading reforms that had been implemented in Egypt. The RFS’s second phase of design and implementation started with the launch of the RFS-SSE activity in 2015. This RFS-SSE aimed to improve early grade reading outcomes by developing and piloting new Arabic reading lessons and materials for grades 1 and 2 and implementing summer enrichment reading programs. The activity’s lessons and materials ushered in a new phonics-based pedagogical approach to teaching reading in Morocco, representing a shift from the historically dominant whole language method.8 The MoE selected four regions for piloting the new lessons and materials—Sous 5 Mullis, I.V.S., Martin, M.O., Foy, P., & Hooper, M. PIRLS 2016 International Results in Reading. (Chestnut Hill, Mass.: Trends in International Mathematics and Science Study & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement, 2017). 6 RTI International, Student Performance in Reading and Mathematics, Pedagogic Practice, and School Management in Doukkala Abda, Morocco, (2012). 7 NORC at the University of Chicago, Desk Review for the Whole-of-Project Performance Evaluation of the Reading for Success Project. (2019). https://pdf.usaid.gov/pdf_docs/PA00W4KT.pdf 8 RTI International and Al Akhawayn University, Research on Reading in Morocco: Analysis of the National Education Curriculum and Textbooks—Final Report (Component 1)—Part 1 (Curriculum Analysis), (2015), p. 18. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 11 Massa, Rabat-Kenitra, Orient, and Fes-Meknes—based on geographic, cultural, and socioeconomic factors. The RFS-SSE activity closed in March 2018.9 RFS-NPR built on RFS-SSE’s work at a greater scale with the overall goal of enabling students in target regions to read fluently with comprehension by the end of grade 2. Started in 2017, this five￾year activity revised and piloted RFS-SSE’s grade 1 and 2 lessons and materials, developed new curricula and materials for grades 3 and 4 and is expanding reading enrichment activities with the ultimate goal of scaling to a national level. In addition, the activity focuses on enhancing reading instruction through better teacher training and improving systems for assessment. The activity piloted its new materials in the same schools targeted by RFS-SSE. Scale-up of grade 1 materials began during the 2017–18 school year, followed by grade 2 materials during the 2018–19 school year. The RFS-NPR activity is scheduled to end in 2022. The two other RFS activities—RFS-HICD and RFS-IDCRT—targeted the MoE and children who are deaf or hard of hearing, respectively. RFS-HICD supported the MoE as it began to implement its comprehensive reform plan—Vision 2030—to create a quality, equitable educational system. Through the administration of a Human Institutional and Capacity Development assessment, RFS￾HICD measured the MoE’s capacity to implement Vision 2030 and associated priority projects in nine technical areas. The activity also proposed performance solutions to address identified gaps. RFS-HICD ran from June 2016 through June 2017. RFS-IDCRT, which was launched in October 2015 and concluded in October 2018, aimed to improve the reading skills of students who are deaf or hard of hearing in grades 1- 3 through the use of assistive technology. In Morocco, more than 85 percent of students who are deaf or hard of hearing cannot access education; those who do attend school often attend schools run by community-based associations and encounter a dearth of materials and properly trained educators.10 RFS-IDCRT’s four major components included the development of educational software for Moroccan Sign Language (MSL), training stakeholders on the software use and other topics related to Deaf education, the creation of a Steering Committee to foster national dialogue on issues in Deaf education, and the development of an assessment to measure beneficiaries’ reading skills.11 9 Chemonics International Inc., Reading for Success-Small Scale Experimentation 2015-2018. (2018). https://chemonics.com/wp￾content/uploads/2018/07/Morocco_RFS-SSE_Final_Report_v2.pdf. 10 All Children Reading Grand Challenge for Development Round 2 Grantee Name, Second Annual Report for IDCRT. (2017). p. 3. 11All Children Reading Grand Challenge, Moroccan Sign Language Assistive Technology for Reading Improvement of Children who are Deaf/Hard of Hearing. (2018). https://www.edu￾links.org/sites/default/files/media/file/R3_EnsembleMedia_SchoolToSchool_IDRTReport_digital_lowres.pdf Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 12 4. EVALUATION METHODS & LIMITATIONS EVALUATION QUESTIONS Six evaluation questions and their related sub-questions guided the RFS performance evaluation. Starting with an initial set of questions drafted for the Project Appraisal Document (PAD) Evaluation Statement of Work, the evaluation team revised them in collaboration with USAID/Morocco and MoE partners at an evaluation design workshop held in Rabat and concurrent interviews with RFS implementing partners (IPs). This independent, whole-of-project performance evaluation aims to answer questions that each address a separate theme—project outcomes, implementation, inclusion, sustainability, and lessons learned: PROJECT OUTCOMES 1. To what extent have the activities under the RFS project contributed to the overall development objective of the RFS project of enhancing educational attainment for Moroccan children at the primary level? 1.1. How and why did students under the RFS-SSE, RFS-IDCRT, and RFS-NPR activities realize the observed learning results? 1.2. To what extent did RFS activities contribute to progress towards each of the 5 Ts of reading instruction?12 1.3. How have the activities performed against the higher-level indicators13 at the project level? IMPLEMENTATION 2. How and why did factors impede and/or facilitate achievement of the RFS project's development objective? 2.1. How and why did factors impede the achievement of the project's development objective? 2.2. How and why did factors facilitate achievement of the project's development objective? 2.3. To what extent have the activities responsively used (or are currently using) activity-level monitoring and evaluation (M&E) data to make necessary and relevant adjustments to program design, implementation, and management? INCLUSION 3. Which primary-school-aged populations of children are not benefiting from the current RFS project activities, and how can they be better served in future projects/activities? (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) 3.1. Which primary-school-aged populations of children (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at 12 USAID’s approach to improving reading and literacy revolves around five goals also knowns at the 5 Ts. According to USAID these 5 Ts are the key to reading success. The 5 Ts are: (1) more time devoted to teaching reading, (2) better techniques for teaching reading, (3) more texts in the hands of children, (4) teaching children in the mother tongue (a language they speak and understand), and (5) testing children’s reading progress. 13 These indicators are defined as: 3.b Percentage of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with United States Government (USG) assistance 3.1.a Number of learners reached in reading programs at the primary level 3.1.b Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance 3.2.a Increased provision of reading activities outside of instructional time (disaggregated by public / private) 3.3.a Number of laws, policies, regulations or guidelines developed or modified to improve primary reading programs Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 13 home, students in rural or vulnerable areas) in schools targeted by RFS are not benefiting from the current project activities? 3.2. What are the barriers to reaching those students who did not benefit from RFS and measures by which these students can be better served in future projects or activities? ROLE OF MOE 4. What was the MoE’s role in leading project activities, and how did this role influence the implementation, management, and results of the activities? 4.1. What was the Ministry's role in leading RFS project activities? 4.2. How did the Ministry's role influence the implementation, management, and results of the activities? 4.3. How did the role of the Ministry vary in implementation, management, and results at the central, provincial, regional, and local levels? SUSTAINABILITY 5. What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? LESSONS LEARNED 6. What key preliminary lessons learned and recommendations can be gleaned from implementation of the program to date? METHODOLOGY OVERVIEW In the spring of 2019, this evaluation began with a desk review that provided the context, background, and understanding needed for designing an appropriate evaluation methodology and identifying the most effective data collection methods to inform that design. It also supported the design of more in-depth and nuanced subsequent evaluation components. The evaluation team studied existing documents from USAID/Morocco and the implementers of the four RFS activities, external documents pertaining to early grade reading in Morocco, and resources from Morocco’s MoE. After completing the desk review, the evaluation team hosted a one-day evaluation design workshop in Rabat on May 7, 2019. The workshop grounded the evaluation in the local context and realities, refined the evaluation questions, and built consensus and buy-in on the evaluation approach. The design workshop included key representatives from the MoE and USAID/Morocco. To review and revise evaluation questions, participants ranked the preliminary questions as one group to ascertain evaluation priorities. The group then divided into subgroups and reviewed each question carefully, proposing improvements or sub-questions where needed. The large group discussed each question and together finalized all revisions. To determine the sample for the evaluation, participants used the updated questions and sub-questions to identify stakeholders most capable to answer each question. This activity included prioritizing those stakeholders as “critical,” “important,” or “helpful” to interview. The outcomes from these activities shaped the design of interview tools used during data collection. To answer RFS’s six primary performance evaluation questions and related sub-questions, the evaluators conducted qualitative fieldwork in Morocco over the course of three weeks in September and October 2019. The team collected data from a variety of RFS stakeholder respondent groups to gain an understanding of perspectives and experiences across levels—national, regional, provincial, school, and community—and to enable triangulation to improve the accuracy of findings. We triangulated qualitative and quantitative data and analysis to gain a holistic understanding of the RFS project and mitigate bias in the data and findings. Moreover, while the research focused primarily on the regions and provinces where RFS activities have been implemented—namely, the Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 14 RFS-SSE’s and RFS-NPR’s pilot regions, provinces, and schools—the sample included key MoE stakeholders from non-pilot regions and provinces to gain perspectives on broader scalability. Qualitative data collection methods included semi-structured key informant interviews (KIIs) and focus group discussions (FGDs). KIIs captured views, experiences, and perspectives from key stakeholders at the national, regional, provincial, and school levels, while FGDs captured similar information from beneficiaries at the school and community levels. To supplement qualitative findings, evaluators reviewed existing RFS project data—including EGRA results from both the RFS￾SSE evaluation and RFS-NPR baseline assessment. Finally, before finalizing this report, the evaluation team hosted a one-day results validation workshop in Rabat on December 11, 2019. A follow-up meeting with central MoE officials who were unable to attend the workshop was conducted remotely on January 22, 2020. With the primary goals to ground the evaluation findings and build consensus and buy-in for recommendations, MoE officials, IPs, and USAID participants provided additional information and identified any missing data sources. In the workshop and meeting, participants stated whether the findings reflected their experiences and provided evidence to support their responses. Participants then reviewed and prioritized the recommendations in small groups and provide additional recommendations where relevant. Outcomes from these activities have been embedded in the report. SAMPLING, DATA SOURCES, AND COLLECTION METHODS The evaluation sample included school-level respondents; MoE officials at the provincial, regional, and national levels; personnel from USAID/Morocco and RFS IPs; and staff from other RFS partner organizations. Evaluators primarily sampled respondents from the geographic areas targeted by RFS interventions. These areas include eight provinces in four regions that participated in both the RFS￾SSE and RFS-NPR activities. The project also included a few respondents from the four provinces where RFS-NPR and MoE are currently scaling up activities. The evaluation team worked with project staff to develop several tools to respond to each evaluation question. We developed six qualitative tools—four KII guides and two FGD guides—to use during fieldwork.14 Each tool aligned with one type of respondent, including USAID/Morocco officials, RFS implementers, and RFS partner organization, MoE officials, school directors, teachers, and parents. KEY INFORMANT INTERVIEWS (KIIS) We conducted KIIs with 87 respondents, including 17 with USAID/Morocco officials and staff from IPs, 56 with Moroccan government officials and other stakeholders, and 14 with school directors and other school leaders. Table 1 details completed KIIs by respondent types and subgroups. 14 While a seventh tool was developed for use with non-MOE government officials, the evaluation team—under consultation with the USAID Morocco education team—determined the associated subgroup was not relevant for this study. Therefore the tool was dropped, and the respondent subgroup removed from the sample prior to data collection. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 15 Table 1. KII qualitative sample by respondent type, subgroup, and completed number by sex Respondent type Subgroups Total number completed (by sex) USAID/Morocco officials N/A 3 (m: 1; f: 2) USAID IPs and subcontractors15 XXXXXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 1; f: 2) XXXXXXXXXX 5 (m: 4; f: 1) XXXXXXXXXXXXXXXXXX 6 (m: 3; f: 3) MoE officials XXXXXXXXXXXXXXXX 9 (m: 7; f: 2) XXXXXXXXXXXX 10 (m: 10; f: 0) XXXXXXXXXXX 10 (m: 8; f: 2) XXXXXXXXXXXXXXXXXXXX 16 (m: 13; f: 3) XXXXXXXXXXXXXXXXXXXXX 3 (m: 3; f: 0) XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX 8 (m: 5; f: 3) School directors XXXXXXXXXXXXXXXXXXXXXXXX 8 (m: 8; f: 0) XXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 2; f: 1) XXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 2; f: 1) Total number of key informant interviews16 87 (m: 67; f: 20) FOCUS GROUP DISCUSSIONS (FGDS) The evaluation team targeted and completed a total of 20 FGDs across 12 schools. As noted in Table 2, we completed 12 FGDs with primary school teachers—one FGD per school visit—and eight FGDs with parents of primary school students—one FGD per RFS-SSE and RFS-NPR pilot school visit. FGDs aimed for gender-balanced participation where possible. 15 Some respondents were engaged with or had experience with multiple activities, such as both SSE and NPR. However, they are only counted once in this table. 16 The total target number of KIIs does not include individuals previously identified the “Other Non-MOE Stakeholders” respondent type, which the evaluation team dropped prior to data collection. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 16 Table 2. FGD qualitative sample by respondent type, subgroup, number completed, and number of participants by sex Respondent type Subgroups Number of FGDs Completed Approx. # participants Teachers XXXXXXXXXXXXXXXXXXXXXXXXX 8 58 (m: 26; f: 32) XXXXXXXXXXXXXXXXXXXX 2 12* (m: 4; f: 2) XXXXXXXXXXXXXXXX 2** 2 (m: 0; f: 2) Total teacher FGDs 12 71* (m: 30; f: 35) Parents XXXXXXXXXXXXXXXXXXXXXXXX 8 67 (m: 23; f: 44) XXXXXXXXXXXXXXXX 0*** 0 (m: 0; f: 0) Total parent FGDs 8 67 (m: 23; f: 44) Total FGDs 20 138* (m: 53; f: 79) *Please note the discrepancy between the total number of respondents and sex disaggregated figures is due to missing data on six participants’ sex from one teacher scale-up FGD. The teacher respondents from RFS-NPR scale-up scales is due to missing information. ** In some cases, only one teacher was available during both of the RFS-IDCRT school visits. While these may be understood as interviews, the figures are provided here to align with respondent types. *** At one visited RFS-IDCRT school, the only parent in attendance at the FGD was the parent of an 18-year old student who was too old to have participated in the activity. At another visited RFS-IDCRT school, no parents attended. At a third attempted visit, evaluators interviewed an additional president of a Deaf association, who is counted among the KIIs above, in lieu of meeting with parents. SECONDARY DATA SOURCES The evaluation team worked with USAID/Morocco and its IPs to secure the following datasets for secondary quantitative data analysis/review: • EGRA data for the RFS-NPR baseline study conducted in May 2018; and • EGRA data for May 2016, September 2016, May 2017, and September 2017 data collections conducted as part of the RFS-SSE impact evaluation. RFS-SSE created new EGRA and Snapshot of School Management Effectiveness (SSME) tools soon after the activity launched and administered them at the four noted time points in 2016 and 2017. These assessments measured the impact of reading lessons and materials. The experimental longitudinal evaluation design included two cohorts of students from 46 treatment schools and 45 comparison schools. RFS-NPR’s baseline EGRA was designed so the treatment group offered representative coverage of the students targeted by the RFS-SSE intervention beginning in 2016. Comparison schools were matched to treatment schools on observable characteristics. Because these matching variables are not those that would be typically influenced by the RFS-SSE intervention over the short term, the RFS-NPR baseline comparison group offers a credible counterfactual for the original RFS-SSE treatment cohorts. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 17 DATA COLLECTION AND ANALYSIS Three research teams collected data over three weeks in Morocco. Each team consisted of three researchers—one qualitative evaluation expert from STS or NORC at the University of Chicago who led each team and provided quality assurance and two Arabic-speaking data collectors recruited locally who served as interviewers and notetakers. The composition of the teams ensured data could be collected in Arabic, French, English, or Amazigh languages. Each team included at least one female and one male researcher. During fieldwork, the three research teams communicated regularly with one another to share lessons and ensure procedural consistency. QUALITATIVE DATA CAPTURE AND ANALYSIS The teams recorded and took handwritten notes during FGDs and KIIs to capture qualitative data.17 Notetakers used that data to produced typed, expanded field notes in French. Each team’s qualitative evaluation expert reviewed the notetaker’s expanded field notes for quality, accuracy, and completeness. Together, they addressed any outstanding questions, concerns, or clarifications. Qualitative data—including audio recordings—were stored on a secured, password-protected server. The evaluation team finalized and imported field notes into NVivo 12, a qualitative data analysis software package, to systematically code and analyze them. Qualitative data analysis incorporated an iterative approach and included a thematic content analysis of narrative data to identify and validate emerging themes. We developed a codebook at two stages—first, when the evaluation design was finalized and second, during data analysis when additional themes emerged. Multiple analysts working in a single software package ensured that data, codes, and findings were stored in a common location. QUANTITATIVE ANALYSIS Evaluators conducted supplementary quantitative analyses/reviews using reports and data obtained from the 2016-17 RFS-SSE impact evaluation and 2018 RFS-NPR baseline assessment. This included expanded interpretation and contextualization of previous findings, as well as the conversion of RFS￾SSE impacts into standardized effect sizes to enable cross-country comparisons to similar evaluations of early grade reading programs in low- and middle-income countries. LIMITATIONS OF THE EVALUATION METHODOLOGY EVALUATION DESIGN This evaluation assesses the extent to which the activities under the RFS project contribute to achieving the Education DO (Enhanced Educational Attainment for Children at the Primary Level); inform the implementation of ongoing or upcoming activities, including that of the RFS-NPR; and inform the next Morocco CDCS design, which is scheduled to be drafted in 2020. This study is not able to speak directly to the causal impact of RFS activities in Morocco, but rather, identified and documented RFS successes, challenges, and lessons learned from written documentation, program implementers, stakeholders, and beneficiaries. LINGUISTIC AND CULTURAL COMPLEXITY The evaluation reviewed program successes, challenges, and lessons learned across diverse languages and cultural contexts, including deaf communication. To minimize misunderstandings and maximize 17 Handwritten notes often included quotes, key points, and themes that emerged for each question as well as non-verbal activity or body language. They also could include any big ideas, thoughts, or takeaways from the notetaker. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 18 learning, the evaluation teams included researchers with French or Arabic language skills as well as previous experience working in the Deaf education sector in Morocco. FIDELITY OF IMPLEMENTATION It is difficult in some cases to disentangle programmatic successes, challenges, and lessons learned from issues with implementation fidelity. For example, the desk review found that scaling RFS-NPR was limited by strikes from inspectors who led trainings in regions targeted for scaling. While the strikes may have undermined program effectiveness, shortcomings may be due to implementation challenges rather than flaws in the project’s design or theory of change. REPRESENTATIVENESS OF SAMPLE Members of key informant and implementation groups were included in the sample; however, they were not randomly selected and thus do not statistically represent participants in the RFS project. Moreover, this evaluation does not target the most highly marginalized populations who may not be in school at all. The evaluation team conducted a stakeholder mapping exercise to identify respondents for KIIs during the evaluation design phase. School site selection for school directors, teachers, and parents was made by purposively selecting schools that had participated in RFS activities to ensure that visits reached a mix of schools. The activity sought to include schools from urban and rural areas, those with majority and minority Amazigh-speaking populations; those with small, medium, and large student populations; and low-, medium-, and high-performing schools, as measured by mean RFS-SSE EGRA oral reading fluency scores. The MoE introduced the evaluation team to all KII and FGD respondents, which may have introduced bias. DIVERGENT FINDINGS When respondents expressed different—sometimes contradictory—views on the project’s effectiveness, the evaluation team triangulated responses. However, it was not always possible to determine whose information was correct. In addition, qualitative and quantitative findings diverged in some cases. Evaluators acknowledged these variations and contradictions in the final report. AVAILABILITY OF RESPONDENTS Three of the project’s activities—RFS-SSE, RFS-HICD, and RFS-IDCRT—closed before the evaluation began, which limited the availability of personnel for interviews. To mitigate this, research teams conducted interviews with stakeholders and personnel no longer residing in Morocco via Skype, Zoom, or telephone. Special efforts were made to accommodate respondents’ schedules and preferred methods of communication. ACCURACY OF QUALITATIVE RESPONSES In each KII and FGD, data collectors explained confidentiality and anonymity to encourage respondents to provide frank and accurate response. Nevertheless, some respondents may have been biased or felt uncomfortable speaking freely. To mitigate this, respondents were given the option to be recorded electronically; some declined. FGD facilitators also asked participants to respect the privacy of their fellow participants and the confidentiality of the conversation. Similarly, the consent procedures clearly articulated that there are no direct benefits for participating in the study to dissuade respondents from providing misleading information in the hopes of receiving some material benefit in the future. USE OF EXPANDED FIELD NOTES Evaluators took notes during KIIs and FGDs, then entered them in an expanded format into computers in French. Notes were checked against electronic recordings of the conversation where Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 19 available. Because these are not full transcriptions and translations, notes did not contain all details of KIIs and FGDs. Nevertheless, evaluators determined that the expanded field notes were sufficient for this evaluation. LIMITED AVAILABILITY OF QUANTITATIVE DATA The evaluation team’s ability to conduct the secondary quantitative analysis proposed in the design report was limited. First, we were unable to gain access to the Moroccan National Database for Education Statistics (MASSAR) data despite sustained efforts. Second, the EGRA assessments used for RFS-SSE and RFS-NPR had not been equated; hence, they were not considered generally comparable, and the data from the two studies could not be merged/analyzed simultaneously. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 20 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS This section presents findings and conclusions from the evaluation, organized by research question and sub-question. Findings are based directly on evidence drawn from the desk review, qualitative data collection, and/or quantitative analyses. Conclusions are presented within each research question or sub-question following each finding. 1. PROJECT OUTCOMES: TO WHAT EXTENT HAVE THE ACTIVITIES UNDER THE RFS PROJECT CONTRIBUTED TO THE OVERALL DEVELOPMENT OBJECTIVE OF THE RFS PROJECT OF ENHANCING EDUCATIONAL ATTAINMENT FOR MOROCCAN CHILDREN AT THE PRIMARY LEVEL? In order to evaluate the extent to which RFS activities enhanced educational attainment for Moroccan children at the primary level, the evaluators investigated the factors contributing to learning results. Namely, we examined how RFS has contributed to progress toward each of the 5 Ts of reading instruction and tracked the activities’ achievements against high-level indicators such as: • Indicator 3.1.a: Number of learners reached in reading programs at the primary level, and • Indicator 3.2.a: Increased provision of reading activities outside of instructional time. In KIIs and FGDs, evaluators asked all stakeholders—including USAID/Morocco officials, activity implementers, MoE officials, school directors, teachers, and parents—about the effectiveness of RFS activities in improving students’ reading skills. The findings gleaned from qualitative analysis of the KIIs and FGDs are organized in this section by the three evaluation sub-questions. The first two evaluation sub-questions are answered together because how and why students have realized observed learning results—sub-question 1.1—are predominantly linked to how RFS activities have progressed towards each of the 5 Ts—sub-question 1.2. In addition to the qualitative analysis, we incorporate secondary analysis and findings from RFS quantitative data where applicable. 1.1 HOW AND WHY DID STUDENTS UNDER THE RFS-SSE, RFS-IDCRT, AND RFS-NPR ACTIVITIES REALIZE THE OBSERVED LEARNING RESULTS? 1.2 TO WHAT EXTENT DID RFS ACTIVITIES CONTRIBUTE TO PROGRESS TOWARDS EACH OF THE 5 TS OF READING INSTRUCTION? RFS activities aim to improve the literacy skills of Moroccan children in primary schools by redesigning the early grade reading curriculum and materials focused on learning in MSA—the official language of instruction in Morocco. Joint research between USAID/Morocco and MoE laid the groundwork for the implementation of RFS through a series of activities and workshops designed to facilitate learning about the national literacy landscape in primary schools. These activities included analysis of the national education curriculum and textbooks; an EGRA conducted in the Doukkala Abda region of Morocco; analysis of pre-service teacher training; analysis of teachers’ attitudes, beliefs, and practices toward teaching reading in Arabic; a review of non-formal education curriculum and learning materials; and a book sector study. One USAID/Morocco official mentioned how building this foundation of research was essential to getting RFS off the ground: “Evidence-building was critical, as it allowed going beyond ‘just enough info to write a scope of work and to design and request something quite specific.” As in other countries where USAID supports early grade reading instruction, synthetic phonics—la méthode syllabique—and balanced literacy approach was used. Instruction in synthetic phonics begins with phonological awareness—that is, knowledge of how sounds make up language and, in time, how they map to letters—then progresses to decoding, vocabulary, fluency, and comprehension. In Morocco, writing is also a key component of this approach. The launch of the revamped curriculum and materials began in 91 experimental schools in grades 1 and 2 as part of RFS-SSE and expanded Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 21 nationwide as part of RFS-NPR to grade 1 for the 2017-18 school year, followed by grade 2 in 2018- 19, and grades 3 and 4 in 2019-20. RFS adopted the rubric of the “5 Ts” to guide program design. Adapted from research conducted by Richard Allington, professor of literacy studies at the University of Tennessee, USAID “developed the ‘5 Ts’ as a framework to use in … producing sound evidence-based policies, practices, and assistance programs.”18 USAID defines the 5 Ts as19 1. Time: More time devoted to teaching reading 2. Technique: Better techniques for teaching reading 3. Text: More texts in the hands of children 4. Tongue: Teaching children in the mother tongue (a language they speak and understand) 5. Testing: Testing children’s reading progress Evaluators sought to understand how RFS activities implemented the 5 Ts and how program participants and beneficiaries perceived their impact. Because RFS’s implementation of the 5 Ts directly impacts the level of students’ learning, this section consolidates the first two sub-questions by describing students’ learning progress through the lens of the 5 Ts. 1. More time: In the redesigned curriculum, additional time is allocated to Arabic language lessons with specific time devoted to reading each day. Overall, time dedicated to Arabic has increased from ten to eleven hours per week in grade 1, seven to ten hours in grade 2, and five to six hours in grades 3 and 4. In the redesigned curriculum, at least 45 minutes of instruction per day in grades 1 and 2, and 30 minutes per day in grades 3 and 4 focus solely on reading. In general, respondents expressed satisfaction with the increased amount of time allocated to reading at school. However, some teachers claimed it was still not enough time to cover all aspects of the new teaching method. Some parents also said their children are spending more time reading at home. 2. Better techniques: According to some respondents, the new approach introduced by RFS— going from sounds to letter knowledge to syllables and words, then to sentences and comprehension—was the main method used in Moroccan schools in the 1970s. It had since been replaced by the whole word method—la méthode globale—where children were introduced to whole words and texts, then parts of words and letters. With the newer synthetic phonics and balanced literacy method, teachers “dynamized instruction,” using games and fun activities— activités ludiques—to help children understand the features of language. Actions included clapping when certain sounds were identified in words, discussing what was happening in pictures, leading exercises designed to help children speak Arabic fluently, and moving their bodies to break words and syllables into parts. These activities were accompanied by pictures, flashcards, and storybooks. Respondents across various MoE levels almost unanimously viewed the new approach with dynamized instruction as an improvement over the whole word approach that preceded it. For example, one school director described it as follows: “The syllabic method is a revolution, and people should understand that.” A regional director highlighted that the method starts with verbal communication, then moves to short texts and finally to longer ones, as one positive feature. The positive experience was echoed by 18 Gove, A. and P. Cvelich. 2010. Early Reading: Igniting Education for All. A report by the Early Grade Learning Community of Practice. Research Triangle Park, NC: Research Triangle Institute. 19 https://blog.usaid.gov/2012/11/education-week-2012-reading-improves-with-the-5-ts/ “THE SYLLABIC METHOD IS A REVOLUTION, AND PEOPLE SHOULD UNDERSTAND THAT.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 22 an MoE provincial director who shared that in his personal experience, “No teachers have resisted this method because they thought it wasn’t good or it’s too much. Each time we visit schools, we find happy people, a good atmosphere.” He also noted that this method made it possible for the teachers to use their imagination. Another MoE provincial director said that, based on his observations from class visits, the focus on synthetic phonics had improved students’ reading ability: “In essence, the stories in the readers and the phonological activities helped children become familiar with the Arabic language and generated a lot of enthusiasm in the class.”20 Several respondents also claimed this approach excelled over the previous one. One teacher commented that while the old method was boring for the teacher and the students, the new one was not. Another teacher reported, “With this new method, students interact with the stories, making their participation more active in class.” One inspector noted that a strong point of the new method was its basis in phonics—conscience phonétique—which was lacking with the prior method. Several parents expressed positive perceptions of changes brought in by the new teaching techniques. For example, one parent shared: “The old method was like a funnel, asking the student to take in everything the teacher said and did. This new method is different.” Another parent observed, “This new method allows the weaker students to believe in themselves, to have confidence. In this way, all students are able to participate in class.” One parent also shared her joy at the new approach’s effectiveness over the old program—one that even had a sentence to explain the letter M instead of simply teaching it in context.21 Some respondents shared their view that students’ comprehension is increasing as well. Another teacher noted that students’ ability to actively engage with stories made it possible for them to understand the meaning.22 The near unanimity of support for the new method notwithstanding, some teachers expressed concerns about the workload. The new approach to reading requires preparation, especially at the beginning of the school year and for grades 1 and 2. This preparation includes time—and sometimes the teacher’s personal resources—to create teaching and learning tools, especially when such materials are not provided. Teachers also reported challenges related to large class sizes, their students’ lack of pre-primary education, and, less commonly, students’ weak mastery of the language of instruction. RFS-IDCRT focused both on the use of the software provided to schools and also on teaching techniques geared toward children who are deaf or hard of hearing. Activity staff conducted three trainings for teachers on software usage and teaching techniques, which students’ parents, government officials, and other stakeholders also attended. Respondents from RFS-IDCRT schools cited these trainings as one of the biggest successes of the activity, believing that the trainings enabled different stakeholders to collaborate and learn from each other in formal sessions, but also during coffee breaks and other informal gatherings. “There were real 20 While there is some evidence from the RFS-SSE EGRAs to indicate the new method supports improvement in foundational reading skills such as phonemic awareness and syllable identification, similar observations around oral reading fluency and reading comprehension have yet to be corroborated in RFS EGRA data. « معلمة أمیمة أم *» 21 22 Similar observations around increased reading comprehension have yet to be corroborated in RFS EGRA data. “THIS NEW METHOD ALLOWS THE WEAKER STUDENTS TO BELIEVE IN THEMSELVES, TO HAVE CONFIDENCE. IN THIS WAY, ALL STUDENTS ARE ABLE TO PARTICIPATE IN CLASS.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 23 opportunities to share things,” one RFS-IDCRT beneficiary said. The connections continued beyond trainings. Teachers created groups on WhatsApp and Facebook to exchange teaching materials and lessons. Parents also created groups on social media. This heightened sense of solidarity extended beyond colleagues. One beneficiary mentioned how training helped to unify MSL, which is not standardized across the country. The trainings also introduced practitioners to experts from the United States. Findings from the RFS-IDCRT evaluation report echo respondents’ enthusiasm, highlighting how the trainings improved educators’ awareness of how to use MSL and oral language in their teaching. Texts in greater numbers and of higher quality: In the initial stages of the RFS project, USAID/Morocco conducted a number of activities with the MoE focusing on textbook quality and availability. In these activities, USAID/Morocco trained MoE officials on how to review textbooks for quality, develop readers appropriate to the interests and levels of students in the early grades, and plan and implement materials development and distribution programs. Then, in collaboration with MoE, personnel from RFS-SSE and RFS-NPR developed teachers’ guides, student textbooks, and storybooks to support the curriculum reform. Developers based the reading program first on oral communication in standard Arabic, then on developing phonological awareness, decoding strategies, comprehension, and writing skills. The resulting curriculum included a package of materials for each classroom—first in grades 1 and 2 (RFS-SSE), then in grades 3 and 4 (RFS-NPR). The package included one teacher’s guide, one consumable textbook per student, and one set of leveled, age-appropriate readers. While this evaluation did not quantify the materials available before or during RFS, the phrase textes entre les mains des enfants (texts in the hands of children) was one of the most frequent refrains heard during this evaluation from respondents at all decentralized levels, from regional directors to school directors, teachers, and parents. In addition to their availability, many respondents praised the quality of the materials. Lecturers from the CRMEF were pleased that the materials were based on empirical evidence of what works in Arabic, including identifying easily understood aspects to teach first and teaching letters in order of ease and frequency rather than in alphabetical order. Some teachers appreciated the sequencing of activities, including exercises that broke words into parts and isolated letters, those that explored meaning through pictures, and others that used comprehension strategies. One teacher said: “By starting with communication exercises, teachers ensure a common level of understanding and communicative competence amongst students before proceeding with letters and words.” Some respondents noted that reading instruction used to start with grammar and rules; now this is saved for grades 3 and 4, which, according to one teacher, “encourages oral expression. Once they start focusing on rules, they limit their ability to express themselves orally.” Said another teacher: “Before, texts were taught in conjunction with rules, which were seen as very important. Now rules are taught, but implicitly, without direct reference. This enables us to improve students’ reading more quickly.” According to one inspector, delaying the teaching of rules to later grades, “enables the student to find opportunities to speak spontaneously.” Other quality improvements included text length and relevance and the use of pictures. One teacher noted that texts are shorter and thus more accessible to the students. Some teachers and parents praised the relevance of materials, including terminology, child-appropriateness, and regional adaptation. In particular, many teachers noted that the pictures included in the textbooks stimulated interest and enhanced learning. Some teachers had begun photocopying the pictures to make them available to multiple students during their lessons. Most of all, respondents at all levels praised the use of “BEFORE HE GOES TO BED, MY CHILD READS HIS TEXTBOOK, EVEN THE STORYBOOKS, WITH JOY.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 24 storybooks to engage the imagination of the child and thereby accelerate learning to read. All students are welcome to take books home. One parent said: “When the teacher reads a story to the students with different scenes, the child’s imagination comes alive. When that happens, the child connects the letters to words and pictures, which really helps her develop her ability to read.” Similarly, another parent shared that “before he goes to bed, my child reads his textbook, even the storybooks, with joy.” Despite the overall positive perceptions of the texts, some respondents highlighted challenges and areas for improvement, including the need to increase the diversity of themes, the relevance of some stories to the lives of children, and the number of pages in the textbook. With respect to the stories’ relevance, one teacher from a rural school said that the stories did not relate to the lives of students in rural areas. On the number of pages, one implementer from Creative observed that while the MoE has recognized the need to go beyond textbooks of 130 pages in the lower grades, this is seen as a political challenge that can only to be resolved if the national education committee in Parliament, once set up, approves it. For RFS-IDCRT, the activity’s implementer developed and provided Clip and Create software to ten schools. This software contained graphical and video depictions of more than 3,000 undocumented signs, including regional variations, as well as the word in MSA. It also enabled teachers to create instructional materials, which were previously much sparser, and included six stories in MSL as well as recommended instructional strategies. The activity’s framework provided teachers with a computer, monitor, keyboard, and printer to use with the software. A teacher summed up the previous dearth of resources in these schools by remarking that the activity was beneficial “Because at least we had something.” Beyond the classroom, the software helped parents learn some signs, follow their children’s progress, and improve the reading ability of letters and words, according to one school director. One school director stated that his school has continued to use the software as a dictionary after the RFS-IDCRT activity officially ended but lamented that parents and teachers could not find certain words because the dictionary is incomplete with only 3,000 signs. This finding matches what respondents said in the activity’s evaluation report. Although the majority of teachers said they used the software’s dictionary or thesaurus to teach their students MSL, they noted drawbacks, including limited vocabulary and the absence of some regional variations of MSL. 3. Mother tongue. While MSA has long been the language of instruction in Morocco, RFS has changed how students learn to read in MSA. When asked about the advantage of teaching reading in standard Arabic, MoE officials and school personnel primarily referenced the 2019 legal framework—known as loi cadre—and the 2015–2030 strategic vision as the two documents that continue to mandate the mastery of the Arabic language.23 In this sense, RFS’s framework was fully aligned with national policy and priorities. Indeed, Morocco is committed to improving reading in Arabic and reforming how reading is taught. Respondents often referred to Arabic as the children’s mother tongue, even though—according to some—“standard Arabic was our first foreign language at school” because the Darija and Amazigh languages are spoken the most widely at home. These diverse responses reflect differing views on the relationship between MSA, as taught in schools, and Darija.24 According to RFS-SSE EGRA data, which is 23 For a school of equity and, quality and promotion: A Strategic Vision of Reform 2015-2030. 24 The relationship between MSA and Darija within the diglossic nature of Arabic is complex, and respondents provided a wide range of perspectives. While both MSA and Darija are Arabic, many respondents viewed them as different dialects given the differences between the two forms of Arabic. RFS-SSE and RFS-NPR IPs reported using material with vocabulary and grammatical structures that are close to the Arabic language spoken at home – “making use of the language corpus which is common to both MSA and the Arabic vernacular (Darija).” One example cited by RFS-NPR was using “folk tales as a means to address the diglossia issue, including the deficit in students’ oral language and the gap between MSA and Darija.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 25 representative of the pilot population of schools, most grade 1 and 2 students reported speaking Darija at home—75.9 and 71.3 percent, respectively—while approximately a quarter of grade 1 and 2 students—22.7 percent and 25.9 percent, respectively—reported speaking an Amazigh language.25 Across the board, respondents endorsed the phonetic approach to teaching Arabic—to communicate orally, to read, and to write. One teacher trainer reported being so taken by RFS and the notion of teaching reading in Arabic that he even published a book on RFS and teaching in Arabic in elementary school.26 Respondents noted that learning to read in Arabic—a language many, but not all, students can understand early in primary grades—contributed to their enthusiasm and ability to learn more quickly.27 One parent also viewed teaching in Arabic as a way to protect the value of Arabic “in the face of the French wave.” For RFS-IDCRT, however, the establishment and use of mother tongue language for students who are deaf and hard of hearing were noted as especially challenging. As one implementer observed, “the number one problem (with these children) is language: not everyone knows how to sign, and signing language isn’t standardized. Different regions have their own variations, making it difficult to communicate across Morocco.” These differences were not comprehensively documented prior to RFS-IDCRT, as noted in the RFS-IDCRT evaluation report, and the first version of the software did not account for enough of these regional differences, prompting the release of an updated version during the project. Members of the Deaf community submitted 110 regional sign variations through an online tool designed to capture them. 4. Testing children’s reading progress. RFS’s framework encouraged two types of initiatives that contributed to the testing of children’s reading progress: - Measures of impact. RFS-SSE, the RFS-SSE summer programs, and RFS-IDCRT all used EGRAs to measure changes in participating students’ reading ability. In the case of RFS￾IDCRT, EGRA was adapted for the first time to meet the needs of students who are deaf or hard of hearing. The final version included a sign language assessment as well as the standard reading subtasks. In each case, MoE officials and local consultants participated in the adaptation of EGRA tools, the collection of data, and, more limitedly, the interpretation of results. These collaborative efforts built local capacity around procedures for obtaining evaluation results and processes for decision-making based on assessment results. According to one MoE evaluation official, RFS personnel helped build his unit’s capacities in the areas of data collection systems (dispositif de collecte de données) and the use of technology, specifically tablets for electronic data collection. He noted that the MoE would soon be working with RFS officials to learn about the impact of these programs as measured by EGRA and methods for supervising similar programs. Importantly, some respondents reported having participated in the presentation of EGRA results by RFS staff; others participated in focus groups with teachers concerning the results. In the case of RFS-IDCRT, the measurement of impact affected teaching practices. One respondent noted how teachers at his school only taught letters and words to grade 1 and 2 students, but that changed after EGRA results showed deficiencies in passage reading and comprehension. Thereafter, teachers incorporated sentences and stories in their instruction. 25 The proportion of Grade 1 and 2 students who reported speaking MSA at home was small—1.27 percent and 2.66, respectively. 26 « Didactique de langue Arabe pour l’enseignement primaire: planification, gestion, évaluation et traitement » 27 Curiously, two respondents—an inspector and a regional director—noted that speakers of Amazigh languages learned standard Arabic faster than Darija speakers. RFS-SSE EGRA data provide limited data to support these observations. The two groups showed statistically similar results on most reading subtasks at baseline and a mixed picture at endline, with speakers of Amazigh languages typically showing greater gains overall but Darija speakers showing greater gains in response to treatment. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 26 However, most MoE officials interviewed for this evaluation reported that, despite their expectations, they had never seen an EGRA report. One MoE official lamented the turnaround time from administering an EGRA to receiving results: “It was conducted in April, we asked for the results, now it’s September, and we still haven’t seen any documentation. It’s too long. These evaluations could inform our implementation.” The same official also wondered why the EGRAs did not find significant improvements in cases where MoE officials saw children learning to read in a matter of months. A USAID/Morocco official hypothesized that one reason why limited improvements were observed in the evaluation reports could be that the text used in the EGRA was different from the ones commonly used in class—that is, the level of difficulty of the EGRA text could not capture students’ abilities and, therefore, did not show improvement. - Formative assessment. The second “testing” approach set forth in RFS’s framework was the formative assessment—evaluation formative—to be conducted by teachers during instruction. In RFS-NPR’s designed curriculum, the 30-week school year is divided into six five-week units, with the fifth and final week devoted to evaluation. RFS-NPR developed a framework to help teachers build their own tools for these week 5 assessments. Teachers’ tools were implemented for the first time during the 2019-2020 academic year in experimental schools. According to an RFS-NPR implementer, the MoE is currently reviewing the efficacy of these assessments with plans to roll them out nationwide. In one type of evaluation reported by a school director, teachers ask the student to read a text, then respond to comprehension questions. One teacher reported that RFS had introduced the use of a timer when assessing students’ reading, as well as an evaluation matrix to track students’ progress by week. According to one CRMEF lecturer and RFS trainer, “When teachers follow each child, they find different results. The teachers have changed since we started this training; now they know how to do it [formative assessment of students] differently.” Some teachers commented that they had already practiced formative assessment, but in some cases, had not explicitly assessed reading. Some respondents reported that due to the inclusion of formative assessments in teachers’ reading instructional programs, some schools had incorporated reading criteria in the interpretation of their end-of-year exams to determine whether a child should pass to the next grade; this was because if the student could not read, s/he could not understand other subjects either. Some respondents, however, reported that since 2015, they had not seen any changes in evaluation practices and continued their established practice of conducting a diagnostic assessment at the beginning of the year, a reading test at the end of the first trimester, and an exam at the end of the school year. Finally, it appears that much of the motivation to revise the curriculum, improve reading, and strengthen evaluation capacity comes from Morocco’s participation on the Progress in International Literacy Study (PIRLS), “[THE EGRA] WAS CONDUCTED IN APRIL, WE ASKED FOR THE RESULTS. NOW IT’S SEPTEMBER, AND WE STILL HAVEN’T SEEN ANY DOCUMENTATION. IT’S TOO LONG. THESE EVALUATIONS COULD INFORM OUR IMPLEMENTATION.” “WHEN TEACHERS FOLLOW EACH CHILD, THEY FIND DIFFERENT RESULTS. THE TEACHERS HAVE CHANGED SINCE WE STARTED THIS TRAINING; NOW THEY KNOW HOW TO DO IT DIFFERENTLY.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 27 on which they ranked 48th out of 50 countries in 2016 and scored well below the low international benchmark.28 1.3 HOW HAVE THE ACTIVITIES PERFORMED AGAINST THE HIGHER-LEVEL INDICATORS AT THE PROJECT LEVEL? As part of the Mission’s third DO, USAID/Morocco selected a high-level indicator to measure the progress of reaching the overall goal of enhanced educational development for children at the primary level as well as other indicators for the project’s three intermediate results. The third DO is measured against indicator 3.b, the percentage of learners who demonstrate reading fluency and comprehension at the end of grade 2 with United States Government (USG) assistance. Three intermediate results feed into the third DO: • DO: Enhanced educational development for children at the primary level o Indicator 3.b: Percentage of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with the United States Government (USG) assistance • IR 3.1: Reading skills improved o Indicator 3.1.a: Number of learners reached in reading programs at the primary level o Indicator 3.1.b: The number of learners in primary schools or equivalent non-school based settings reached with USG assistance. • IR 3.2: Community support for reading increased o Indicator 3.2.a: Increased provision of reading activities outside of instructional time (disaggregated by public / private) • IR 3.3: Policy environment to support reading improved o Indicator 3.3.a: Number of laws, policies, regulations or guidelines developed or modified to improve primary reading program This section addresses the extent to which RFS activities performed against the indicators for the overall DO as well as the three intermediate results. Development Objective Level The top-line objective of children’s enhanced educational development is measured by top-level indicator 3.b: Percentage of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with United States Government (USG) assistance. To measure the development of students’ fundamental reading skills, USAID implementers have administered EGRAs as part of the RFS-IDCRT, RFS-SSE, and RFS-NPR activities. RFS-IDCRT RFS-IDCRT created an adaption of EGRA called the Early Grade Reading and Sign Language Assessment (EGRSLA). After piloting in late 2016, RFS-IDCRT administered the baseline EGRSLA in December 2017 and January 2018 and an endline in May 2018. Eight schools participated, with a total of 71 grade 2 students assessed. No comparison group was included in the study. The analysis showed that while grade 2 students had statistically significant higher scores at endline than baseline on some subtasks—letter name identification, syllable identification, and familiar word reading—their passage reading fluency and comprehension scores did not change significantly from baseline to endline. Grade 2 students signed an average of 12.6 words in two minutes at baseline and 13.4 words at endline from the 46-word reading passage; in addition, nearly six out of ten students at 28 While not a formative assessment per se, PIRLS appears to have provided much of the motivation to revise the curriculum, improve reading, and strengthen evaluation capacity in Morocco; they country scored near the bottom in its participation. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 28 baseline and approximately half at endline did not sign a single word correctly. In reading comprehension, on average, grade 2 students answered only 0.4 and 0.6 questions correctly at baseline and endline, respectively, out of a maximum of five questions. At both baseline and endline, more than seven out of ten students did not answer a single reading comprehension question correctly; at endline, only 34 of the 71 students performed well enough on the reading passage subtask to be asked more than one reading comprehension question. RFS-SSE RFS-SSE’s activity staff adapted new EGRA and Snapshot of School Management Effectiveness tools with help from the National Center for Evaluation and Examinations (CNEE) at the outset of the project and administered a series of EGRAs to measure the impact of the reading lessons and materials. The experimental longitudinal evaluation design assessed two cohorts of students in 46 project schools and 45 control schools (Figure 2). In the first half of 2016, only grade 1 students were assessed as part of cohort 1 because only grade 1 materials were introduced during the 2015– 16 school year. A baseline was conducted in January 2016, followed by the first midline—or Midline 1—in May 2016. During the 2016–17 school year, cohort 1 students moved on to grade 2, and the activity staff selected new grade 1 students for cohort 2. The second midline assessment—Midline 2—for cohort 1 students was conducted at the start of the new school year; this EGRA also served as the baseline for cohort 2. The study ended with an endline assessment of both cohorts in May 2017. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 29 Figure 2. RFS-SSE Evaluation Cohort Design While results at endline showed students in both cohorts had statistically significant gains on every subtask from the September 2016 assessment to the May 2017 assessment, differences between treatment and control groups were more limited. In cohort 1, students from the treatment group outperformed those in the control group at endline on two subtasks—phonemic awareness and syllable identification. The differences in endline gain scores on passage reading and reading comprehension between the two groups, however, were not statistically significant.29 In cohort 2, students from the treatment group had statistically significantly higher gain scores than the control group at endline on four subtasks—phonemic awareness, syllable identification, passage reading, and reading comprehension. Table 3. EGRA Results in Comparison for RFS-SSE from Midline 2 to Endline Subtasks on which experimental group outperformed control group at endline Subtask Cohort 1 Cohort 2 Phonemic Awareness Syllable Identification Non-word Reading Passage Reading Reading Comprehension Listening Comprehension Gain score differences, the average change in scores between the groups, for the passage reading and comprehension subtests for all cohorts and groups, are presented in the following two tables. 29 In the final sample for cohort 1, 1,579 of 1,729 students (91 percent) assessed at baseline were also assessed at endline. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 30 Table 4. Gain Scores for Passage Reading and Reading Comprehension for RFS-SSE Cohort 1 at Endline Subtask Treatment Group Control Group Group Comparison Midline 2 Endline Change since Midline 2 Midline 2 Endline Change since Midline 2 Difference between treatment and control groups at endline Is this difference statistically significant? Passage Reading: Correct Words Per Minute 7.16 26.31 19.15* 7.40 25.69 18.29* +0.62 No Reading Comprehension: Number of questions correct out of five 0.63 1.09 0.46* 0.68 1.09 0.41* 0.00 No Note: An asterisk (*) indicates the average score difference between Midline 2 and Endline is significantly different at p<0.05. Table 5. Gain Scores for Passage Reading and Reading Comprehension for RFS-SSE Cohort 2 at Endline Subtask Treatment Group Control Group Group Comparison Baseline Endline Change since baseline Midline 2 Endline Change since baseline Difference between treatment and control groups at endline Is this difference statistically significant? Passage Reading: Correct Words Per Minute 1.06 13.90 12.84* 0.90 12.13 11.24* +1.77 Yes Reading Comprehension: Number of questions correct out of five 0.05 0.33 0.28* 0.05 0.26 0.21* +0.07 Yes Note: An asterisk (*) indicates the average score difference between Baseline and Endline is significantly different at p<0.05. While the RFS-SSE endline evaluation showed statistically significant impacts across a range of EGRA subtests, it is important to situate these impacts within the broader literature on strategies for improving early grade reading performance in similar contexts. It is not possible, however, to make meaningful comparisons across studies or contexts when outcomes are measured in different ways.30 To enable cross-country comparison of program impacts, the evaluators, therefore, converted RFS-SSE impacts (i.e., subtest mean differences between treatment and control that were observed to be significant at endline) into standardized effect sizes. This metric—Cohen’s d— standardizes different measures to a common scale and is, thus, commonly used to synthesize 30 Though oral reading fluency (ORF) is among the outcome measures used in the cross-country studies, the EGRA assessment itself is adapted within each country and therefore reflects differences in language and difficulty. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 31 scientific research through systematic reviews or meta-analyses.31 The results of this conversion are summarized below in Table 6. Table 6. RFS-SSE Impacts at Endline, by Grade (Cohort) and Subtest EGRA Subtest Grade Treatment Mean Control Mean Difference Standard deviation Cohen’s d Phonemic awareness 2 7.24 6.27 0.97 4.04 0.24 Syllable identification 2 55.18 52.49 2.69 24.75 0.11 Phonemic awareness 1 5.74 4 1.74 4.47 0.40 Syllable identification 1 40.26 35.66 4.6 25.39 0.18 Non-word reading 1 11.99 11.02 0.97 9.62 0.10 Oral passage reading 1 13.9 12.13 1.77 12.53 0.14 Reading comprehension 1 0.33 0.25 0.08 0.65 0.12 Note: All reported differences are statistically significant at p<0.05. Average: 0.18 Effect sizes at endline range from a lower bound of 0.10 standard deviations to an upper bound of 0.4, with an average effect size of 0.18 across all grades, cohorts, and subtests. According to statistician Jacob Cohen, an effect size is conventionally considered “small” when it ranges from 0.2 to 0.5 standard deviations, medium at 0.5 to 0.8, and large at 0.8 and above. Per the United States Department of Education’s Institute of Education Sciences, an effect size of 0.25 and above is typically regarded as “substantive.”32 Subsequent to the launch of the RFS-SSE evaluation, several meta-analyses have been conducted to summarize the impacts of educational programs in low- and middle-income countries (LMICs). Foremost in relevance are 3ie’s 2016 Systematic Review no. 7 on the impact of education programs in LMICs and the World Bank’s 2018 Evidence Review on the effects of early grade reading interventions.33, 34 According to the 3ie meta-analysis, language arts outcomes linked to structured pedagogy programs—defined broadly as programs that introduce evidence-based curricula, instructional planning, training, and materials—have an average effect size of 0.23, with a lower bound of 0.13 and an upper bound of 0.34. While the inclusion criteria for the 3ie review encompassed a broad range of grades/levels and language arts skills, the World Bank review was more narrowly focused on early grade (grades 1-4) reading impact evaluations that utilized the EGRA for primary outcome measures. Reviewed evaluations were focused on early grade reading interventions that emphasized teacher training on evidence-based curricula with simplified instruction, as well as one or more of the following 31 Cohen’s d is given by 𝜇𝜇𝑡𝑡−𝜇𝜇𝑐𝑐 𝜎𝜎 where 𝜇𝜇𝑡𝑡 is the mean score for the treatment group, 𝜇𝜇𝑐𝑐 is the mean score for the control group, and σ is the pooled standard deviation. 32 Graham, J., & Kelly, S. (2018). How effective are early grade reading interventions? A review of the evidence. The World Bank. Retrieved from https://openknowledge.worldbank.org/handle/10986/29127. 33 Ibid. 34 Snilstveit, B., Stevenson, J., Menon, R., Phillips, D., Gallagher, E., Geleen, M., & Jimenez, E. (2016). The impact of education programmes on learning and school participation in low-and middle-income countries. Retrieved from http://www.3ieimpact.org/media/filer_public/2016/09/20/srs7-education-report.pdf. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 32 components: instructional guidelines, coaching and monitoring, supplementary instructional materials, and student assessment support. Given the similarities between the inclusion specifications in the World Bank’s systematic review and the RFS-SSE program design, coupled with the use of EGRA and the large number of USAID￾supported early grade reading programs covered,35 contextualizing RFS-SSE impacts using findings from this review can offer direct insight on the comparative impact of RFS-SSE. Figure 3 shows standardized effect sizes for oral reading fluency, alphabetic by country, with the RFS-SSE impact of 0.14 standard deviations added. Like RFS-SSE, the curricula in such programs is based on a clear sequencing of skills, where higher-order skills build upon lower-order skills (e.g., decoding builds upon the alphabetic principle and phonological awareness). The program must also be sensitive to context and should reflect both the unique characteristics of the language and the reality of schools’ and students’ situations. Finally, the program must effectively prepare teachers to deliver curricula through intensive training and clear, simple guidance that they can take back to the classroom, such as scripted lesson plans or teacher guidebooks.36 Figure 3. Oral Reading Fluency Effect Sizes by Country, as Measured by EGRA As shown above, the Morocco/RFS-SSE fluency effect size is lower than many other early grade reading programs, including many in Africa and the Middle East.37 Like RFS-SSE, two of the evaluations—Jordan and Egypt—used an Arabic EGRA tool with grade 2 pupils and reported reading fluency effect sizes of 0.46 and 0.55 respectively. It is important to note that while enabling environments and assessment difficulty can vary in across contexts, it is notable that the RFS-SSE endline did not observe any statistically meaningful impact on reading fluency scores for grade 2 pupils. 35 All but three of the featured evaluations (Mali, Papua New Guinea, and Tonga) were of USAID-supported early grade reading interventions. 36 Graham & Kelly, 2018, p. 4-7 37 Note that comparing effect sizes in terms of units of standard deviations when the underlying distributions are very different, can be misleading, as the measurement artificially inflates the effectiveness of interventions done on more homogeneous groups, all else equal. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Uganda Tonga South Africa Papua New Guine Nigeria Morocco Mali Liberia Kyrgyz Republic Kenya Jordan Egypt Dem. Rep. of Congo Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 33 To summarize, the RFS-SSE program’s impact on oral reading fluency—considered “the bridge between decoding and comprehension”38—is considerably below impacts conventionally considered to be substantively meaningful. Despite the negligible impacts of RFS, other early grade reading programs in LMICs have observed small to medium impacts using similar curriculum and pedagogical approaches.39 RFS-SSE gains in phonemic awareness—0.4 and 0.24 standard deviations for grades 1 and 2, respectively—were notably larger, however. The program, therefore, had conventionally meaningful impacts on lower-order reading skills, which are considered predictive of higher-order skill acquisition, including fluency and comprehension. The RFS-NPR baseline assessment, conducted in May of 2018, provides some evidence as to whether these lower-order skills translated into better fluency and comprehension over the longer￾term.40 The experimental group surveyed in the RFS-NPR baseline offers representative coverage of the students targeted by the RFS-SSE intervention beginning in 2016. In addition, RFS-NPR’s comparison schools were matched to treatment schools on observable characteristics, including the total number of students, number of classes, school location, electricity, source of electricity, water connection, type of water connection, sanitation, and type of sanitation. Because these matching variables are not those that would be typically influenced by the RFS-SSE intervention over the short term, the RFS-NPR baseline comparison group may offer a credible counterfactual for the original RFS-SSE treatment cohorts. If the above assumptions hold, grade 3 pupils assessed in the RFS-NPR baseline are approximately representative of cohort 2 and would have had exposure to RFS-SSE during grades 1 and 2. By the end of grade 3, however, none of these pupils performed statistically significantly better than the comparison group. Furthermore, grade 2 pupils—who are representative of the RFS-SSE evaluation’s cohort 2—may not have sustained any of the EGRA subtest gains observed in the RFS-SSE endline.41 It is important to note that this analysis assumes minimal attrition and contamination between RFS￾SSE and non-RFS-SSE schools from 2016-18. In other words, pupils in the RFS-SSE treatment schools should have remained in the treatment schools, with a minimal influx of new pupils post-RFS-SSE (this is generally supported by the low rate of attrition observed in the RFS-SSE evaluation). The extent to which the RFS-NPR baseline data can reliably speak to the long-term impacts of the RFS￾SSE program is an area for further exploration with RFS-NPR IPs.42 RFS-NPR RFS-NPR has built on the work of RFS-SSE by piloting new early grade reading curriculum and materials for grades 1-4 in the same experimental schools targeted by RFS-SSE and then scaling them nationwide as well as expanding reading enrichment activities. In addition, the project focuses on enhancing instruction through better teacher training and improving systems for assessment. The implementation plan for the scale-up of the new curriculum has shifted throughout the life of the project based on the MoE directive. Implementers did not plan to expand the curriculum nationally until the third year of the project—2019–20—with grades 1-4. The national scale-up was accelerated, however, starting with grade 1 in 2017–18 and grade 2 in 2018–19. Scale-up for grades 38 RTI International. (2016). Early Grade Reading Assessment (EGRA) Toolkit: Second Edition. Retrieved from https://www.globalreadingnetwork.net/resources/early-grade-reading-assessment-egra-toolkit-second-edition. 39 See Annex V for program and evaluation contextual details. 40 Management Systems International (2018). Baseline Report 2018: Reading for Success – National Program for Reading (RFS-NPR) in Morocco. Unpublished report draft. 41 Grade 2 pupils in the SSE did perform statistically significantly better than the comparison group (0.23 standard deviations) on the newly introduced dictation subtest. 42 At the time of writing, MSI has yet to confirm the years of the MASSAR data used in the statistical matching of comparison schools. In addition, balance checking on time invariant pupil- and school-level characteristics gathered through the NPR baseline could offer further insight on the credibility of the NPR comparison group as a counterfactual for SSE. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 34 3 and 4 took place as intended in 2019–20. In addition, the MoE has added grades 5 and 6 to the project’s scope. When the activity first launched, RFS-NPR was able to collaborate with RFS-SSE, which had yet to close. In the summer of 2017, RFS-NPR produced its first set of teaching and learning materials for grades 1 and 2. Only prioritized changes could be made to materials because of the compressed time frame for development and revision. Associated training activities were also developed, including training for grade 1 teachers nationwide. Preparation then began for the 2018–19 school year. New curricula were developed and validated for grades 1- 4; new materials were created for grades 1 and 2, which would be distributed nationally, and for grades 3 and 4, which would be piloted in experimental schools. As part of new materials for grades 1 and 2, national textbooks and teacher guides were developed during a series of trainings and workshops with MoE staff, textbook authors, and editors. In the fall of 2018, the plan for nationwide teacher training for the grade 1 and 2 scale-up fell through due to a dispute between the MoE and inspectors, who boycotted regional trainings they were intended to lead because they wanted extra compensation for their work from MoE. This boycott did not impact training activities for the project’s experimental schools. Additional activities for RFS-NPR have included the development of an online training program for in-service teacher development; the design of a pre-service training curriculum that incorporates the new approach to early grade reading; and the development of other activities encompassing the project’s intermediate results, including formative assessment, supplemental materials, and community engagement. In 2018, RFS-NPR conducted an Early Grade Reading and Writing (EGR/W) baseline study to allow for future evaluation of the effectiveness of its interventions in improving the reading and writing skills of students in grades 1-4. The EGR/W for all four grades included six subtasks: listening comprehension, syllable sound knowledge, nonword reading, oral reading fluency, reading comprehension, and dictation. Students in grades 1 and 2 also participated in a phoneme isolation subtask. For its sample, the study selected 90 experimental schools (the same schools as in RFS-SSE) and 90 validation schools. Intermediate result 3.1 The first of three intermediate results focus on reading skills improved. Progress on this IR is tracked with two top-level indicators: • Number of learners reached in reading programs at the primary level, and • Number of learners in primary schools or equivalent non-school based settings reached with USG assistance. According to end-of-project reports, RFS-SSE reached a total of 11,742 students at the primary level—in grades 1 and 2 in 91 intervention schools with its revised phonics-based reading curriculum—and RFS-IDCRT reached 233 students at the primary level—in grades 1 and 2 in ten intervention schools. RFS-NPR has reached a greater number of students than initially anticipated due to the accelerated scale-up. From October 2018 to September 2019, RFS-NPR reported in its annual report that the activity reached 595,622 students in reading programs at the primary level. Despite these reported numbers, evaluators’ visits to provinces where the scale-up has occurred revealed that not all teachers had participated in trainings or, if they had participated, the training was very short. At both scale-up schools visited, three of six teachers said they had not participated in any trainings. One Grade 1 teacher who had not received training said she had done research on the Internet to supplement her teaching, notably on YouTube. MoE officials in the scale-up provinces echoed the teachers. One provincial MoE director observed that some teachers are not using the new approach because they have not participated in any trainings due to the sheer number of teachers in the province. He had difficulty envisioning the possibility of training at such a scale. One inspector said Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 35 he had not trained certain teachers yet, but he was planning to solve the problem by offering to coach them during field visits to classrooms. Intermediate result 3.2 The second of three intermediate results centers on community support for reading increased. Progress on this IR is measured with one top-level indicator: • Increased provision of reading activities outside of instructional time (disaggregated by public versus private). No RFS activities, however, reported against this specific indicator. In its interventions aimed to increase community support for reading, RFS-SSE collaborated with civil society organizations (CSOs) to reduce students’ reading loss during summer vacation. The activity measured its progress against two indicators: • Number of students participating in community-based enrichment programs, and • Number of local CSO representatives and organization staff receiving training with the support of USAID. RFS-SSE reported that 520 students participated in community-based enrichment programs, and 25 representatives from ten CSOs received training. RFS-NPR continued RFS-SSE’s work with CSOs during the summer of 2019. For this intervention, RFS-NPR reported on the second of two indicators: RFS-NPR trained 18 representatives from three CSOs. RFS-NPR developed other plans to engage with the community and parents on reading. Despite RFS￾NPR’s interest and ideas, the accelerated scale-up at the outset of the activity, coupled with the expanded scope that will include grades 5 and 6 in 2019, exhausted the resources needed to follow up on plans and fully implement initiatives as first conceived and planned for this IR. As a result, RFS￾NPR respondents reported the activity would remove all community support activities targeted for this IR. Summer enrichment reading activities are the most notable programs implemented under this IR. RFS-SSE first launched these activities in 2017. RFS-SSE supplied grants to ten CSOs to develop and implement enrichment activities. After participating in capacity-building workshops with RFS-SSE, the ten CSOs implemented programs that lasted two weeks on average and aimed to reduce students’ learning loss during summer vacation. These reading programs centered on improving children’s reading skills and building their confidence through engaging, active reading games. Overall, 4.4 percent of the 520 students who benefited from RFS-SSE interventions also participated in the summer activities. All the programs culminated in the final events that the community attended. “You got a sense the community was involved and grateful,” one CSO implementer said. RFS-SSE conducted an EGRA evaluation of its summer enrichment activities. The EGRA included a treatment group of students who participated in enrichment activities and a control group of students who did not. RFS-NPR administered a baseline EGRA before the activity was implemented, followed by an endline EGRA afterward. There were no significant differences between the control and experimental groups’ gain scores from baseline to endline on any of the six subtasks administered. In addition, the results revealed students in both the control and experimental groups had statistically significantly lower scores at endline on three of six subtasks—syllable identification, nonword reading, and passage reading—but statistically significantly higher scores on one subtask— “YOU GOT A SENSE THE COMMUNITY WAS INVOLVED AND GRATEFUL.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 36 reading comprehension. These summer enrichment programs continued with RFS-NPR. In 2019, the activity selected three CSOs—including two that RFS-SSE had used—to implement interventions. The three CSOs all implemented the same RFS-NPR-developed programming in the morning, followed by their own discretionary programming in the afternoon. These summer programs, according to activity implementers, focused more intensively on reading and included more involvement from parents than RFS-SSE’s enrichment program. RFS-NPR also trained the CSOs on management and program content. While the impact of these RFS-NPR activities on reading performance was not formally evaluated, CSOs who implemented the activities spoke to their success during interviews. In 2019, one CSO organized enrichment activities for 50 children during the month of July. In addition to the RFS-NPR￾mandated content, which included four hours of reading per day, the CSO engaged the children in theater, games, songs, and stories and also conducted three trainings with parents on how to continue engaging with their children in August to retain the reading skills they had attained. During these sessions, parents told CSO officials how their children’s reading abilities had improved, and the CSO itself noted progress; for example, children who had difficulty pronouncing letters had improved that skill by the end of the month. Students and parents were assessed as part of the activity, but evaluators were unable to obtain results from IPs. A CSO implementer mentioned that MoE officials conducted an evaluation on tablets before, during, and after its activities, but he had not seen any data. RFS-NPR initially planned to implement more activities as part of IR 3.2, but those plans have changed due to the scale-up of the curriculum to grades 5 and 6. According to its monitoring, evaluation, and learning plan, RFS-NPR originally envisioned several packages under its Result 3— Reading enrichment programs expanded. These packages included the establishment of reading clubs, the provision of diagnostic tools and early warning systems for struggling students, the development of activities to increase parental involvement in children’s reading, and the support to CSOs for summer programs. According to interviews with both USAID/Morocco officials and implementers, some of these initiatives have been developed, including systems for parental engagement and communication with parents, but were not fully implemented as envisioned because “nearly all” of RFS-NPR’s resources went to curriculum and materials development and teacher training as part of the accelerated national scale-up. Now, with scale-up for grades 5 and 6 ramping up, Result 3 is going to be dropped altogether from RFS-NPR. One respondent regretted the component’s elimination from RFS-NPR: “We can’t limit teaching to the classroom, and things like theater, reading clubs…are so valuable. It’s a shame that that component will…be dropped now from RFS-NPR. [The] MoE does believe this is important, but it’s just not a priority right now in their mind.” Despite limited implementation, RFS may have had an indirect impact on community-focused literacy interventions. Some MoE officials and school directors mentioned student participation and interest in the Arab Reading Challenge launched by the United Arab Emirates as a sign of RFS’s success. One school director in an experimental school said that RFS has given his school a boost for the contest. Other MoE officials mentioned the establishment of reading clubs at schools, but it was unclear if these were directly linked to RFS. “WHEN PARENTS GOT INVOLVED, IT WAS GOOD. AND IF SOME WERE IN GOVERNMENT, THAT ADDED A DEPTH AND SUPPORT. BUT IT WASN’T SYSTEMATIC.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 37 Intermediate result 3.3 The third of the three intermediate results aims for the improvement of the policy environment to support reading. This IR is measured with one top-level indicator: • The number of laws, policies, regulations, or guidelines developed or modified to improve primary reading programs. None of the four RFS activities reported against this indicator or devised alternative indicators associated with this intermediate result. Therefore, evaluators relied on insights from respondents to determine RFS’s impact on national education policy. When asked if RFS had strengthened policies, guidelines, or laws around reading, respondents’ understanding of the question varied. In some cases, respondents understood “policy” to mean a manual or guide—as in “the MoE distributed a student’s manual for the new reading approach” or “the MoE distributed a methodological framework.” However, most respondents, at both central and local levels, referenced two things—the loi cadre and the national strategy. For example, a respondent from the evaluation division of the MoE reported that the legal framework would orient the entire reform, and that reading “was at the center of this concern—that is, the policy was in reaction to this issue that the reading reform is amongst the most important components of the education system at the primary school level.” A trainer in the field also noted that “the legal framework and national strategy declared the necessity of mastering the Arabic language, and that the training centers published guidance to improve reading. The Arabic language has become the center of all strategies and general guidance.” That said, one CRMEF trainer commented that while the legal framework provided guidance for teaching in general, it did not provide specific guidance for reading. Moreover, the farther the respondent sat from Rabat, the more inclined to defer to the central government, some even saying that RFS had not had any political impact. One inspector said, “We’re interested in policies at the provincial level, but we don’t have the power to change or establish laws or texts, that’s at the central Ministry.” One implementer referred to the revised curriculum as a policy and a great achievement, noting that it was the first revision in Morocco since 2002. Previously, reading had been only seen as only one piece of the Arabic curriculum, but when MoE decided to revise it, educators used reading as a mechanism to update some other things as well, and it snowballed—hence listening and speaking as a major piece of the storybooks. However, the director of the same activity took a different view, countering that “you don’t need policymaking to get practical things done; rather, new practices can lead over time to changes in policy… Here the law doesn’t bring innovation, but innovation changes the law. Tell me what is good, and I can create a law for it, it’s how the system functions.” 2. IMPLEMENTATION: HOW AND WHY DID FACTORS IMPEDE AND/OR FACILITATE THE ACHIEVEMENT OF THE RFS PROJECT'S DEVELOPMENT OBJECTIVE? To understand the implementation of USAID/Morocco’s RFS activities, evaluators explored how and why different factors either facilitated or impeded RFS activities from reaching the program objective of enhancing educational development for children at the primary level in Morocco. Evaluators asked key stakeholders—including USAID/Morocco officials, USAID implementers, MoE officials, school directors, teachers, and parents—about factors they encountered or observed during their engagement with RFS activities.43 In addition, we discussed the role of M&E data in project 43 School directors, teachers, and parents were drawn from a purposive sample of RFS pilot and scale-up primary schools engaged in RFS￾SSE and RFS-NPR, as well as schools where RFS-IDCRT was implemented. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 38 implementation with RFS-NPR, RFS-SSE, and RFS-IDCRT activity personnel. Findings are summarized in this section by relevant evaluation sub-questions. Relevant support from project source documents is also cited. 2.1. HOW AND WHY DID FACTORS IMPEDE THE ACHIEVEMENT OF THE PROJECT'S DEVELOPMENT OBJECTIVE? Although respondents overwhelmingly identified MoE’s collaboration and leadership in implementing as key to the project’s success, challenging shifts imposed on the implementers’ work plans and timelines due to fast changing political realities and shifting MoE priorities required the implementers, as well as USAID and the MoE, to adapt quickly and demonstrate a high level of flexibility and responsiveness. In KIIs and FGDs, both activity implementers and USAID/Morocco officials shared that they felt challenged by MoE’s expectations around the timing and pacing of their activity’s implementation, which at times negatively affected the quality of delivery of the intervention and had indirect impacts on implementers’ staffing and initial work concerning indicators and M&E systems. These impeding factors may have stemmed in part from earlier successes; USAID/Morocco reported that the MoE wanted to seize the momentum created by the encouraging results of the RFS-SSE and knew that if the GOM had to scale up immediately to keep the momentum of change going. These challenges should be considered in light of the facilitating factors discussed in the following section, including support provided by USAID/Morocco in terms of facilitating contacts and implementation. The MoE played a leading role within a political context that required fast action. Several implementers noted that with the MoE in “the driver’s seat,” officials could call for a fast-paced implementation or shift the activity’s priorities. At times, these changes required some compromises to be made by implementers in terms of the content or quality of products being delivered. For example, one RFS-SSE implementer noted that after a slow start-up, MoE officials expected the grade 1 curriculum to be developed, printed, and disseminated into schools with short turnaround time and half-way through the school year. While RFS-SSE was able to meet the MoE’s desired timeline, the implementer felt that the curriculum’s quality suffered; s/he believed the curriculum would have been stronger, more coherent, and better thought out if the process had aligned with the start of the following school year in September. Indeed, the MoE’s call to review and revise grade 1 and 2 curricula was a shift from RFS-SSE’s original priority of piloting the reading program and supplementary materials, a change in implementation that was seen as valuable by the MoE. USAID/Morocco reported both strong leadership and support to these changes, as is discussed further in the following section. Similarly, both implementers and USAID/Morocco officials described RFS-NPR’s national-level scale￾up and roll-out of the grade 1 and 2 curricula, as well as the addition of grade 5 and 6 curricula, as a success in terms of being responsive to the MoE. The MoE requested these changes in implementation to meet their new priorities in ensuring that the curricula reached as many grades as possible as quickly as implementers could find feasible. The same respondents, when reflecting from a project implementation lens, found such changes challenging to implement because they required major revisions to the activity’s work plan. As one RFS-NPR implementer noted, “The RFS-NPR team needs more time or to be of a larger size to avoid sacrificing the quality of the technical work when demands from MoE are so rapid and broad as they are now (accelerated scaling, grades 5 and 6). MoE specifications on books also make the cramming of all this new material so hard and time￾consuming, putting further pressure on the team.” Implementers found RFS-NPR’s large-scale pivots difficult, especially as the changes can contribute to knock-on effects, ensuring they have the right staff, skillset, indicators, and monitoring, evaluation, and learning framework. While challenging, as implementers have described, USAID/Morocco officials observed that these challenges result in great successes, as discussed in the following section. During report validation, one USAID/Morocco official contextualized this impeding factor within USAID’s larger goals, observing that “USAID has been trying to promote adaptive management and flexibility -- so in some cases, the ability to rework workplans when presented with an opportunity like the host government wanting to move Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 39 to full scale expansion, at their cost, would be considered a very significant success story.” USAID/Morocco officials echoed the concerns raised by implementers, noting that the speed with which implementers had to move was challenging and at times hindered the quality of the work. For example, one USAID official described the impact of projects having to resort to two-staged training of trainers approaches to train teachers, noting that the cascade model is “leaky,” namely that quality of training can be deeply affected by the time the content reaches teachers. Both implementers and USAID noted the projects would have benefited if they had more time to “mature” before scaling up. However, they reported such compromises as key to ensuring sustainability and a demonstration of cooperating country ownership that is consistent with the journey to self-reliance. With RFS-HICD, implementers cited initial delays in receiving MoE’s approval, followed by high expectations and shortened timelines to produce deliverables. As a result, implementers felt the quality of work they were able to produce was negatively affected by the initial delays. For example, while the RFS-HICD activity was originally envisioned and contracted as a six-month program, a three-month delay in obtaining MoE approval greatly reduced the time available for experts to conduct the necessary fieldwork, data collection, analysis, and report writing. Delays were compounded by changes in Ministry leadership, including the Minister of Education. USAID/Morocco respondents characterized the delays differently by highlighting high expectations from MoE decisionmakers, including the secretary general, central, and regional officials, who expected more analytical depth in the report that HICD presented. Implementers also discussed challenges related to reporting to “multiple masters.” For example, implementers are contracted and funded by USAID to carry out a certain scope-of-work; however, they work with and respond to the MoE officials who require them to make compromises and revisions to deliverables. Expectations around the lines of communication and reporting were also unclear, and some implementers were uncertain if they could engage directly with the MoE or needed to communicate through USAID. This shift in how USAID is working with the MoE to put the country on the path to self-reliance is the result of high levels of country ownership and collaboration on programming. Under USAID's new self-reliance agenda, USAID/Morocco respondents reported striving “full force to get this level of government collaboration and buy in – which does sometimes present implementation challenges that need to be addressed.” Inspectors within MoE had conflicting views on roles and responsibilities that affected RFS-NPR’s teacher training roll-out. Both central MoE officials and USAID/Morocco officials identified the inspector boycott of RFS activities in 2018 as hampering RFS-NPR’s training efforts for scale-up activities in grades 1 and 2. The cascading training design began in July 2018 with a training in Rabat of inspectors from all 12 regions in Morocco. These inspectors were then expected to train teachers in their respective regions. Inspectors, however, did not consider this work as falling within their typical job as supervisors; therefore, they considered training teachers for the RFS-NPR activity as additional work for which they should be provided additional compensation. Conversely, officials from MoE central offices understood the training as falling within inspectors’ job descriptions and duties. Based on such differing views, inspectors boycotted teacher trainings, and school directors were reluctant to support teachers in the new approach. As a result, and as noted in an RFS-NPR quarterly report, the 2018 scale-up was “done in a de facto manner. Teachers were left to figure out on their own how the new textbooks might be used.” RFS-NPR implementers and CRMEF staff ended up having to train teachers during the 2018-19 school year across Morocco. Encouragingly, according to an interview with central MoE officials, the MoE was able to negotiate a resolution with the inspectors to ensure relevant trainings would take place in their respective regions during September and December of 2019. According to RFS-NPR implementers, training activities have gone “way better” during this span. In July 2019, training was conducted with master trainers, who then led regional trainings with inspectors. Inspectors then had from September to December 2019 to determine with teachers when training should be delivered. The design was considered more as continuous professional development than structured training. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 40 Lack of allocated resources, logistical constraints, and other challenges affected delivery of trainings. Respondents noted issues around a lack of allocated resources, logistical constraints, and other challenges around implementing training. Issues included insufficient time dedicated to teacher training; limited availability of, and follow-up to, in-service teacher training; limited availability and willingness of inspectors to lead trainings and provide follow-ups; and lags between various trainings due to the cascade model. Within the cascade model, the quality of content and delivery, the distance teachers need to travel to participate in training, and the costs associated with participants’ transport, lodging, and food were also challenges. Finally, implementers and USAID/Morocco officials noted that delays in communication between MoE’s levels meant that teachers were sometimes unaware they were expected to attend training until the day of. One implementer summed up the importance of devoting resources to training: “MoE needs to really invest in teacher training, and that is to even make a cheap model of training work at all. Teachers need real training to get this initiative going and make it official.” One USAID/Morocco official suggested that training includes 15 days at a minimum, not five days as currently structured. While respondents from MoE’s central offices viewed the new reading approach positively, some regional and provincial officials were initially hesitant about it. As previously discussed, USAID conducted extensive research prior to the design of the RFS PAD, including reviewing the quality and appropriateness of the existing curriculum, teaching and learning materials, and reading approach in the early primary grades in Morocco as well as during the RFS-SSE pilot. Respondents often saw this collaboration as a crucial step to creating an evidence base and engaging in the advocacy needed to introduce a new reading approach in Morocco. However, even after the MoE saw the benefits of the new approach, some local officials, such as inspectors, were skeptical. One inspector reported doubting the credibility of the new reading model’s approach, relating that individuals within the MoE questioned changing something long defended and had uncertainty around how to convince teachers to accept the new method. Moreover, he had to be convinced during training, and reported that some teachers would be especially hard to convince to adopt new approaches. Another inspector shared his perception that MoE’s decision to adopt a new reading approach was made without the involvement of lower-level officials and, therefore, those same officials were resistant to it. In addition, one AREF director in a scale-up region noted the lack of communication from the central office regarding the new approach’s inception. He ended up contacting former colleagues in experimental regions to learn what happened during the pilot phases of RFS. Insufficient supplementary materials at the school level, especially in the scale-up regions of RFS-NPR. Multiple stakeholders at the school and local MoE levels—including teachers, school directors, and inspectors—cited a lack of supplementary materials and delays in receiving materials as a challenge in implementing the new reading approach. Teachers at one RFS-NPR scale￾up school noted that it was a month into the school year before they received the teacher’s guide. Moreover, once teachers did receive materials, many felt they were insufficient. For example, one AREF director in a scale-up region commented that while he had a manual and guide at the activity’s launch, these two materials were not sufficient—other instructional materials that simplify reading are necessary for rolling out the new approach at scale. Similarly, one teacher in an RFS-NPR school noted that although the new approach is nice, without sufficient materials, “it’s nothing.” As noted in research question 1, teachers observed that phonetic reading demands lots of support and means. Several teachers reported spending their own money to create supplementary materials to make up for the lack of activity-provided ones.44 When discussing the lack of supporting materials, one school director referenced the need for photos, printouts, and pictures to go with the stories. 44 For example, teachers cited posters and images of letters as materials they were providing using their own funds. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 41 Apart from FGDs with teachers at RFS-NPR scale-up schools, the lack of materials was also discussed at some experimental schools. Several USAID/Morocco officials and implementers also recognized that the production and distribution of materials must be accounted for when considering national scalability. Limited implementation of community and parental engagement interventions was viewed as a constraint to RFS’s overall potential impact. Multiple stakeholders—including USAID/Morocco education officials, USAID implementers, MoE officials, and several school directors—cited the lack of strong community and parental engagement at the local level as an impediment to RFS’s overall objective of supporting Moroccan children’s reading. For example, one USAID/Morocco official commented that, aside from the summer school enrichment programs organized by RFS-SSE and RFS-NPR, RFS did not have the opportunity to implement any activities that directly targeted parents. In addition, in the fall of 2019, in order to accommodate the MoE’s wishes for it to expand its scope of work to grades 5 and 6, RFS-NPR eliminated its intermediate result devoted to community and parental engagement, thereby discontinuing plans to implement certain interventions it had been developing. Stakeholders attributed this absence to several different issues or constraints: 1. The intense focus on revising the curriculum left little opportunity for other program aspects; 2. Working with CSOs on parental engagement were cited as lower priorities for the central MoE; 3. Parents' limited awareness of RFS activities and the new approach to reading; and 4. Low parental literacy levels hindered a family’s ability to support their children’s reading. One USAID/Morocco official, when reflecting on the RFS-SSE activity, commented that the demands at the curriculum level left the activity’s small team “no bandwidth” for other things. Respondents observed that working on parental engagement directly or through CSOs seemed to be lower priorities for the central MoE. Similarly, USAID/Morocco officials noted that it was unlikely that RFS￾SSE’s and RFS-NPR’s approach—with summer enrichment programs—would be continued by the MoE when the project closed. One CSO respondent who implemented summer enrichment activities noted that MoE does not have the resources or structure to take on these activities in the future. Multiple respondents listed parents’ limited awareness of RFS activities and the new approach to reading as well as low literacy levels as hindrances to a family’s ability to support children’s reading. One teacher trainer felt that only inspectors and trainers were aware of RFS-NPR and also reported that, in the trainer’s rural district, it is challenging for parents to ask their children to read because many are illiterate themselves. Several school directors shared similar concerns, citing parents’ illiteracy as a barrier to engagement in and support for their children’s reading. Moreover, in FGDs with parents, evaluators found that many parents are unfamiliar with RFS, RFS-NPR’s objectives, or RFS-SSE’s reading approach. Many of the same respondents expressed a desire for USAID/Morocco to support extracurricular activities such as parent and community support programs. One USAID/Morocco official described several interventions that could not be implemented because of limited resources, including a television awareness campaign that was planned to feature a popular Moroccan actress and parents of varying reading levels. These short clips were expected to discuss the importance of reading as well as show parents reading to other illiterate parents. In addition, a community intervention was being designed to feature an expert discussing strategies with parents on how to promote reading at home regardless of their literacy level or mother tongue. For RFS-IDCRT, positive engagement of parents and community members was noted—such as social media groups established by teachers—alongside stronger and more targeted parental engagement as critical for supporting children who are deaf and hard of hearing, especially given the broader, systemic challenges children with disabilities encounter in obtaining a quality education. Most pertinently, as highlighted in the RFS-IDCRT evaluation report, teachers at RFS-IDCRT schools expressed concern that parental engagement was not sufficient, attributing parents’ low level of Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 42 engagement to “a lack of understanding, pre-existing biases about deafness, and a lack of MSL abilities.”45 Targeted engagement of parents can play a crucial role in addressing these issues as well as better enable parents to support their children who are deaf and hard of hearing in their education. While RFS-IDCRT stakeholders shared positive experiences with the MSL Clip and Create software, a number of constraints were identified as impeding its broader utility and more sustainable use. The creation of 3,000 signs in the MSL Clip and Create Software is a notable achievement, yet the software did not necessarily meet the needs of teachers (and parents); for example, one implementer commented the number of signs was insufficient and it lacked regional variation. Similar constraints were outlined in the RFS-IDCRT evaluation report which noted the software being utilized as more of a supportive tool because it did not have all the vocabulary teachers needed nor more complex or in-depth subject content areas.46 One teacher also commented on how the stories were not viewed as aligned to the Moroccan context. Again, this was echoed in the RFS-IDCRT evaluation report, which highlighted stakeholder concerns that “selection of words and clip art images included in the software were culturally inappropriate, biased, and not streamlined to the curriculum.”47 Another issue that emerged in both the discussions with teachers, as well as the previous evaluation report, was the accessibility of the software, namely that access to the software on computers can be quite limited, whereas it would be much more useful and practical if teachers (and parents) could access and use the software–especially the dictionary or thesaurus–via smartphones.48 One of the largest impeding factors, however, relates to the transfer or handoff of the management of the software at the end of the project. More specifically, one USAID official noted the MoE only received a “static” version of the software from the implementers at the end of the project, and is, therefore, unable to fully take over the software in a way that will enable them to continue using and modifying it over time. This was also alluded to by an RFS-IDCRT implementer who referenced an issue related to intellectual property and the RFS-IDCRT evaluation, which noted that “IDCRT was not instructed to create a software that could be owned and managed by the Moroccan government after the end of the project.”49 Moreover, the RFS-IDCRT evaluation summarized the situation as follows: the MSL Clip and Create Software is “propriety and cannot be modified or updated without a skilled technical team, and due to the limited number of words included, USAID respondents noted their doubts about whether the software will continue to be useful and impactful for teachers and students in project schools.”50 As highlighted by the constraints discussed previously, careful considerations and planning are essential to determine how best assistive technologies and software can be utilized in a way that is accessible, appropriate, and sustainable within the Moroccan context in the future. Broader systemic issues around the educational environment and situational context of children with disabilities in Morocco–especially students who are deaf and hard of hearing–impeded the project’s ability to support improving their reading and MSL skills. Although outside the scope of the IDCRT project, children who are deaf and hard of hearing face important-to-acknowledge constraints in obtaining a quality education. Key issues noted in interviews and in the IDCRT evaluation report include: 45 School-to-School International, Evaluation Report: Moroccan Sign Language Assistive Technology for Reading Improvement of Children who are Deaf/Hard of Hearing, (2018). P.53 46 Ibid. P.50 47 Ibid. P.47-48 48 Ibid. P.48 49 Ibid. (2018). P.56 50 Ibid. P.50. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 43 • Prevalence of negative attitudes, perceptions, and biases against individuals who are deaf or hard of hearing in Morocco; • Limited formal, systematized training of teachers in special education, inclusive education practices, and/or working with students who are deaf or hard of hearing specifically—for example, pre-service training, certification or specialization; • Limited role of government—and MoE specifically—in Deaf education (most students who are deaf or hard of hearing are educated in segregated centers run by non-governmental organizations (NGOs) without direct oversight or management from the MoE); • Current curriculum not adapted/appropriate to the specific needs of the deaf and hard of hearing student population; • Limited documentation, standardization, and/or linguistic understanding of MSL, including teachers’ and parents’ limited MSL fluency and skills; and • Limited options for students who are deaf or hard of hearing to attend pre-school. 51 2.2 HOW AND WHY DID FACTORS FACILITATE THE ACHIEVEMENT OF THE PROJECT'S DEVELOPMENT OBJECTIVE? MoE and USAID/Morocco have a strong, collaborative relationship with MoE “in the driver’s seat.” Interviews with officials from central MoE and USAID/Morocco reiterated how essential the collaborative, trusting relationship between USAID/Morocco and the MoE was to RFS’s success. One USAID/Morocco education official described the strength of the relationship and trust between USAID and the Ministry: “They have respect for us, and we have respect for them. We have a very, very collegial relationship. We work very closely with them, and we have been very responsive to all of their demands, and we’ve also made sure they are always in the driver’s seat.” Central MoE officials echoed the positive sentiments about the collaborative nature. One official described his involvement with USAID/Morocco as “completely perfect,” noting that he works in consultation with USAID/Morocco, is in direct contact with the Mission, and was in the room with USAID/Morocco during RFS’s conception. Similarly, an MoE official shared the example of branding the new classroom materials—USAID/Morocco obtained a logo waiver from USAID’s headquarters in Washington, D.C., so that the newly produced textbooks bear only MoE’s logo, thus demonstrating MoE’s full ownership of the new curriculum and materials developed under RFS. Evidence-based programming demonstrated success. USAID/Morocco used a two-year project—RFS-SSE—to pilot the new phonetic reading approach and build evidence-based research with the MoE to inform design, as well as the later scale-up and expansion of RFS activities. This approach ultimately allowed stakeholders to gain an in-depth understanding of the issues at hand in Morocco and generated strong buy-in around how to address those issues. One USAID/Morocco official noted that it takes time to build trust and spending two years on evidence- and relationship￾building was essential to the process. Similarly, other USAID/Morocco officials highlighted how evidence-building was critical to designing RFS’s activities in a way that would enable MoE ownership; s/he shared that engaging, training, and supporting the MoE in taking part in the studies, research, and analyses were critical. For example, one of the research studies analyzed gaps in reading textbooks; therefore, textbook editors were targeted as part of RFS activities. One USAID/Morocco official noted how editors said that they had never met with stakeholders before. The same respondent felt that the MoE’s system of textbook review and revision developed during RFS should 51 Ibid. “THEY [MoE] HAVE RESPECT FOR US [USAID], AND WE HAVE RESPECT FOR THEM. WE HAVE A VERY, VERY COLLEGIAL RELATIONSHIP.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 44 remain sustainable. Moreover, respondents felt the research enabled the MoE to identify problems related to Morocco’s approach to early grade reading. Once the Ministry’s central office internalized the issues, then officials acted quickly and could explain their decision to revamp the curriculum because of the extensive evidence RFS had gathered. One implementer highlighted how—based on the evidence developed under RFS—the MoE, USAID, and other agencies had gained consensus around the problems and are working together to address these issues. Despite the advantages of building consensus through research, one USAID/Morocco official did acknowledge the inherent drawbacks, as detailed later in this report. The official said that getting the MoE’s buy-in takes considerable time to germinate, but once they are convinced to act on an idea or approach, work must happen quickly: “Of course, that meant once things are moving, they had to move really fast, which was very challenging and sometimes hindered the quality of the work.” When parents and communities were reported to be engaged with activities, their involvement yielded more positive results. RFS-NPR respondents reported that when CSOs engaged parents, students were more motivated to read at home. For example, some parents of children engaged in the summer enrichment program indicated they no longer have to push their children to read. Parents also shared with CSOs that their children’s reading levels have improved. CSOs only worked at a small scale, however. For instance, in the summer of 2017, each CSO worked with a group of 50 students on average. Respondents felt that providing ongoing, structured engagement with parents would prove challenging at a national level. The RFS-IDCRT project also highlighted some positive outcomes that emerged from engagement with parents. An RFS-IDCRT implementer noted the activity specifically included trainings targeted at families, and family members also partook in training for teachers and academy representatives. By encouraging and providing support to parents to learn MSL skills—for example using the Clip and Create Software to learn signs for letters and words—RFS-IDCRT and its associated schools were able to help bridge the communication gaps between proactive parents and their children who are deaf and hard of hearing. However, as noted in the RFS-IDCRT Evaluation Report, overall parental engagement with the schools was still viewed as uneven and low overall. A more targeted focus and investment were cited as needed, especially to strengthen family members’ understanding of deafness as well as communication skills in MSL to better equip them to support their children’s education.52 Local MoE officials observed positive engagement from teachers and buy-in at the school level. Several inspectors, provincial directors, and AREF officials noted that teachers were engaged and open to the new approach to teaching reading, especially after observing positive results in their classrooms when applying the new methods. One local MoE official observed that there had been a genuine commitment to continue with this program because it is clearly seen as a success. The same official also described the syllabic approach as an easier approach leading to more positive impacts on student learning. While some respondents noted that some teachers were reticent about the new approach—especially those older or closer to retirement who were not eager to adopt a new style of teaching—this was the exception rather than the rule. RFS-IDCRT’s engagement of experts and individuals with disabilities supported positive shifts in beneficiaries’ perceptions around the capacities and abilities of individuals who are deaf and hard of hearing. One RFS-IDCRT implementer noted how before the project, “the perception of what deaf people could achieve was very low, and that really changed.” More specifically, engaging and training individuals with disabilities in the project, as well as bringing in international experts with disabilities—including those with advanced degrees—had a considerable 52 School-to-School International, Evaluation Report: Moroccan Sign Language Assistive Technology for Reading Improvement of Children who are Deaf/Hard of Hearing, (2018). P.53 Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 45 impact on shifting beneficiaries’ perceptions about the abilities, capacities, and potential of people who are deaf and hard of hearing. The RFS-IDCRT evaluation report highlighted similar findings, noting the positive impact of engaging and working with individuals with disabilities in the project. However, the report also recommended that to truly support more systematic inclusive practices, there is a need to engage experts with disabilities more purposefully in the design and leadership of activities such as, for example, in the Steering Committee, project leadership, as well as teachers and leadership roles within schools.53 IDCRT facilitated increased dialogue, collaboration, and sharing around the education of students who are deaf and hard of hearing in Morocco. Discussions with RFS-IDCRT implementers as well as USAID officials pointed to the collaboration and sharing of resources, knowledge, and lessons learned across project schools and Deaf associations as one of the most valuable contributions of the project, and a key component to helping improve education for students who are deaf and hard of hearing. Similar sentiments were highlighted in the RFS-IDCRT evaluation report.54 Furthermore, these collaborations have been cited as leading to stronger advocacy for the education of children with disabilities and helping lay the groundwork for the MoE’s increased awareness, attention, and involvement in supporting inclusive education and children with disabilities. 2.3 TO WHAT EXTENT HAVE THE ACTIVITIES RESPONSIVELY USED (OR ARE CURRENTLY USING) ACTIVITY-LEVEL M&E DATA TO MAKE NECESSARY AND RELEVANT ADJUSTMENTS TO PROGRAM DESIGN, IMPLEMENTATION, AND MANAGEMENT? Evaluators asked USAID IPs about the role and impact of M&E data on RFS’s activities. Overall, leadership and technical staff from both the RFS-SSE and RFS-IDCRT activities reported using reading assessment findings—EGRA for RFS-SSE and the EGRSLA for RFS-IDCRT—to inform implementation. RFS-SSE implementers said they collected their own M&E data to review against midline EGRA results with the hope of identifying successful course corrections and developing building blocks. However, the activity’s fast pace and high intensity of the pilot made it difficult for RFS-SSE implementers to take the time to purposively implement change—as one RFS-SSE respondent noted, “we were just running a mile a minute.” The same respondent noted that the timeframe between implementation and evaluation points—a year and a half—was not sufficient. In the case of RFS-IDCRT, implementers shaped the intervention after performing a capacity assessment of teachers and visited schools extensively to observe students. Also, based on the findings from the EGRSLA that highlighted challenges to students’ reading comprehension skills, RFS￾IDCRT trained teachers on how to do vocabulary drills and how to incorporate different types of questions to elicit deeper cognitive reflection and understanding to engage students’ MSL knowledge for MSA reading. Furthermore, respondents noted that, from the onset of the proposal, the RFS￾IDCRT implementer set goals for the activity, each with evaluation components that could be referenced and evaluated against. While not specific to M&E data, USAID/Morocco and MoE officials affirmed the crucial role initial studies played in establishing the partnership between the Mission and the MoE, as previously discussed. These studies—as well as subsequent research and analysis conducted under RFS—helped identify and build an evidence-base that addressed the specific early grade reading needs and gaps in Morocco. However, delays in the validation and sharing of EGRA findings, as well as results from RFS-NPR’s April 2018 baseline data collection, were noted by central MoE officials as limiting their 53 Ibid. P.55 54 Ibid. P.57 Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 46 ability to identify potential issues.55 Moreover, multiple stakeholders—including MoE officials at both the local and central levels—stated they personally had observed vast improvements in students’ reading skills due to the new reading approach but that these gains are not evident in the EGRA results. A central MoE official even expressed concern that there may be an issue with the EGRA itself. While no evidence indicating a quality concern was identified with the RFS-SSE EGRAs, the short timeline between EGRA evaluation points may not have provided sufficient time to capture changes in learning outcomes. In addition, MoE officials at all levels observed they did not receive EGRA reports in accessible languages and were unable to use these results to inform their own implementation. 3. INCLUSION: WHICH PRIMARY-SCHOOL-AGED POPULATIONS OF CHILDREN ARE NOT BENEFITING FROM THE CURRENT RFS PROJECT ACTIVITIES, AND HOW CAN THEY BE BETTER SERVED IN FUTURE PROJECTS/ACTIVITIES? (CONSIDERING GENDER, STUDENTS WHO ARE DISADVANTAGED/MARGINAL, LESS SUCCESSFUL, OR SPECIAL NEEDS, THOSE WHO DO NOT SPEAK ARABIC AT HOME, STUDENTS IN RURAL OR VULNERABLE AREAS) In order to better understand the role and level of inclusion in USAID’s RFS activities, evaluators asked key stakeholders—including USAID/Morocco officials, USAID implementers, MoE officials, school directors, teachers, and parents—about which populations of primary-school-aged children may not be benefiting from RFS, the barriers for reaching those children, and recommendations for better serving those children in the future. Findings are summarized below by sub-question. 3.1 WHICH PRIMARY-SCHOOL-AGED POPULATIONS OF CHILDREN (CONSIDERING GENDER, STUDENTS WHO ARE DISADVANTAGED/MARGINAL, LESS SUCCESSFUL, OR SPECIAL NEEDS, THOSE WHO DO NOT SPEAK ARABIC AT HOME, STUDENTS IN RURAL OR VULNERABLE AREAS) IN SCHOOLS TARGETED BY RFS ARE NOT BENEFITING FROM THE CURRENT PROJECT ACTIVITIES? When asked which children might not benefit from RFS, regional and central MoE respondents initially noted that 99 percent of children are in school. One respondent summarized a commonly expressed view by stating that “Morocco does not have a problem with access.” The high level of primary school access was reiterated by USAID/Morocco officials, multiple IPs, and found in the literature—the 2017 Global Gender Gap report notes that the net enrollment rate for primary schools in Morocco is 99 percent (98.3 percent of girls and 98.6 of boys). Gender gaps related to enrollment in secondary education, however, persist—as do gaps in overall literacy rates.56 Similarly, when asked about gender or differences between girls’ and boys’ engagement in RFS, findings from the literature review echo respondents’ indication of gender parity in education enrollment. For example, the Global Gender Gap Report from 2017 highlights Morocco’s success in reducing the gender gap in educational attainment overall—moving from a 2006 score of 0.848 to a 2017 score of 0.920 (with 1.0 indicating parity between boys and girls). Most pertinently for RFS, Morocco has reached gender parity (0.997) for enrollment in primary (98.3 F/98.6 M).57 Respondents tended to report that gaps between girls and boys were minimal; many recognized the strong gains shown by Moroccan girls in terms of school enrollment and completion. Some respondents even believed that girls were outperforming boys, but several school directors noted girls tended to drop out after primary school due to deficiencies in the school environment, specifically relating to sanitation and school safety. In addition, RFS-SSE reports indicate gender differences in EGRA performance. In Cohort 1, girls’ gain scores from midline 2 to endline were significantly higher than boys in both the experimental and control groups on three subtasks—syllable identification, 55 As of November 2019, the NPR baseline EGRA report has not been approved for release, more than a year after submission. 56 World Economic Forum, (2017) Global Gender Gap Report, 244. 57 Ibid., 244. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 47 nonword reading, and passage reading. In Cohort 2, girls’ gain scores between the two time points were significantly higher than boys in both groups on four subtasks—syllable identification, nonword reading, passage reading, and reading comprehension. When evaluators pushed respondents for details about specific student populations that might not be benefitting from RFS, a more nuanced picture regarding the uneven quality of educational experiences and stigma that students face emerged. Students with disabilities, students who spoke languages other than Arabic, as well as students from poor families, divorced families, or nomadic families, were all identified as populations whose specific needs may not be addressed by RFS. Students with disabilities. The limited or lack of inclusion of people with disabilities was a key cross-cutting issue identified across all four RFS activities. Although the statement of work for RFS￾NPR states that the majority of funds should be focused on the overall goal of improving early grade reading instruction, it also notes consideration to building the MoE’s capacity in areas that support this focus, including the instruction of children with disabilities. However, in the interviews, most respondents did not see any specific outreach to students with disabilities, except for those specifically working with RFS-IDCRT. One notable exception identified by two RFS implementers was transportation provided during one of the summer enrichment programs to encourage all students to benefit from the activity. As one implementer described it, “In one of our summer programs there were a couple of kids who were in a wheelchair, and the community-based organization was very proud to offer summer school to those children. The summer programs were better able to handle that and be more inclusive because they weren’t rigid in terms of the classroom and what was happening in the classroom.” Although RFS-IDCRT specifically worked with students with disabilities, even there some respondents felt there were limits to the improvements that the activity could offer. When asked if there were aspects of the activity that some students could not participate in, one teacher responded that they do not know how to help students who are both deaf and have an associated disability. They described children with multiple disabilities—such as autistic children who are deaf or children with low intelligence quotient who are deaf—as challenging to teach and stated that they did not think the USAID program can be helpful in supporting these students. In addition, students receiving support from RFS-IDCRT showed only modest results following the activity, with the majority of grade 1 and 2 students unable to answer a single reading passage comprehension or MSL comprehension question correctly; as previously discussed, most also did not sign a single correct word on the reading passage subtask. While these results might not capture all the learning gains students might have had, they highlight that students who were deaf and hard of hearing did not appear to benefit as greatly as anticipated in terms of improving their reading skills. Perhaps more importantly, although RFS-IDCRT reached 233 students at ten schools in grades 1 and 2, the actual number of students with disabilities who could benefit from RFS is unknown. As one MoE respondent stated, “no one knows how many children with disabilities are out there. Many parents keep them hidden at home out of shame.” This includes students with autism, dyslexia, or who are deaf, mute, or blind. For example, an AREF Director said that while overall enrollment is 99 percent, students with disabilities needed to be supported in order for them to integrate into the system and did not link RFS activities to addressing this population’s needs. A USAID/Morocco respondent agreed that students with disabilities are indeed vulnerable, “Most of these students are in school through grade 6, then they leave the formal system. Last June, the MoE launched a national strategy for inclusiveness, saying, “the school is a space for all.” It was an ambitious strategy, supported by donors, but did not provide clear steps on how it will implement.” This strategy follows three years after Parliament passed a law about inclusive “NO ONE KNOWS HOW MANY CHILDREN WITH DISABILITIES ARE OUT THERE. MANY PARENTS KEEP THEM HIDDEN AT HOME OUT OF SHAME.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 48 education, indicating a slower pace of change. 58 Students who speak languages other than Arabic were also identified as possibly excluded from the scope of the current RFS activities by parent and teacher FGD respondents. Results from the RFS-SSE activity support this finding and show students who speak different languages benefit differently from the intervention. The findings from the activity’s EGRAs indicate that students who speak Darija made greater gains than their peers who reported speaking Amazigh languages at home, perhaps due to the similarities between Darija spoken at home and MSA taught in schools. According to Moroccan census data from 2014, 90.9 percent of the population reported using Darija, while 26.0 percent reported using an Amazigh language. Usage of an Amazigh language was higher in rural areas than urban areas—34.8 percent and 20.4 percent, respectively—while usage of Darija was lower in rural areas than urban areas—82.7 percent and 96.3 percent, respectively.59 RFS-NPR was designed under the consideration that Arabic is not the mother tongue of any teachers or students, according to activity implementers. Therefore, learning occurs slowly, starting with an emphasis on phonemic awareness and alphabetic principle and then shifting to listening and speaking. Instruction in the activity is uniform, with no variations for students who speak an Amazigh language and those who speak Darija. Students from poor families, divorced families, or nomadic families may also require special supports and were not explicitly addressed by RFS activities. RFS supports for poor families were not recognized by respondents, although the Tayssir program was cited by several MoE and school-level respondents as important support by providing funds directly to needy families that enrolled students.60 In FGDs with parents, respondents identified students with family problems as needing extra support. Respondents noted that schools were not equipped to help those students and that students from these families face stigma and might stay away from the classroom. In one district, both teachers and parents cited nomadic families as a population unserved by RFS as they transferred schools frequently or kept children, especially girls, out of school altogether. RFS-SSE endline EGRA results illustrate the disparity in performance between students in urban and rural areas. In both cohorts at endline, students attending urban schools outperformed students attending rural schools on all six subtasks. 3.2 WHAT ARE THE BARRIERS TO REACHING THOSE STUDENTS WHO DID NOT BENEFIT FROM RFS AND MEASURES BY WHICH THESE STUDENTS CAN BE BETTER SERVED IN FUTURE PROJECTS OR ACTIVITIES? Differing views on access and attendance for all children at MoE schools. Overall, the majority of respondents felt that access to the school system was not a barrier for groups that might be marginalized, including those with disabilities. Yet, as one of the USAID/Morocco stakeholders acknowledged, while access is very high, it is not uniformly applied: “I would say access to primary education is like 99 percent, and then you’ve got that one percent who are all of these children with special needs. So, you’ve got that last mile to 58 See for example Chapter III of the Loi-cadre n° 97-13 adopted April 27, 2016 and related to the protection and promotion of individuals with a handicap. 59 Responses to the 2014 census question about language usage were non-exclusive. 60 Tayssir is a program managed by l’Association Marocaine d’appui à la scolarisation (AMAS), which provides cash transfers to poor families in rural areas to promote school enrollment and limit dropouts. “PROMISE ME A PLACE IN SCHOOL, AND THEN TALK TO ME OF QUALITY… IF YOU TRY TO ENROLL A CHILD WITH DISABILITIES, THE SCHOOL DIRECTOR WILL TELL YOU THERE IS NO PLACE.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 49 go, and that’s going to be the hardest mile. It’s going to be really expensive because you’ve got to figure out how you are going to get all these children into school. So, you’re going to have children who can go into a normal classroom and a lot of children who can’t.” Many students with disabilities remain outside the formal MoE education system, which is a significant barrier to attaining quality education. As one implementer highlighted, “Promise me a place in school and then talk to me of quality…if you try to enroll a child with disabilities, the school director will tell you there is no place.” The number of students with disabilities in Morocco is unclear, although a 2004 estimate places the number of school-age disabled children at 230,647.61 According to a needs assessment USAID conducted for students who are blind or have low vision and students who are deaf or hard of hearing, there are considerable inconsistencies regarding disability prevalence rates in Morocco, depending on the data source, method, and ministry that collected the data, with rates ranging from 2.3–15 percent of the Moroccan population.62 This lack of data may extend to the MoE as well. One RFS-IDCRT respondent claimed the MoE did not count the number of students with disabilities until the implementation of RFS-IDCRT spurred them to change the practice. The activity also compelled the MoE Curriculum Directorate to open a unit dedicated to inclusive education, according to an RFS-IDCRT respondent and a USAID/Morocco respondent. While RFS-IDCRT activities had a specific focus on supporting students who are deaf or hard of hearing, there remain many issues beyond the activity’s scope that impact how well RFS activities can support these students. A previous RFS evaluation report cited systematic barriers facing students who are deaf or hard of hearing and their educators. Those barriers include the lack of public primary education for this student population. The majority of students who are deaf or hard of hearing attend privately-run education centers, many of which charge tuition and hire teachers with insufficient training in special education who are not fluent in MSL. In addition, negative attitudes about people who are deaf or hard of hearing remain persistent in Morocco. Parental support and engagement need to be strengthened. Most of the teachers in FGDs expressed great frustration with the perceived lack of support from parents. One teacher stated, “Families don’t support the school and don’t help the student; they just accompany the child to the outside of the school.” Some teachers provided examples of creative ways they tried to reach the parents of their students—such as through messaging services or in-person meetings—with very little success. The types of support respondents expressed needing from parents were to read with their children at home and make sure their children study at home. A head teacher attributed the lack of engagement from parents as a result of low literacy rates among parents. As discussed in previous sections, many respondents felt that parents do not know how to read themselves so they cannot help their children. Weak parental support, capacity, and engagement most negatively affect the students who need support the most. As one teacher observed, “troubled” students are most affected by a lack of encouragement from their families to study. Two different MoE officials remarked that parental support is more likely in urban areas than in rural areas, given the higher levels of parental education in urban areas. Respondents felt that urban parents seemed more likely to be able to read with their children and to be involved in parent associations. In general, across respondents, parent associations were reported to be weak and not helping the system. One MoE official highlighted parents as a key area for future improvement to support reading, noting there is work still to be done with parent associations and with parents themselves. As for parents of children who are deaf 61 RTI, 2016, Situation and Needs Assessment for Students who are Blind/Low Vision or Deaf/Hard of Hearing in Morocco. USAID EdData II Task Order 15: Data for Education Programming in Asia and the Middle East (DEP/AME), 26. 62 RTI, 2016, Situation and Needs Assessment for Students who are Blind/Low Vision or Deaf/Hard of Hearing in Morocco. USAID EdData II Task Order 15: Data for Education Programming in Asia and the Middle East (DEP/AME), 26. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 50 or hard of hearing, the RFS-IDCRT evaluation report found that not enough engagement with parents was conducted during the activity in order to improve their knowledge or change their attitudes about their children’s abilities. Inequitable access to pre-primary school poses a challenge to acquiring reading skills. A variety of stakeholders across all respondent types cited the lack of early educational experiences as a barrier to reading. One IP talked at length about the challenges associated with students beginning formal education in public school at age six. The partner highlighted how such a class might have 30 to 40 students with different backgrounds, including some children with Qur’anic school experience who know the alphabet; others from private schools who know some formal Arabic and some French; and still some others who have never been in any school. With large class sizes, teachers fail to differentiate instruction and focus their attention only on those who already know the alphabet. Students without a background level of knowledge or with additional needs rapidly fall behind and eventually drop out. In the absence of interventions or a systematic solution, this respondent described the low starting points of some children as a “parents’ problem.” When asked about supports that could help students not being currently served, one IP described the need for a nursery system to help children learn the language of instruction at an early point in their lives. They felt it would also benefit students with disabilities. Limited pre-primary enrollment can most severely affect students with disabilities. A teacher at an association-run school for students who are deaf or hard of hearing described a perspective that parents are not involved in their children’s learning and generally only discover their child is deaf at a late age, such as five years old. When such a child arrives at school for grade 1, he or she would have very little base knowledge of sign language or reading and would begin well behind hearing students. Earlier identification of students with special needs and earlier intervention with specialized supports would help limit such gaps between students. As a director of an association-run school for students who are deaf or hard of hearing noted, this student population needs to start reading and experiencing other learning activities very early, even at age two, using images and other techniques to replace the auditory experience that hearing children have when reading aloud or reading with others. Pre-primary school challenges were frequently cited by respondents as a disproportionally rural issue. One MoE respondent from a rural area complained that the RFS interventions began with grade 1 and not with pre-primary school, as he noted a large number of children in his region would join school without any reading-related knowledge, in contrast to cities. A school director described pre-primary school as the “base” of reading knowledge and cited the lack of trained pre-primary schoolteachers as the primary factor contributing to problems learning to read. Several respondents from the MoE and IPs noted the MoE is beginning to discuss greater pre-primary supports and interventions, a sign that this area for improvement may already be identified. In fact, in June 2019, the World Bank unveiled a $500 million program to help provide quality pre-primary education to every child in Morocco as the MoE aims to make pre-primary education universal by 2027; in 2017, only half of children aged four and five attended pre-primary school classes.63 Insufficient training and resources for teachers. Implementers and central MoE respondents think teachers need more training on inclusion. In contrast, regional MoE respondents and teachers expressed differing views. They believed only specialists and separate schools should work with students with disabilities, implying working with this population of students was beyond their job responsibilities. Some noted that each type of disability should have its own class, in contrast to the inclusion model. Finally, teachers said that they lack materials to address special populations and even noted the RFS materials were too dense to work through and, therefore, did not address the 63 https://www.worldbank.org/en/news/press-release/2019/06/20/morocco-to-expand-access-to-early-education-and-improve-quality-of￾learning Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 51 needs of all students. As for teachers of students who are deaf or hard of hearing, the RFS-IDCRT activity evaluation report noted that Deaf education teachers were not sufficiently trained and would benefit from an improved training system. At the time of the evaluation, RFS-NPR is attempting to address this gap by providing training on differentiation. The MoE needs to address these issues better. Many respondents believe the MoE could do more to support students with special needs. For example, one USAID/Morocco representative said, “Last June, the MoE launched a national strategy for inclusiveness, saying ‘the school is a space for all.’ It was an ambitious strategy, supported by donors, but did not provide clear steps on how it will be implemented. Most of the time, it is a theory.” This was supported by one respondent knowledgeable of the associations supporting special schools for students with disabilities, who said there was a bad relationship between the MoE and the associations—that “what happens on the ground is not what is written in administrative papers.” This was also supported by a different official who cited the lack of funding for the new approach to students with disabilities as a real barrier to ensuring that any MoE approach goes beyond just words. Even MoE respondents seemed to obliquely indicate that more action is needed from the MoE. When asked about supports for students with disabilities, MoE respondents frequently cited papers, policies, or committees intended to improve education for students with disabilities. Notable by omission were references to tangible MoE action to support students with disabilities. By contrast, parents and teachers did not talk about the MoE and provided no critique for their role in supporting these populations of students. 4. ROLE OF MOE: WHAT WAS THE MINISTRY’S ROLE IN LEADING PROJECT ACTIVITIES, AND HOW DID THIS ROLE INFLUENCE THE IMPLEMENTATION, MANAGEMENT, AND RESULTS OF THE ACTIVITIES? To address this question, evaluators investigated the design of RFS activities and the MoE’s involvement from central to local levels in its implementation. Evaluators asked relevant stakeholders—including USAID/Morocco officials, activity implementers, and MoE officials—about the MoE’s role in RFS and relationships between the MoE and other RFS partners as well as reviewed documents from each of the four RFS activities. These qualitative findings are organized by sub-question, while the responses to the last two questions were combined to paint a clearer picture of the MoE’s influence at various levels. 4.1 WHAT WAS THE MINISTRY'S ROLE IN LEADING RFS PROJECT ACTIVITIES? RFS-SSE and RFS-NPR Both RFS-SSE and, even more so, RFS-NPR are anchored by and directly responsive and tailored to the MoE’s needs and vision. As described earlier in this report, this approach is exemplified by respondents’ remarks that “the MoE is in the driver’s seat” and “the MoE is the locomotive.” USAID/Morocco and IPs highlighted this model as a welcome departure from previous education projects in Morocco. One USAID official noted, “Previously, the activities were add-ons to what was already there and would not budge, but now we’re responding directly to a need identified by the MoE to implement reform. This means the project can only do what the MoE wants and needs.” Indeed, RFS-SSE was initially designed only to provide supplemental reading materials for grades 1 “LAST JUNE, MOE LAUNCHED A NATIONAL STRATEGY FOR INCLUSIVENESS…MOST OF THE TIME, IT IS A THEORY.” Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 52 and 2 to complement the existing curriculum.64 After requests from USAID and the MoE, however, RFS-SSE broadened its scope by designing and developing new primary reading lessons for grades 1 and 2 as part of the comprehensive overhaul of the Arabic reading curricula. Both RFS-SSE and RFS-NPR sought to encourage the MoE’s involvement at every stage of the activity. From RFS’s inception, USAID/Morocco and the MoE worked together to assess the state of the Moroccan education system, identify key shortcomings with early grade reading, and build broad consensus around the impact that early grade reading skills have over the course of students’ lifetimes. Then the RFS-SSE and RFS-NPR teams used a facilitative approach to implement their respective activities, working hand-in-hand with the MoE and providing sustained operational support to the MoE. To foster the sense of ownership with MoE officials, IPs provided MoE staff with the technical support and capacity-building it needed to maximize its role and provide leadership in RFS. As one RFS-SSE team member noted, they had people from the Ministry who had been seconded to the project and were learning from RFS-SSE’s international experts. This approach made it easier to complete certain tasks requiring the MoE’s sustained input, including the revision of the Arabic reading curriculum and associated materials, the training of teachers and inspectors, and piloting and scale-up of the new curriculum. By being placed in the driver’s seat, the MoE enabled and took ownership of RFS-SSE and RFS-NPR in ways that would likely not have been otherwise possible. The accelerated national scale-up of the new grade 1 curriculum for the 2017-18 school year illustrates the MoE’s leading role and ownership of RFS-SSE and RFS-NPR. The scale-up occurred two years earlier than initially planned, essentially short-circuiting RFS-NPR’s intended phased approach. The accelerated national scale-up was done with some urgency and against the advice of the RFS-NPR activity team, which encouraged further piloting and cautioned that there were insufficient time and resources to finalize materials and train teachers as part of the accelerated scale-up.65 However, the MoE went ahead with the scale-up, which further underscored the nature of the MoE’s involvement and leadership. The recent expansion of RFS-NPR’s scope to include grades 5 and 6 is another example of the MoE’s leading role in RFS-NPR. The inclusion of grades 5 and 6 was never anticipated, and the RFS-NPR IP indicated this expansion would require budgetary and scheduling tradeoffs, including the potential reduction or elimination of activities centered on community support for reading. RFS-HICD The MoE played an important leadership role in the inception of RFS-HICD, as well as its design and early implementation. Unfortunately, the MoE’s involvement ultimately fell short of what was initially anticipated by the activity’s two funders—USAID and the Millennium Challenge Corporation (MCC)—namely refining the findings and recommendations, formally validating them, and then beginning to enact key recommendations. The reasons behind the unfinished nature of RFS-HICD are complex, however, including the MoE’s reception of the assessment and changes in MoE leadership while the activity was ongoing. A former national Minister of National Education and Vocational Training explicitly requested the RFS-HICD activity. He was a proponent of it largely because he wanted the MoE to be well￾positioned to implement and achieve the goals of the Vision 2030 roadmap, which he played a key role in introducing. In addition to the former Minister, the central level of the MoE also played a leadership role in several important aspects of the design phase discussed later in the report, and not 64 Chemonics International Inc., RFS-SSE 2015-2018 Final Report, (2018), p.7. 65 Creative Associates International, Reading for Success – National Program for Reading Quarterly Report 4, Year 1, (2017), p. 12. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 53 only enabled but facilitated the implementation of the RFS-HICD assessment.66 However, the commitment of the MoE’s leadership to the RFS-HICD activity later diminished after the assessment was completed, and the findings were made available, leading to the findings and recommendations not being approved and formally implemented. Respondents cited several reasons for this outcome. USAID officials stated that the MoE viewed the assessment’s results as overly critical of the Ministry and requested that they are conveyed more diplomatically, which respondents said was still being completed at the time of fieldwork. In addition, USAID officials noted that changes at the top level of the MoE, including the departure of the Minister who requested the activity, were also a factor. An AREF official involved in the assessment corroborated this view, noting that most of the officials who played an important role in leading and facilitating the activity had moved to different positions both within and outside of the MoE. Finally, one high-ranking MoE official identified two other factors. First, the study was “conducted in silos” and was, therefore, unable to comprehend the MoE as a system. In addition, the study did not consider certain changes at the MoE and their implications, including its expanded purview that now includes higher education and scientific research as well as an organizational structure that continues to evolve. RFS-IDCRT The MoE does not appear to have played much of a leading role in the RFS-IDCRT activity, namely because it does not oversee the non-governmental Deaf associations, which were RFS-IDCRT’s primary beneficiaries. Nonetheless, the RFS-IDCRT team sought to raise the MoE’s participation and buy-in at several stages of the activity, largely because many children who are deaf or hard of hearing attend public school and a number of MoE teachers received training under RFS-IDCRT. For example, an MoE official was invited to all Steering Committee meetings and generally attended, and the RFS-IDCRT leaders noted having “a number of meetings about the project and the direction of the project” with top officials at the MoE. There are several possible reasons for this limited leadership, but the evaluators were unable to pinpoint the most salient ones. In contrast with the other three RFS activities, the MoE did not lead the inception of the IDCRT activity, which occurred under the All Children Reading: A Grand Challenge for Development grant program jointly backed by USAID, World Vision and the Australian Government. USAID/Morocco recognized an opportunity and alignment with its objectives and made additional financial commitments that enabled an expanded scope of work. Although RFS￾IDCRT documents indicate that the MoE’s director of curriculum, “volunteered his agency as the lead in providing guidance to and coordination with the project in order to improve its institutional standing and ensure the longevity of the project’s benefits,”67 this described role does not match the level of engagement, buy-in, and leadership by the MoE seen in the other three RFS activities. Multiple factors hindered the MoE’s ability to provide leadership for this activity, including its level of involvement in deaf education relative to other Government of Morocco stakeholders as well as apparent political issues. The RFS-IDCRT team initially intended to work with both the MoE and the Ministry of Solidarity, Equality, Family & Social Development “because, in truth, at the time the Ministry of Solidarity was doing more with deaf education, but then we were directed by USAID to work exclusively with MoE.” Indeed, although 66 As the evaluators heard repeatedly from RFS IPs and NGOs (and saw firsthand during fieldwork), the central MoE’s involvement is necessary to gain access to its staff at all levels for an activity like this one. 67 Institute for Disabilities Research and Training Inc., Final Report for USAID-608-A-15-00002 – Improving Deaf Children’s Reading Through Technology, (2018), p. 3. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 54 the MoE oversees the integrated classrooms68, it is not in charge of the non-governmental deaf associations that were the activity’s primary beneficiaries. In addition, the decentralized approach to education for students who are deaf or hard of hearing may have also been a factor. In addition, the RFS-IDCRT team noted it initially engaged and worked directly with the MoE, but it was later instructed by USAID/Morocco to only share documents and coordinate with the MoE through the Mission. According to respondents, this altered process caused delays and miscommunication that frustrated key MoE stakeholders. This communication breakdown may have impeded the MoE from playing a larger role in the activity. As evaluators observed during planning and fieldwork, USAID is quite careful in managing its relationship with the MoE and knows how the institution operates. 4.2 HOW DID THE MINISTRY'S ROLE INFLUENCE THE IMPLEMENTATION, MANAGEMENT, AND RESULTS OF THE ACTIVITIES? 4.3 HOW DID THE ROLE OF THE MINISTRY VARY IN IMPLEMENTATION, MANAGEMENT, AND RESULTS AT THE CENTRAL, PROVINCIAL, REGIONAL, AND LOCAL LEVELS? RFS-SSE and RFS-NPR Virtually all levels of the MoE were and are deeply involved in both the RFS-SSE and RFS-NPR activities. However, it is worth delving into the specifics of the MoE’s role and how different levels within the MoE influenced various stages of the project. Many directorates at the central level of the MoE played an important leadership and technical role. At the central level, respondents indicated the Directorate of Curricula was a leading force behind both RFS-SSE and RFS-NPR as the principal conduit for decision-making and coordination and a source of oversight. However, other Directorates and Centers within the central level of the MoE also played specific roles in both RFS-SSE and RFS-NPR. For example, the National Center for Evaluation and Examinations (CNEE) was a key stakeholder and participant in the evaluation of both activities, including EGRA development, the sampling approach for the RFS-NPR baseline, and the training of MoE staff who served as enumerators and coordinators. The Directorate of Communication was also involved in plans to raise parents’ awareness about the importance of early grade reading through different means, particularly television and social media; this effort may be scrapped, however, due to the expansion of RFS-NPR’s scope to include grades 5 and 6. The National Center for Pedagogical Innovation and Experimentation (Centre National des innovations pédagogiques et de l'expérimentation) participated in the design of integrating new elements into teacher training and in the ongoing development of the massive open online courses (MOOCs) that will reinforce in-person training, and the CRMEF played a central role in the training of inspectors and the planning and development of pre-service teacher training under RFS-NPR. The Directorate of Statistics and Planning helped to select RFS-SSE experimental and control schools, develop evaluation indicators, adjust data systems such as MASSAR to accommodate the activities, and design processes to obtain data on what teachers are doing at the local level. The MoE facilitated the implementation in various ways that would be out of reach for activity teams. The MoE’s central level serves as the gatekeeper for all public schools, and, without its formal approval, MoE staff at regional, provincial and local levels would simply not have 68 Integrated classrooms are classrooms in certain Moroccan public schools where children with mild disabilities are educated for up to three years. After the three years, the children are tested and, depending on the results, are either enrolled in segregated schools for those with disabilities (when available and within reach, otherwise they likely drop out) or integrated into general education classes. For more information, please see https://ierc￾publicfiles.s3.amazonaws.com/public/resources/Morocco%20Inclusion%20Study%20Report%20ENGLISH.pdf Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 55 participated in RFS. This is even more salient when activities have political implications and directly affect students in the way that both these activities did. Indeed, several interviewees noted that the curricula are widely viewed as political—and not just technical—hence, MoE support was critical for their acceptance. There are ongoing political debates about the role of Arabic in schools and about the suitability of the curricula across all regions of the country.69 Curricula revisions are also the domain of an influential Directorate within the MoE, and while certain decisions related to changing the curricula are beyond the MoE’s own power and responsibilities, it is able to convene and persuade key external stakeholders to at the very least examine proposed changes. A good example of this is the page count of the MoE-approved school textbooks, which are the domain of the National Education Committee in Parliament. Realizing that the currently approved page count is too limited to accommodate the revised curriculum at various grade levels, the RFS-NPR team worked to persuade the MoE that this is a binding constraint on the activity, requiring significant extra work and time to fit the content of the revised curriculum within the existing parameters. The MoE agreed, and textbook criteria and parameters are now under review in Parliament as a result. The MoE at the central level was also critical to the implementation of the summer reading program under RFS-SSE and RFS-NPR. The selection of CSOs that implemented the program was done with input from the MoE and using a list of CSOs that had previously worked with the MoE. Decision-making related to the implementation and management of RFS-SSE and RFS￾NPR was highly centralized, but key aspects of both activities’ implementation required the buy-in and participation of the AREFs and provincial branches. The MoE especially needed AREFs and provincial branches to buy into monitoring and training activities, including training of all inspectors and then all teachers under RFS-NPR. CRMEFs were also responsible for NPR’s pre-service teacher training and for assessing the new curricula, testing new teaching approaches, and documenting their findings so that they could provide useful guidance for pre￾service and in-service teachers. At the local level, school directors and teachers were also an important source of feedback that both RFS-SSE and RFS-NPR leveraged to improve their trainings and material. Certain regions and provinces took on their own initiatives to enhance participation in RFS-SSE and RFS-NPR. One AREF reported that it decided to involve school directors in some RFS￾NPR workshops and trainings on an optional basis, given their role in coaching and supervising teachers. A CRMEF reported organizing NPR-related compulsory distance training for pre-service teachers to complement their regular training and weekly field visits. MoE staff at the regional, provincial, and local levels also implemented certain reading support activities recommended by RFS-NPR, particularly reading clubs and reading competitions in schools. Regional and provincial officials were also responsible for distributing the revised school textbooks and teacher manuals to individual schools. Furthermore, these officials also completed a significant portion of the technical work. Provincial inspectors and Arabic language trainers working for the CRMEFs served as members of the teams who most closely worked with RFS-SSE and RFS-NPR IPs to develop the new curriculum and materials. Despite their involvement, regional, provincial, and local MoE staff widely reported they felt out of the loop about RFS-SSE’s results and RFS-NPR’s implementation and decision-making. They indicated that except for occasional meetings to discuss milestones, activity progress reports and evaluation reports were never shared with them. In addition, regional MoE staff stated they wished to have a greater role in developing the curricula content and associated materials, so they better reflected their regions’ characteristics. In fact, this 69 In Morocco, various aspects of public education and by extension, the content of the curriculum, are subject to much debate. Among them, two in particular are relevant to RFS. One is the level of say that regions have over the content of the curriculum taught. For example, several respondents, particularly within the AREFs and the provincial branches, expressed their disappointment over the sometimes limited relevance of the curriculum to the daily life and experience of students in their geographic areas. Some school directors and teachers also noted that the realities in terms of support materials are quite different in urban or well-off areas than they are in more rural or poorer areas. Another is primarily tied to the role and importance of the different languages (particularly Arabic and French) in the curriculum, including when they are first taught, how much teaching time is allocated to each, and which is used to teach other subjects such as sciences, social sciences, etc. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 56 regional input is part of announced MoE reforms on decentralization, but it has not yet been enacted. For all the advantages associated with the MoE’s leadership and extensive involvement in RFS-SSE and RFS-NPR, several drawbacks have primarily affected their implementation and management. IPs, as well as USAID/Morocco, had to be flexible with time and resources in order to act on the MoE’s requests. Both RFS-SSE and RFS-NPR activities experienced recurring delays and difficulties with scheduling workshops and trainings and getting timely feedback and approval of key materials. These delays were often tied to the MoE’s unavailability during summer vacation and the hectic first month of each school year, but they also occurred during other periods. In isolation, RFS-SSE and RFS-NPR staff could manage these constraints. However, after the MoE requested the accelerated scale-up, activity implementers had greater difficulty meeting the MoE’s tight timelines due to these delays. Even before becoming aware of these issues, the RFS-NPR team tried to organize a governance, decision-making, and review structure to formalize the MoE’s key management and technical role. The RFS-NPR team indicated RFS-SSE implemented a similar approach, but it proved more difficult under RFS-NPR because efforts to create Steering Committees and working groups did not succeed due to what RFS-NPR implementers reported was “reticence”70 from the MoE. The RFS-NPR team reported that it adjusted its approach to rely principally on the Director of Curricula for decision￾making, oversight, and coordination while creating a three-layer structure to develop most of the technical material. This structure started with a group of three MoE staff—two inspectors and one Arabic language trainer—who are deeply involved in the curricula design. This team was purposefully small based on lessons learned under RFS-SSE, which had a larger 19-person team that was not easy to manage. The second layer comprised a larger team of around ten individuals that validates the ideas of the core team and brings diverse local expertise and expectations. Finally, the third layer is made up of the wider community of school leaders—inspectors, teachers, school directors, and provincial branches—who implement the scale-up and provide crucial feedback. Under RFS-SSE, the central MoE’s key coordination role led to some difficulties in reliably organizing workshops and trainings. For example, according to the RFS-SSE team, participants who had indicated they were available would not be able to join unless central MoE officials sent an official notice about trainings and workshops in a reliable and timely fashion to lower levels.71 Project documents indicate RFS-SSE staff made follow-up phone calls and developed personal relationships at the regional and provincial levels in order to confirm arrangements had been made. The MoE’s role and leadership also had consequences for the training of teacher trainers and teachers. Implementers, as well as a few teachers and school directors, indicated the MoE required that all trainings be conducted during the school year and approved only up to three to four days of training at a time. RFS-SSE judged this amount of time was insufficient, according to project documents, and had to develop alternative solutions, including continuous support. One USAID official suggested that, ideally, 15 days should have been the minimum amount of time devoted to training, instead of the approximately five days (sometimes in a format of half days over several weeks) offered. RFS-NPR encountered similar issues, except the training cascade now had one extra step—the training of all inspectors by those trained by the master trainers—which created further risks that teachers would not be sufficiently trained. The RFS-NPR team also mentioned several more days of training would be desirable. Some teachers and inspectors echoed 70 Creative Associates International, Reading for Success – National Program for Reading Quarterly Report 3 - FY 2018 (Apr. 1–June 30, 2018), p. 7. 71 Chemonics International Inc., RFS-SSE Annual Report, (Sept. 23, 2015–Sept. 30, 2016), p.15-16. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 57 this point, while also expressing concerns that the training was too theoretical and not practical enough. RFS-HICD As noted in the previous sub-evaluation question, top MoE officials played a leading role in RFS￾HICD’s inception, and that role continued during activity design and implementation and in how the results were used. During the design stage, the MoE’s top officials contributed meaningfully. The MoE’s contribution included providing guidance on the development of the RFS-HICD activity, information about the MoE and its staff, and approval to collect data. For example, project documents indicate that the MoE suggested the regions and provinces the assessment should cover and provided key feedback on the perceived value of certain elements of the work under consideration by USAID/Morocco and MCC. The MoE also shared documents necessary to understand the operations and structure of the organization and its departments and directorates, thereby facilitating discussions and the design of the activity. During the implementation stage, the central level of the MoE continued to support and facilitate the activity. The MoE helped to select relevant staff at various levels of the Ministry and enabled the implementation partner to navigate through existing MoE structures and collect the qualitative and quantitative data necessary to complete the assessment. These data, which shaped the activity’s findings and recommendations, came from a mix of central, regional, and provincial MoE staff and officials. Unfortunately, the exact number interviewed or surveyed at each level was not available to the evaluators. Despite the extensive involvement of the MoE, the RFS-HICD team also faced delays associated with the start of the school year and difficulties associated with the ongoing reorganization of AREFs, which meant that many AREF staff were working in an “acting” capacity. Awareness about the RFS-HICD activity was quite limited outside of top MoE officials. With the exception of top officials at the MoE’s central level, the evaluators found nearly no awareness of this activity by name or based on a short description of its purpose. Although 192 MoE officials at various levels were interviewed as part of the RFS-HICD assessment, and 400 answered a web-based survey, the respondents’ lack of knowledge about RFS-HICD did not surprise evaluators given that the MoE has not formally approved the report, and only top MoE officials were directly involved. One individual who had been involved in the activity while working at the central level at the time was the notable exception, and even he did not know any details on its progress or its outcome. The RFS-HICD assessment’s findings and recommendations did influence donors and, possibly, the MoE. The MoE’s influence on the RFS-HICD activity’s results is undeniable, even if the evaluators did not fully establish the exact factors, as discussed in the evaluation sub-question 4.1. Despite the MoE’s lack of formal approval and action, two USAID/Morocco respondents stated that the RFS-HICD results yielded some immediate benefits. They noted the findings and recommendations enabled the MoE to take stock of where things stood, at least internally and informally. USAID/Morocco respondents also noted that they did consider the RFS-HICD assessment results as part of program design and planning, and they believe MCC did the same. In addition, USAID/Morocco reported that the release of the results prompted the MCC and USAID to convene a meeting with the MoE and donors to improve coordination, reduce duplication, and see which organizations may be interested in addressing specific findings. USAID mentioned UNICEF as another donor who used the RFS-HICD activity’s results. Indeed, the evaluators discovered evidence that at least a few of the RFS-HICD recommendations had been considered. These considerations include ongoing discussions to revise the MoE’s textbook policy, adjustments to decentralization, and increasing the use and capabilities of the MASSAR system. In addition, a central MoE respondent said that one of the axes of the European Union’s Educ II project is largely aligned with certain RFS-HICD recommendations. Another central MoE official seemed to corroborate this Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 58 fact by reporting that RFS-HICD results had been used as a basis for designing human-resource training within their Directorate with the support of a European Union project. RFS-IDCRT Once RFS-IDCRT implementation began, the MoE coordinated with partners, but its key functions, primarily at the central level, appear to have been mostly logistical in nature with limited technical guidance. The MoE’s lower levels appear to have been involved on a limited basis. Participating MoE￾managed schools, which worked with the NGOs involved in RFS-IDCRT, were involved of course. The MoE’s central level regularly communicated with the AREFs of the relevant regions about the activity and coordinated with regional and provincial officials so that activities associated with these schools were authorized. The MoE primarily influenced the management and logistics of the activity but had limited influence over the technical aspects. Project documents and interviews with MoE and RFS-IDCRT stakeholders indicate that the MoE helped to select the CSOs for the activity. MoE officials also chose the teachers working in integrated classrooms and enabled them to participate in trainings. The MoE’s technical role and involvement were more limited. For example, MoE staff participated in the evaluation phase, such as the development of the adapted EGRA. The MoE also reviewed training-related items such as agendas and topics, and an MoE official was noted as also being present often during the training sessions. However, this technical role may have been insufficient as a project document indicates “it was evident that the selection of training topics was not developed in consultation with MoE nor was it based on a formal needs assessment of the teachers’ skills.” The MoE did not appear satisfied with the coordination between itself and the activity, and the request for a coordinator to liaise with the MoE was not approved. While there was ongoing coordination with the MoE, the RFS-IDCRT Steering Committee (comprised of members of Deaf associations in Morocco) did not formally include the MoE, although MoE staff did attend most meetings. The activity IP’s request for the addition of a dedicated coordinator to liaise with the MoE was never implemented. Interviews with central MoE staff did not provide much insight regarding their perception of coordination and involvement. However, a project document notes that an MoE representative “did not appear to be satisfied with the coordination between themselves, the project, and USAID.” Specific grievances included the perception that the MoE and Ministry of Solidarity, Women, Family, and Social Development were not adequately consulted on the selection of schools, technical activities, and training locations, as well as insufficient coordination with, and participation of, the MoE regarding training of MoE integrated classroom teachers. Stakeholders had split opinions about whether the MoE was active and responsive enough during the activity. Representatives from NGOs that had participated in the activity had differing opinions about whether the MoE was active and responsive enough. One respondent believed the MoE’s level of involvement was generally “just right,” although it fluctuated at times between insufficient and excessive. Another was more upbeat, saying the MoE was there when the activity needed its involvement. A third respondent felt the MoE’s involvement and its relationship with other stakeholders was occasional with too little follow-up. Project documents also suggest that some members of the Steering Committee believed the MoE was not responsive to letters with formal requests. There are indications that politics and mistrust between stakeholders may have hindered the management of the activity. Two respondents—one from the implementation partner and the other working for a participating NGO—reported politics might have interfered at times with implementation. One respondent noted that “some of the schools and CSOs were maybe becoming more emboldened to make requests of the MoE” due to the activity, and that did not sit well with its leadership. As discussed in evaluation question 4.1, this tension led to USAID requesting that the IP no longer interact directly with the MoE, and the relationship between the implementation partner and the MoE was strained, which was apparent through complaints and Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 59 frustration expressed by MoE staff. This respondent also noted, “Some simple things were made more complicated and less streamlined because there was too much pomp and circumstances put in places where it just became an impediment.” Another person believed that the MoE was concerned that CSOs were looking for money, whereas CSOs felt that the MoE teachers were not interested in teaching their students and instead were mostly keeping them occupied. Despite this tension, the respondent felt the RFS-IDCRT activity reinforced the ties between the MoE and the CSOs, and that there is now greater recognition of the CSOs by the MoE. Stakeholders did note that recent or upcoming national policy changes, including the national plan for inclusion and the upcoming loi cadre, should lead to improvements for children who are deaf or hard of hearing. Unfortunately, one respondent felt that not enough improvements had been made by the start of the 2019-20 school year and attributed this to the fact that the MoE is not yet ready to handle all children with disabilities. 5. SUSTAINABILITY: WHAT ARE THE KEY STRENGTHS AND CHALLENGES THAT HAVE AFFECTED THE RFS PROJECT ACTIVITIES IN CREATING A SUSTAINABLE NATIONAL READING SYSTEM? Sustainability is a central tenet of USAID. The concept is embedded in USAID’s Journey to Self￾Reliance framework: “Development investments in poor countries, of whatever form, should catalyze the economic, political and social processes within those countries that yield ever-improving lives for their citizens.”72 The central insight embedded in global best practices is that external aid investments are more likely to yield sustained development processes if they reinforce a country’s internally determined development priority, arrangements, and existing systems, and embed principles of inclusiveness.73 Based on the findings outlined in previous sections, RFS does clearly reinforce Morocco’s internally determined development priority—literacy and reading in the broader context of concerns for youth employment and civic participation—and was structured to work within and leverage existing systems and arrangements in Morocco—not only the MoE but also CSOs already working to support summer activities for children and support education for children who are deaf or hard of hearing. Inclusiveness was also a tenet and a mandate of all four RFS activities. Given this framing, this section looks at each individual RFS activity in turn and evaluates whether the processes and outcomes are expected to last and whether Morocco is positioned to maintain them in the future. RFS-SSE and RFS-NPR In terms of sustainability, RFS-NPR is a continuation of RFS-SSE. Therefore, both activities will largely be treated as one in this discussion, although the focus will be on RFS-NPR. The RFS-SSE activity ended in March 2018 and overlapped with RFS-NPR for nine months, during which the two teams worked together to facilitate the transition. This evaluation occurred approximately two years into the RFS-NPR activity, which is scheduled to end in May 2022. This discussion, therefore, will review the various early indicators available related to the sustainability of RFS-NPR through the end of the activity and beyond. RFS-SSE and RFS-NPR activities both benefit from the MoE’s strong ownership and leadership. Recent education programs and reforms driven by the MoE in Morocco, such as the plan d’urgence and pédagogie d’intégration, have faced serious challenges with respect to their widespread continuity and sustainability. However, RFS-SSE and RFS-NPR both share certain 72 USAID, Local Systems: A Framework For Supporting Sustained Development, (2014), p.3, available at https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf 73 Ibid. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 60 features that are expected to foster the sustainability of the revamped Moroccan national reading system. In particular and as described in the findings for evaluation question 4, the MoE’s level of leadership and ownership with these activities is high, and the activities’ IPs have been highly responsive to the MoE’s needs even when they do not fall within the established scope of work. Furthermore, both activities developed and reinforced the capacity of the MoE and its staff. These components, along with what the MoE learned more organically from its extensive involvement in RFS-SSE and RFS-NPR, are expected to bear fruits for the ongoing scale-up of the new approach, the monitoring and evaluation of that scale-up, and any adjustments made based on their findings. MoE officials are optimistic but ready to monitor scale-up and adjust as needed. MoE respondents from the central, regional, and provincial levels all shared eagerness to see how the implementation unfolds and what results from the new approach are found through the monitoring and evaluation system deployed by the MoE and the RFS-NPR team. They widely expressed the view that all key findings should lead to adjustments, as opposed to abandonment, of the new approach, and they were ready to make those adjustments as needed. This willingness to monitor the scale-up closely, trust more than just anecdotal evidence, acknowledge that issues may arise, and implement any necessary changes is positive and bodes well for the Arabic reading program’s sustainability. MoE officials at all levels displayed a high level of enthusiasm and commitment to the new method. It is critical to ascertain how enthusiastic and committed the MoE and its stakeholders are to the activities’ overarching objectives. MoE officials at all levels and in all regions—from those at the central level to school directors and teachers—view the new Arabic reading curriculum and what it entails as both necessary and beneficial. The strong buy-in to the new method systematically expressed by inspectors and Arabic language trainers also suggests they will follow through with the efforts needed to implement the scale-up effectively. A few MoE officials, particularly inspectors, did note that some teachers during training were initially skeptical of the new teaching approach for various reasons, including the fact that several other approaches had been tried in the last 50 years and that MoE had defended the reading approach being replaced by RFS-SSE and RFS-NPR’s new approach as effective. According to these inspectors, the teachers who were initially skeptical of the new approach were quickly convinced of its merits after learning about the research underpinning it and observing a demonstration of the new method and how the students reacted to it. Indeed, in FGDs, teachers widely endorsed the new syllabic approach and voiced the same sentiments as other respondents, such as inspectors, that it results in greater student engagement and faster learning. Parents also gave positive feedback on the new approach. In FGDs at schools that participated in RFS-SSE and RFS-NPR, parents indicated they share and value the goal of improving their children’s reading skills and perceived the impact of the new approach. For example, parents often said that they believed their children expressed more interest in reading. A lower level of and commitment for summer enrichment reading activities, however, was reported and reflected in actions within the MoE. The MoE’s enthusiasm and commitment to the activities’ summer program activities, however, may be significantly less strong, even if schools and communities seem to widely value them. According to several project stakeholders, this component never really captured the MoE’s attention, and unlike the national scale-up, it is not a priority. Furthermore, RFS-NPR will likely discontinue its work on this component due to the need to redirect resources for the unanticipated expansion of its scope to grades 5 and 6. According to respondents from CSOs that had implemented summer activities, they learned significant lessons from their work, retained materials such as books and training guides that can be reused, and expect to remain active moving forward. Without funding and support, however, it is unclear to what degree and at what scale this work can be continued. The question of scale is particularly relevant because only a small number of CSOs were involved in RFS-SSE and RFS-NPR’s summer enrichment programs, and therefore their combined reach will always be limited. The accelerated scale-up and the adoption of a law related to comprehensive reform of the education system are signs of sustainability. MoE staff not only at the central level but Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 61 also at the regional and provincial levels, widely stated that the acceleration of the scale-up timeline demonstrated their strong commitment to the new reading approach and confidence in its effectiveness. They also noted that the RFS-NPR activity is well-aligned with the MoE’s strategy and vision, which also improves the chances that the implementation is sustained and successful. Furthermore, RFS-NPR stakeholders all pointed out that the revision of the Arabic primary school reading curriculum is now part of a larger reform that has been enacted—via the Projet de Loi-cadre N°51.17. The new approach will also influence how other subjects are taught in primary school (and likely beyond). Respondents viewed this formal adoption and political recognition of the new syllabic approach as crucial in ensuring that its implementation continues, and its progress sustained amid potential changes in leadership at the MoE. Many of these same stakeholders also noted that there was very little, if any, the controversy associated with the new approach among experts, the public, and the media. They characterized this lack of disagreement as highly unusual in the Morocco context and a good sign for RFS-NPR’s sustainability. On the other hand, the evaluation identified several hurdles that may impact RFS-NPR’s sustainability to various degrees. Issues with the training of teachers on the new approach were a recurring concern of various RFS stakeholders within and outside the MoE. Effective teacher training is key to the success of the scale-up. The teacher training model relies on a multi-stage cascade, which has been disrupted in several instances. The model used by the MoE and RFS-NPR has suffered from important limitations that USAID, IPs, and some officials in the MoE view as negatively affecting its effectiveness and coverage. The training cascade for in-service teachers relies on the accurate and systematic transfer of information in three stages—first from expert trainers to a subset of inspectors, then from this subset of inspectors to their peers, and finally from all inspectors to the teachers they supervise. This process presents multiple challenges. First, some information is inevitably not passed on during each step of the cascade, and this inherent loss is compounded by a significant gap in time—sometimes two to three months—between the second and third stages of training. Furthermore, other disruptions, including inspector strikes over the last few years, have led to the outright cancellation of certain teacher trainings, particularly those related to grade 2. These trainings as originally designed were never carried out, and the few inspectors who reported this indicated they did the best they could to train teachers during their occasional school visits. The concerns with the cascade model are compounded by issues related to the training’s format and frequency. Teachers, school directors, inspectors, and regional-level MoE staff all expressed concern about the training format for teachers. The MoE authorizes just a few days for inspectors to be initially trained. Inspectors then have limited time to pass on this training to teachers—generally a half-day per week over a period of five or so weeks. The rationale for this approach is to limit teachers’ time outside of the classroom, but the respondents believed it was not enough time to pass on all the knowledge needed to teach the new method. Time outside of the school year or school hours, however, is viewed as off-limits by the MoE leadership, inspectors, and teachers. It is worth noting that the MoE faces challenges on the whole with respect to training, not just with NPR. MoE officials widely acknowledge that pre-service training is problematic. Teachers often do not have a degree in the fields they are assigned to teach, and continuing education is not the norm. In addition, inspectors in several provinces said they are already stretched thin in monitoring and coaching all teachers they supervise. Therefore, it may be too much to ask inspectors in the limited time they have with each teacher to fill in the gaps left from the in-service teacher training of RFS￾NPR. USAID/Morocco and RFS-NPR are working with the MoE to address these systemic training-related issues by developing MOOCs for teachers. Whether these MOOCs will ultimately be successful rests on the implementation of the online training as well as the quality of the content. Teachers in rural schools may have difficulty accessing the MOOCs, but the MoE indicated that virtually every school is now equipped with an internet connection because of the MASSAR rollout. One AREF Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 62 official suggested that USAID/Morocco and RFS-NPR should also consider using or developing a web-based evaluation platform for teachers’ skills in Arabic. This respondent mentioned a project operated jointly by the MoE and the Centres Culturels Français called EV@LANG that serves to test teachers’ knowledge and written and oral proficiency in French. According to the respondent, inspectors in this region require all 388 teachers to use it, and since the project platform already supports Arabic, it could be readily used as part of RFS-NPR and beyond. School directors have rarely received training about the new approach. Although school directors coach and advise their teachers throughout the year, they were typically not provided training on the new method. Nearly all school directors who were interviewed pointed out this fact and recommended the MoE and RFS-NPR reconsider their stance. School directors said they found it difficult to adequately fulfill their coaching and advising roles because they were not aware of the specifics of the new method. Some directors indicated they were teaching themselves from existing material and ad-hoc discussions with inspectors and MoE staff. In addition, MoE officials in at least one region said they encouraged school directors to attend some of the teacher trainings on a voluntary basis, although they acknowledged this makeshift solution was insufficient and unsystematic. Teachers, school directors, and inspectors lamented the lack of availability of supplementary material for the scale-up. Teachers in particular, but also provincial and regional MoE officials, reported that the availability of support materials for the new reading method was insufficient. Guides written for teachers and inspectors that complement training were reported to be only available in an electronic version, which is not ideal, especially in rural areas where computer availability is limited. Several inspectors and teachers lamented that they had to pay for the printing and binding of their own copies. Other items with limited availability include visual teaching aids such as letter and word flashcards, illustrations, and graphics, and animations that can be projected in classrooms equipped to do so. Based on FGDs with teachers and KIIs with school directors and inspectors, this problem is not new. Teachers are accustomed to having to make their own materials, which the evaluators observed firsthand during several classroom visits. Support materials were clearly homemade and varied significantly between classrooms, illustrating that they are not standardized or provided by the MoE. In addition, multiple teachers and inspectors, as well as Arabic language trainers, remarked that significantly more teaching aids are required to teach the new method effectively. In FGDs, many teachers also expressed the need for projectors and computers in the classroom to facilitate their work. In some urban classrooms, the evaluators observed teachers had brought their own equipment from home and used it to demonstrate some of the new teaching methods in the classroom. The reported lack of storybooks is also concerning with respect to both NPR’s success and sustainability. RFS-SSE and RFS-NPR persuaded the MoE to carve out more time specifically for students to read—rather than teaching Arabic via reading—but the distribution of reading materials such as storybooks is lacking under RFS-NPR. Teachers and school directors indicated and showed the evaluators that they do their best, and often rely on their own means, to put together a small library of age-relevant books. MoE staff with a pedagogy background as well as teachers and school directors all agreed that the availability of diverse and relevant reading materials was critical for students to cement the gains they are making as part of the new teaching method. The development of student textbooks has been constrained by specifications in MoE’s cahier de charge, and their release was delayed significantly in several instances. Several critical challenges with student textbooks may affect the larger reform. First, the current MoE policy regarding textbooks’ format and length hinders their improvements in tandem with the larger adoption of the new method. This issue is an area of focus for RFS-NPR and the MoE as of the writing of this report and will hopefully be resolved. Secondly, the release of textbooks was delayed due to the revision of their content. For example, textbooks for the 2019-20 school year were available about a month after classes started. That said, issues regarding school textbooks in Morocco are incredibly complex, and new challenges may arise. Many textbooks are approved or at Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 63 least produced in compliance with the MoE cahier de charge for a given grade, and they are dramatically different from each other. In addition, the price of textbooks has political implications. A longer textbook—or the use of two textbooks for a grade—would likely increase costs, which are borne by the government and parents in primary schools. RFS-HICD The goal of the RFS-HICD activity was to recommend performance solutions to further enhance the human and institutional resources required to meet the goals of Vision 2030. RFS-HICD’s scope and mandate, therefore, encompassed more than just the Arabic reading component of the curricula, assessing key functional areas and directorates of the MoE on which RFS-SSE, RFS-NPR, and RFS￾IDCRT relied. The RFS-HICD activity should, therefore, play an important role in facilitating RFS activities, particularly RFS-NPR. Indeed, RFS-NPR requires a whole-of-MoE effort for the coordination, management, evaluation and monitoring, and development of RFS activities. However, the full potential of RFS-HICD was never fully realized (as discussed under evaluation question 4.2) and its sustainability—meaning the process of implementing recommended changes, even if they are modified somewhat to reflect the current state, structure, and operations of the MoE—remains in question. Formal approval and steps for adaptation and implementations of the RFS-HICD recommendations may still occur in the near future. As of mid-2017, the formal portion of RFS-HICD’S work—namely conducting the human and institutional capacity assessment of the MoE and reporting findings and recommendations—was completed. However, the MoE never formally approved or acted upon these deliverables. According to USAID/Morocco officials, formal approval did not occur, not because of a lack of perceived or intrinsic value of the RFS-HICD activity, but rather due to issues regarding timing and change in leadership within the MoE as well as framing and communication. Indeed, when new leaders joined the MoE, they requested the findings be reworded to be reconsidered for formal approval. USAID hired one of the local consultants who worked on the original RFS-HICD assessment to work closely with the Director of Curricula to complete this task. However, the revised presentation had not yet been delivered to the Minister when the fieldwork for this evaluation was conducted. In addition, one top MoE official expressed concerns about the assessment’s siloed nature and the fact that the MoE’s structure had changed significantly since the release of the assessment results. If the Minister grants his approval, RFS-HICD’s findings and recommendations may yet be able to spur renewed buy-in for the next phase—updating and adapting some or all of the recommendations made in the report and formally and systematically implementing them. Sustainability could be measured during this second phase, which would require extensive MoE leadership and which USAID/Morocco envisions would also involve one or more external advisors. RFS-IDCRT The RFS-IDCRT activity ended in mid-2018, and interviews with the implementation partner and CSOs and teachers involved indicate valuable relationships were formed that could lead to beneficial partnerships in the future for Deaf education in Morocco. Some of these stakeholders thought the activity generated important lessons on how Deaf education could be improved and scaled yet many respondents reported that it seemed the MoE has not capitalized on these strengthened relationships as some of the lessons learned have been implemented yet. In addition, the software developed and used during the two-year activity is not viable for future use because it cannot be updated, which resulted from a lack of clarity regarding its use once the activity closed. Relationships and interactions between CSOs developed during the activity remain in place. Project stakeholders indicated that the relationships and collaboration that started under RFS-IDCRT have largely been sustained. For example, participating CSOs confirmed that they continue to interact with each other through joint activities and visits. They are also leveraging their relationships to advocate more effectively and extensively for the educational needs of children who are deaf or hard of hearing. The linkages between CSOs and schools also appear to remain largely in Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 64 place, and stakeholders mentioned that participating teachers and parents are now more invested in Deaf education. One person also perceived that the ties between participating CSOs and the MoE had been strengthened, and there is greater recognition of the CSOs’ value by the MoE. Teacher training, however, appears to have largely stopped once the project ended. The MoE has made efforts to capitalize on RFS-IDCRT that are not fully recognized by all respondents. Respondents, including regional/provincial MoE officials and school-based respondents, commonly reported seeing few signs that the MoE intends to build upon the work done under RFS-IDCRT in practice beyond the recent passing of legislation meant to protect the rights of individuals with disabilities. Policy-level changes included a 2016 constitutional change requiring Education for All as well as the development and release in 2019 of the MoE’s Frame of Reference of Inclusive Education for Children with Disabilities.74 During report validation, USAID/Morocco education officials provided additional details on actions taken by the MoE to build on the accomplishments of RFS-IDCRT. For example, the MoE now recognizes its responsibility to provide education to students who are deaf or hard of hearing and has expressed commitment to working with NGOs to support the education of students with disabilities while MoE capacity is strengthened. In addition, the MoE entered a memorandum of understanding with USAID, a leading NGO, and the University Mohammed V to start a new, comprehensive teacher training program for all teachers working with students who are deaf or hard of hearing. The new program is tentatively titled, “Reading for Success-Teacher Training for Inclusive Education.”75 As described above, USAID/Morocco officials reported that RFS-IDCRT also led to the MoE Curriculum Directorate to open a unit dedicated to inclusive education. The reasons why many respondents outside of Rabat shared the perception that the MoE has not capitalized on RFS-IDCRT’s accomplishments are opaque, but it is also likely that they were unaware of the most recent Central MoE developments at the time of data collection in October 2019. It may also take time for the most recent policy-level developments to reach and be recognized respondents far from Rabat. Nonetheless, a couple of regional stakeholders as well as USAID/Morocco did note that some changes were made on an ad-hoc basis at the regional and provincial levels, such as reserving time to teach reading to students who are deaf or hard of hearing, providing them access to computers, and providing space within public schools for NGOs to reach students who are deaf or hard of hearing. CSOs did report developing their capacity but did not seem to be in a position to take over the activity. The limited sustainability of RFS-IDCRT gains seems to largely revolve around the lack of clear ownership and planning when the activity ended. Most CSOs do not seem able to organize to the point where they can continue certain key activities, such as providing regular training to new teachers. That said, one CSO did indicate providing training to new teachers who start at the organization, largely through the work of the Director of Pedagogy who is very invested and involved. CSO leaders largely reported that they still see much of this work to be the responsibility of the MoE, while they play a supporting role and have been advocating that end with limited results. Among their principal concerns are the lack of adaptation of the MoE curriculum and of the end of primary cycle exams for the needs of children who are deaf or hard of hearing, and better pre-service training for teachers to work with children with disabilities. The latter is particularly relevant because teachers tend to move between schools frequently in Morocco, which means that some of the teachers trained under RFS-IDCRT have been replaced by others without the benefit of training provided as part of the activity. 74 See for example Chapter III of the Loi-cadre n° 97-13 adopted April 27, 2016 and related to the protection and promotion of individuals with disabilities. 75 USAID/Morocco, RFI-608-20-0001, USAID/Morocco Reading for Success-Teacher Training for Inclusive Education (March 31, 2020). Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 65 The lack of clarity regarding usage of the RFS-IDCRT software once the activity closed resulted in a software product not viable for future use because it cannot be updated. The Clip and Create software developed and used during the RFS-IDCRT activity included a dictionary of approximately 3,000 signs in Moroccan Sign Language (MSL) with regional variations. The signs were documented with photos and videos, as well as the corresponding word in MSA. The software also enabled teachers to create instructional materials incorporating both MSA and MSL and included six stories translated into MSL from MSA. Educators used the software during the life of the project, and those interviewed for the evaluation study said they have continued to do so. Now that the activity has closed, however, the software’s usefulness is quite limited. Respondents noted the dictionary is incomplete with only about 3,000 signs, and they would like to add words to it. They are unable to do so because the software can no longer be updated since the project’s closure in 2018. This inability to modify the software results from a lack of clarity before the project closed about how the software would be used in the future, according to interviews with USAID officials and project implementers. Although USAID/Morocco intended for the software to be handed over in a state in which the MoE could update it, including adding new words the dictionary, the contract for the RFS-IDCRT activity, which was similar to other activities in the All Children Reading: A Grand Challenge for Development Round 2 portfolio, did not clearly spell out what should happen with the software once the activity closed. Therefore, after the closure of RFS￾IDCRT, the software developed authorized the MoE to use and replicate the software for any interested party to use, but it could not make any improvements or modifications to it. Therefore, the software is not a viable, completed product, as noted by the CSO respondents’ suggestions for improvements. CONCLUSIONS In order to formulate conclusions from the evaluation study, we synthesized findings from all five cross-cutting research questions and identified five themes to which high-level takeaways are pegged—MoE ownership; implementation; inclusion; research, monitoring, and evaluation; and community engagement. These conclusions then informed the development of key lessons learned and recommendations addressed in the sixth and final research question later in this section. 1. MOE OWNERSHIP: Evaluators observed a high degree of MoE ownership with RFS, including USAID promoting the role of the MoE as the driver of the project; however, if the MoE is the principal driver of RFS, not all strategies worked as intended, and information was not always shared sufficiently. Respondents from MoE personnel at the regional and provincial levels to project implementers and USAID/Morocco officials all pointed to one party as being in charge of the RFS project—the central MoE. The country’s legal framework and national educational strategy all have positioned the MoE to sit in the proverbial driver’s seat for the project, and USAID/Morocco officials, project implementers, and others have promoted the MoE’s leadership role since the inception of RFS, which featured a series of research activities and workshops to build consensus around the early grade reading landscape in the country. Once the project activities were designed and implemented, implementers empowered MoE staff in multiple directorates through technical support and capacity￾building to design the activities and direct and monitor their rollout, which was primarily exemplified by the MoE’s decisions to expand the new reading curriculum and materials nationwide earlier than initially planned and to more grades than expected—grade 5 and 6 in addition to grades 1-4. Regional and provincial MoE officials supported the central MoE by buying into the new approach to teaching reading in classrooms and provided crucial support, especially in the regions where activities were piloted. Despite the MoE’s initiative in taking the reins of RFS, its role did lead to some drawbacks. Project implementers had to be nimble in the allocation of resources and personnel in order to follow through on some of the MoE’s decisions that countered the initial plans of project implementers and USAID/Morocco, notably the earlier-than-anticipated expansion of the revamped reading curriculum. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 66 Some activities were, therefore, affected by delays in the scheduling of trainings and workshops and the approval of materials; the quality of materials was also compromised at times because of the accelerated timelines for rollout. In addition, the roles that central MoE officials should play in conjunction with project implementers in implementing activities were also unclear at times, and constraints on certain activities the MoE put in place, notably the design and length of teacher training activities, did not always meet the ideal specifications of project implementers. Further, communication between different parties was not always smooth. RFS research, including EGRA reports, did not always reach MoE officials. Provincial MoE offices also reported uneven participation in RFS operations, noting their role was to “to take their orders from the top” and, to the extent possible, coordinate training and materials provided to the schools. However, provinces were not involved equally in the training of trainers, did not receive materials for all schools, and did not have sufficient independent budgets with which to manage the rollout. 2. IMPLEMENTATION: Stakeholders praised the new teaching methodology and materials for transforming teaching, animating classrooms, and improving reading; however, evaluators found considerable variation between the degrees of support received by schools across provinces as well as other obstacles in delivering the new methodology and materials as effectively as possible. Across interviews, respondents perceived that—and sometimes had high praise for how—the new teaching methodology and materials had transformed teaching, animated classrooms, and improved reading in a short period of time. Though not fully borne out by the quantitative analysis, respondents frequently cited anecdotal evidence from the RFS-SSE schools as evidence that children can learn to read in a matter of months, not years. The contributions of the new early grade reading approach to each of the 5 Ts of learning instruction—time, technique, text, tongue, and testing—was evident from input from MoE officials, teachers, and school directors. Respondents said they were satisfied with the increased amount of time allocated to reading at school, and they almost unanimously viewed the new phonics-based approach as superior to the former whole word approach. As for the new textbooks, respondents praised their quality and wide availability and how they sparked students’ imaginations. However, this evaluation also found considerable variation between the degree of support received by participating schools as well as other obstacles to implementing the new curriculum as effectively as possible. While RFS-SSE teachers and schools received extensive training and material support, many respondents from the RFS-NPR schools reported having received little or, in some cases, no support. In some instances, inspectors reported having trained only a sole teacher in a school, and that teacher was expected to share his or her new knowledge with colleagues. If teachers had not received any materials, they sometimes reported looking online. This level of variation can be attributed in part to the inspector boycott that affected the training efforts for the scale-up of the curriculum in grades 1 and 2 for the 2018-19 school year as well as the lack of allocated resources, logistical constraints, and other challenges that negatively affected training activities. Other challenges cited by multiple stakeholders in implementing the new reading approach included a lack of supplementary materials and delays in receiving materials. 3. INCLUSION: Impressively high levels of access for students were noted at the primary level; however, obstacles continue to hinder greater participation of vulnerable populations, including children with learning needs. Morocco has impressively high levels of access, with approximately 99 percent of children enrolled at the primary level. Nevertheless, access to quality education has remained limited for children with special needs or vulnerable status, such as language minorities. Recently, several initiatives in Morocco have sought to address this concern. USAID/Morocco supported RFS-IDCRT to explore strategies to teach students who are deaf or hard of hearing to read in the early grades; the MoE collaborated with several CSOs in this endeavor. Additionally, the MoE established a policy requiring Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 67 all schools to provide access to children with special needs and created the Inclusive Education Unit to oversee the development of initiatives to include vulnerable populations. However, numerous obstacles continue to hinder greater participation by vulnerable populations, including children with special learning needs. The MoE’s provision ensures each school has a classroom for these children, but it does not require schools to staff the classroom with specially trained personnel or equip the classrooms with appropriate equipment and materials. As in many countries, Moroccan children with special needs are often stigmatized, and no official strategy has yet been developed to counter these attitudes. Additionally, where schools do exist for children with special needs—such as for children who are deaf and hard of hearing—the centers are often privately managed with no mechanism for MoE oversight or ongoing support, let alone a transition plan for MoE’s ultimate administration of the centers. Finally, it can be daunting to meet the needs of these populations. As was learned in the RFS-IDCRT activity, children who are deaf or hard of hearing face a lack of standardized sign languages across Morocco, limited equipment and learning materials, and insufficient support from families and communities at home. While RFS-IDCRT focused on deaf and hard of hearing students, RFS as a project did not direct specific attention to the needs of other subpopulations, including children who are blind or visually impaired, children with learning difficulties (e.g., dyslexia, dyscalculia), children with emotional or behavioral problems (e.g., autism, Attention Deficit Disorder, trauma), and children from vulnerable backgrounds (e.g., poor children, children who had not attended pre-primary, language and ethnic minorities). 4. RESEARCH, MONITORING, AND EVALUATION: The evaluation identified positive examples of RFS utilizing research, monitoring, and evaluation over the course of the project; in spite of these efforts, however, the sharing of data has often been limited, and potential avenues for discussing results across donor and MoE departments have not been fully exploited. USAID has conducted research, monitoring, and evaluation through efforts such as institutional capacity assessments, reviews of textbook procurement and quality, EGRAs, and program evaluations. The MoE has monitored the rollout of its programs and the expansion of the curriculum reform by tracking progress against national indicators, consolidating data and data collection, and processing procedures in its MASSAR system. Information from these efforts has been presented in national forums to inform decisions about the ongoing design of the curriculum reform as well as its scale-up to grades 3–6 and all schools nationally. In spite of these efforts, the sharing of data has often been limited, and potential avenues for discussing results across donors and ministry departments have not been fully exploited. There are several reasons for this. First, shifts in RFS-SSE’s and RFS-NPR’s activity priorities and scopes have limited the amount of time available for purposeful engagement and reflection. Second, mechanisms often did not exist to ensure the results of studies were made available in a timely manner or accessible language. Third, systems have not yet been linked, including USAID’s indicators and data and those of the MASSAR system. Finally, while some respondents noted that RFS’s framework had provided opportunities to learn about research—including, how to understand analyses and interpret EGRA data—such opportunities have been rare. 5. COMMUNITY ENGAGEMENT: The activities geared toward community engagement that RFS implemented, notably summer reading enrichment activities, were successful, but their impact was narrow due to limited implementation and little focus from project implementers and the MoE; overall, respondents expressed a desire for more community engagement in schools and with parents. In the results framework in USAID/Morocco’s PAD that outlined the initial design of RFS, the second of three intermediate results centered on increasing community support for reading. The Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 68 implementation of summer reading enrichment activities by Moroccan CSOs during RFS-SSE and RFS-NPR was the embodiment of this intermediate result, and respondents noted how communities were supportive of these activities. Their impact was limited, however, because only ten CSOs put on activities during RFS-SSE—with an average of 50 students per group—and three did so for RFS￾NPR; in addition, despite project implementers’ plans to implement other interventions to engage communities, they had to drop most of them. They had no choice but to dedicate their resources to the national scale-up of the reading curriculum to ensure its success; in addition, the MoE naturally viewed this expansion as a higher priority than activities addressing community engagement. The desire to implement these types of activities and the community’s demand for them were common refrains, however, during KIIs and FGDs. Project implementers and USAID/Morocco officials noted their value and described ideas that had to be left on the cutting room floor. Parents, teachers, and school directors expressed how children’s reading skills could be improved by organizing extracurricular activities and educating parents on strategies to support their children’s literacy. They also discussed how one community engagement activity not associated with RFS—the United Arab Emirates-supported Arab Reading Challenge—has played a role in promoting reading in their schools and communities. 6. SUSTAINABILITY: Evaluators observed varying levels of sustainability among the four RFS activities. The sustainability of the Arabic primary school reading curriculum appears strong. Many of the recommendations raised by RFS-HICD remain relevant but may need to be re-examined in light of the MoE’s recent evolution, while the prospects for IDCRT’s sustainability are mixed. The work started by RFS-SSE, and continued with RFS-NPR, displayed several encouraging signs of sustainability that, by extension, bode well for the sustainability of the national reading program in Morocco. For example, the revisions to the Arabic primary school reading curriculum are now part of a larger reform that has been adopted politically. Evaluators found significant evidence of the MoE’s strong ownership and leadership, which is exemplified by the accelerated national scale-up of the new Arabic reading curriculum and corresponding changes to the pre-service teacher training curriculum. Furthermore, MoE staff from the central level to the teachers and school directors consistently expressed their enthusiasm and optimism for the reform and the associated teaching methods. In fact, the activities’ scope continues to grow; RFS-NPR is now responsible for the development and scaling of the curriculum in grades 5 and 6. MoE officials also expressed their readiness and commitment to monitoring the scale-up closely using quantitative and qualitative methods so they could identify problems and address them promptly. This attitude is conducive to a successful and sustainable implementation of reforms. However, the MoE, USAID/Morocco, and activity implementers need to carefully manage potential impediments to the successful national scale-up of the new Arabic reading curriculum, including the reach and frequency of in-service teacher training, the lack of training provided to school directors, and the lack of available supplementary material for the scale-up. Project partners are already focused on addressing training￾related issues with the development of online training, and the provision of support material does not seem to constitute a major challenge so long as it is also viewed as a priority. The full potential of RFS-HICD was never fully realized and its sustainability—meaning the process of implementing recommended changes, even if they are modified somewhat to reflect the current state, structure, and operations of the MoE—remains in question. Yet several of the core RFS-HICD challenges and recommendations emerged organically during interviews with various stakeholders, suggesting they remain relevant to a certain degree, even if they may not completely reflect the current situation within the MoE. On the other hand, the prospects of RFS-IDCRT’s sustainability are mixed. While participating CSOs have maintained strong relationships and are, therefore, better positioned to assist each other, support students who are deaf or hard of hearing, train new teachers, and conduct effective advocacy, funding is a constraint and the core software deliverable is now of limited use. The software cannot be migrated to mobile platforms, which are more readily available to families and teachers, and additional content cannot be added to address gaps. This situation may have been avoided if a clear transition plan for the activity had been established prior Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 69 to closeout and if explicit contractual terms regarding the nature of the software’s use once the activity closed had been spelled out. LESSONS LEARNED AND RECOMMENDATIONS 6. WHAT KEY PRELIMINARY LESSONS LEARNED AND RECOMMENDATIONS CAN BE GLEANED FROM THE IMPLEMENTATION OF THE PROGRAM TO DATE? In order to capture key learnings from USAID’s RFS program to date, evaluators asked stakeholders—including USAID/Morocco officials, USAID implementers, MoE officials, school directors, teachers, and parents—to share their observations of lessons learned and recommendations from the implementation as well as educational work in Morocco moving forward. We also reviewed existing RFS project documents and other relevant literature identified in the desk review. Drawing on these findings, we identify cross-cutting themes of lessons learned that emerged as most pertinent. The following section also provides recommendations for future USAID and MoE activities focused on supporting early grade reading outcomes in Morocco. LESSON 1: MOE OWNERSHIP Additional measures should be taken to strengthen MoE’s ownership of the reform, including clarification of roles, improved information sharing, and negotiation of priorities within contractual requirements. It is essential that the MoE and its national-level structures work directly with stakeholders to support the implementation and scale-up of a national-level reading system and to sustain RFS activities and outcomes beyond the life of the project. To that end, one of the most important conclusions of this evaluation is the high degree of MoE’s ownership with RFS. Evaluators identified five recommendations to enhance and strengthen the MoE’s ownership of early grade reading activities. Recommendations 1. As RFS extends to all schools in Morocco and upward through grades 5 and 6, spell out the roles more clearly to be played by MoE personnel and departments at central and provincial levels in areas of program design, development, and implementation, and ensure these roles are performed. 2. Identify, mobilize, and deploy resources required to roll out RFS nationally at each grade level, and the role to be played by each MoE personnel and departments in coordinating training and managing resources. Include the provision of detailed budgets and expenditure authorizations at the provincial level. 3. Enable the RFS Steering Committee to provide program design review, monitoring, and quality assurance. Empower the Steering Committee to align MoE strategy and priorities with USAID’s contractual requirements. 4. Establish a mechanism for sharing information with MoE personnel at central and provincial levels on a timely basis. Prioritize resources for personnel to reflect on progress and outstanding issues and to develop strategies that improve implementation. 5. Ensure that any additional institutional capacity development efforts reflect the evolving systemic nature of the MoE and include provisions for involving the MoE in developing strategies to strengthen these systems. LESSON 2: IMPLEMENTATION Respondents universally recognized the excellent quality of the RFS reading program; however, as the program expands to grades 5–6 in all schools in Morocco, careful consideration must be given to how it is scaled. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 70 Evaluators identified two lessons learned by respondents. First, a viable strategy must be in place to ensure all teachers and schools are supported. Even if training and materials support cannot achieve the intensity of RFS-SSE schools at the expansion stage, a minimum package of support must be defined. This should include modalities for delivering teacher training, including some combination of face-to-face training, school-based support, and distance training and support. Similarly, a formula for determining appropriate quantities of materials will need to be established so that each teacher and school receive enough teacher training guides, storybooks, textbooks, parent support guides, and other print resources. Importantly, given the kinds of changes sought by RFS—including changes in attitudes, behaviors, and roles for teachers and school directors, and parents—the emphasis should extend beyond “teacher training and materials” to also include how RFS can support long-term change that creates the conditions necessary for the RFS reading program to take root. Second, in order to scale the RFS reading program in an effective and equitable manner, the MoE with the support of USAID/Morocco should develop a strategy for deploying instructional materials and support for program actors—including teachers, school directors, parents, and university and CRMEF lecturers—to deliver the program over time to all schools through grade 6. This would involve linking the teacher and school change modalities cited above with roles, a rollout plan, deadlines, budgets, and indicators of success and how they will be measured and reported. Recommendations 1. Define a minimum package of support to be provided to schools participating in each activity, including modalities for teacher and school director support and minimum levels of training and materials. Ensure that the plan provides for a review of current teacher training, support methods, and materials for their efficacy. In addition to minimum requirements, such a package will likely vary based on region, as more remote locations may require larger proportions of distance support via computers and mobile devices. 2. Develop and sustain a dynamic plan for scaling that is updated and followed by all relevant stakeholders; delineates the roles of key actors by grade, year, and province; and identifies resources required for scaling, including transportation, communication, finance, and technical assistance. Ensure that the plan considers requirements for both initial and ongoing training of CRMEFs, universities, and MoE units supporting the scale-up. 3. Establish indicators for donors and the MoE to use to determine the extent to which the objectives of the scale-up are being met. Establish mechanisms through which progress against these indicators are reviewed on a regular basis, and decisions are taken to improve the effectiveness of the scale-up. 4. Conduct further research on the reasons underlying the limited impacts of SSE. This might include a review of program implementation fidelity (e.g., in terms of teacher training, support supervision, and/or availability and use of materials) as well as an exploration of other enabling/disabling environmental factors in targeted schools and communities. Furthermore, because the structured pedagogy approach used in SSE has been impactful in other contexts, there may be an opportunity to share lessons learned with other USAID Missions around the world. 5. Ensure that sustainability is at the center of discussions. Identify the kinds of support to be solicited in the near term, as well as the level of support the MoE should be prepared to provide in the medium and long term. LESSON 3: INCLUSION Important strides have been made to ensure all children are included in public education in Morocco. However, vulnerable populations and children with special needs will not have sufficient access to a quality education until a strategy is defined to determine how to address each child’s needs, and a national effort is made to mobilize support for inclusive education at school, in the community, and at home. Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 71 Three primary lessons emerge from this evaluation to strengthen Morocco’s provision of inclusive education services: (1) There is a need to identify the different types of inclusion that Morocco wishes to foster; (2) Once identified, specific types of support must be developed to target the particular needs of these populations; and (3) An effort should be made to cater both to children outside the education system and also to those within the system who face barriers that hinder their ability to learn. Recommendations 1. Develop a national strategy for increasing the MoE’s capacity to provide inclusive education over time. This would result from two prior actions: a. Conduct an analysis of the types of inclusion issues that need to be addressed in Morocco, with a balance between the prevalence of different conditions and the potential cost to address each of them. b. Based on this analysis, prioritize the types of inclusion to foster over time—in the near, medium, and long term. Determine which populations to support initially and which strategies to use to ensure they are adequately addressed. Importantly, this strategy should include a provision to transition the responsibility for providing inclusive education opportunities from CSOs to the MoE. 2. Develop an inclusive education plan to be shared with development partners ensuring political, financial, and technical support for the rollout of the national strategy, including needs-specific staffing and training and the provision of appropriate infrastructure and material support. 3. Develop an inclusive education communication strategy to enable the MoE and its partners to implement effective outreach strategies that help parents understand the educational needs and capabilities of children with special needs and vulnerable children as well as their own role in supporting their children at school, in the community, and at home. 4. Strengthen the technical understanding and capacity of the MoE to implement its inclusive education strategy. LESSON 4: RESEARCH, MONITORING, AND EVALUATION USAID and the MoE have conducted research on their programs and tracked progress based on results frameworks and national indicators. However, to use the information from these efforts to inform decision-making routinely, M&E systems and research processes require strengthening. The lessons from these efforts, then, are three: (1) “Common ground” should be explored between future USAID-supported projects and the MoE’s indicator framework so that shared data collection, analysis, and use strategies can be explored; (2) Mechanisms for sharing results in both formal and informal ways should be established; and (3) The capacity of the MoE to continue playing a role in monitoring, evaluating, and researching the curriculum reform—including design effectiveness, scaling issues, and impact—should be strengthened. Recommendations 1. Develop an information strategy to be used by USAID/Morocco and the MoE to establish a common research, monitoring, and evaluation framework. This should include the identification of standard indicators, types of data to be collected, data collection and archival protocols, and data analysis and interpretation procedures. Where possible, the strategy should specify formats for sharing data, actors with whom data are shared, and the types of decisions that can be made in each instance (e.g., evaluation briefs shared in French with the Steering Committee every six months to identify bottlenecks in scale-up). 2. Where appropriate, support efforts to “collaborate, learn, and adapt” (CLA) among USAID implementers, the MoE, and partners. While this approach is assumed in the previous recommendation, a full CLA plan would start with a theory of change, then document its evolution in a variety of ways—not just through the use of rigorous research methods but Contract No. GC-10F-0033M/AID-OAA-M-13-00010 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 72 more informal approaches, such as the use of observational data, anecdotal sources, and testimonials. A CLA approach would also encourage informally sharing information with beneficiaries through interviews, community meetings, drama, and other outlets. 3. Strengthen the capacity of the MoE and IPs to conduct research, monitoring, and evaluation. Address means for the development of indicators, research design, test construction, electronic data collection, core analysis techniques for EGRA-type assessments, and data interpretation strategies. LESSON 5: COMMUNITY ENGAGEMENT At both central and provincial levels, respondents stressed the importance of education outside the classroom, including the need for materials and activities to engage parents and communities. However, RFS’s scope limited support for parental and community engagement, raising questions about the importance and effectiveness of such initiatives. This evaluation found substantial support for the notion that the process of learning to read is complicated and needs to continue outside the classroom. Stakeholders from school directors to teachers to parents frequently cited the need for libraries and extracurricular activities as well as the essential need to educate parents and community members about supporting their children in reading and equipping parents with materials and strategies to do so. The RFS-supported summer enrichment program aimed to address this need and, by many reports, was successful in motivating children to try to read and, in some instances, improving their reading. As noted elsewhere in this report, several factors limited the success of these programs, including misalignment with MoE priorities, the limited size of the beneficiary population, and competing demands for time and resources on the part of implementers. Importantly, because the summer program was implemented through local CSOs, it is unlikely that efforts like these could be sustained over time. These challenges notwithstanding, several lessons can be learned from RFS concerning community engagement, including the need to identify effective types of out-of-school activities that can motivate parents, the need to prioritize time and resources for these activities, and the need to ensure that the MoE agrees to its importance and supports a strategy to achieve it. Recommendations 1. Conduct a landscape review to identify effective strategies for supporting early grade reading learning outside the classroom, both within and outside Morocco. The review should examine a broad range of types of interventions for higher and lower-income contexts, including programs that support children with special needs or other vulnerable populations. 2. Investigate the political will of the MoE and potential partners—including parents associations, CSOs, private sector—to support extracurricular activities to support early grade reading. If appropriate, organize events to raise awareness of decision-makers about the importance of family and community support for children’s learning in the early grades. 3. If needed, develop a model in collaboration with the MoE for parent- and community-based support for early grade reading that can be deployed at the provincial level. Explore mechanisms for sharing the cost across the MoE, community groups, and the private sector in order to improve the chances of sustainability. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 73 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS / STATEMENT OF WORK C.1 PURPOSE OF THE EVALUATION The purpose of the performance evaluation of the RFS project is threefold: 1) To assess the extent to which all the activities under the RFS project together contribute to achieving the Education DO; 2) to inform the implementation of ongoing or upcoming activities, including that of the National Program for Reading; 3) to inform the next Country Development Coordination Strategy design. The results of the RFS evaluation will be used by the USAID/Morocco Education Office to inform the design of the new strategy and make any adjustments needed to improve the implementation of the National Program for Reading and other activities, including those related to deaf education. The results will be shared with broader USAID audiences, including USAID/Washington and other Missions, to highlight best practices and lessons learned, and to inform upper management of the results achieved by the Morocco Mission in basic education and early grade reading. Finally, relevant results will be shared with Government of Morocco (GOM) counterparts and other donors to further promote improving basic education in Morocco through enhanced reading instruction in early grades. Note: This evaluation is not intended to be an impact evaluation. As explained in detail in the present SOW, the Evaluator is expected to use a primarily qualitative approach to addressing the research questions of interest. Nonetheless, the Evaluator is highly encouraged to consider and suggest the collection and use of quantitative data, whenever possible and relevant, to help strengthen and nuance the qualitative findings. Whenever both qualitative and quantitative data are used, the Evaluator should clearly describe how it would effectively weigh and integrate the two types of data to address the research questions. C.2 SUMMARY INFORMATION OF ACTIVITIES FOR RFS PROJECT Activity Name Implementer Agreement / Contract Number Total Estimated Cost Time Frame (start / end date) Geographic Regions Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE) Chemonics, International AID-608-TO￾15-00002 $5,820,752 Sept 22, 2015 to March 22, 2018 4 regions: Rabat-Kenitra; Oriental; Souss-Massa; Fes-Meknes Reading for Success - NATIONAL PROGRAM FOR READING (RFS￾NPR) Creative Associates International, Inc. AID-608-C-17- 00001 $19,272,436 May 19, 2017 to May 19, 2022 National USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 74 Activity Name Implementer Agreement / Contract Number Total Estimated Cost Time Frame (start / end date) Geographic Regions Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT) Institute for Disabilities Research and Training (IDRT) AID-608-A-15- 000002 $1,563,935 Oct 9,2015 to Oct 8, 2018 8 provinces: Sale, Rabat, Tangier, Marrakech, Agadir, Fes, Meknes, Casablanca Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD) Kaizen Institute AID-608-TO￾16-0001 $2,013,404 Jun 16, 2016 to June 30, 2017 National The SOWs and the Monitoring, Evaluation and Learning Plans will be shared with the bidders as references, and these will include information on beneficiaries for each of the activities; activity-level data collected, the timing of that data, and the sample size for that data, and all reports completed to date for each activity. C.3 BACKGROUND Description of the Problem Problem Statement: In the past decade, Morocco has made significant gains in making primary schooling nearly universal with net enrollment rates in grade one surpassing 97 percent. However, in terms of learning achievement, Morocco lags behind other lower middle-income countries, and has nearly the lowest overall literacy rates in the MENA region. The youth literacy rate is 87 percent for males and 72 percent for females76. These numbers hide significant disparities in literacy between rural and urban areas in Morocco, with a staggering illiteracy rate of over 51 percent among rural girls77. Based on enrollment and literacy data, the United Nations Development Program (UNDP) considers Morocco to be 15 years behind Tunisia, ten years behind Algeria, and five years behind Egypt in terms of overall educational outcomes78. Ultimately, performance indicators for each of the RFS activities are focused on closing this gap. On international assessments, Moroccan students rank among the lowest achievers. In the 2011 Progress in International Reading Literacy Study (PIRLS), Moroccan fourth graders ranked last among the 48 participating countries, with an average scale score of 310 (PIRLS scale centerpoint is 500). In 2007 and 2011, Morocco ranked eighth out of the nine participating Arab countries, in the Trends in International Mathematics and Science Study (TIMSS)79. A National Analysis of Student Performance (PNEA) conducted in 2008 by the Higher Council of Education (a research and watchdog organization reporting directly to the King) showed that only 24 percent of fourth graders and 36 percent of sixth graders possess the basic reading skills mandated 76 UNESCO 2011, including the Education for all 2000 Assessment. 77 Direction de la Lutte Contre l’Analphabétisme (DLCA), Ministry of Education, 2011. 78 World Bank Report No: 52749-MA, 20 April 2010. 79 PIRLS and Trends in International Mathematics and Science Study 2011 Reports. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 75 in the national curriculum. The study showed that less than 32 percent of students could read at grade level. At all four grade levels examined in the study, girls outperform boys in reading but not in the other subject areas. It is hypothesized that girls seemed to better master the various reading sub-skills than their male peers. This national study associates the causes of poor levels of literacy to a multitude of factors including a weak reference framework for reading skills (standards and benchmarks), a rigid and overloaded curriculum that does not respond to children’s needs, inadequate reading assessment tools, a school environment not conducive to learning, and the persistence of inequality in access to school and education opportunities80. In 2011, USAID funded an Early Grade Reading Assessment (EGRA) in the Doukkala Abda region. The EGRA mirrored previous findings and showed that only 34 percent of students in grades two and three read well enough to fully comprehend a grade-two level text. Boys and girls equally lack basic letter naming skills and are not able to sound out simple words, and 33 percent of second graders and 17 percent of third graders (21 percent for girls and 29 percent for boys both grades combined) could not read a single word of text in Classical Arabic, the language of instruction in the formal education system81. Only 2.5 percent of students tested responded correctly to five of the six reading comprehension questions on the EGRA. The assessment showed that factors inhibiting student performance are multifaceted and complex: • Inadequate levels of teacher preparation: Both the Snapshot of School Management and Effectiveness (SSME) component of the study and a recent review of the pre-service teacher education curriculum showed that Moroccan teachers receive little to no specialized training in reading instruction and assessing students’ reading abilities, leaving them ill-equipped to help children excel in the early grades. The absence of supplemental reading material for all students at school severely constrains the overall learning environment. • The absence of reading materials in the home environment: Many children do not experience a print-rich environment at home. It is well-documented that children whose family members read to them and who grow up in homes where reading is valued do better in school. The Snapshot of School Management Effectiveness showed that few grade two and three students read outside the school environment. Only 9.5 percent of students reported using their textbooks at home; an equal percentage reported reading at home every day. Over 52 percent of students reported never reading aloud at home, and 32 percent reported sometimes reading aloud at home. • The lack of parental involvement in student and school performance: While parents reported meeting with teachers at least once a year, teachers largely reported parental involvement to be insufficient. • The language context in Morocco adds further complexity to the learning environment. Classical Arabic is the official language of instruction. However, children’s first exposure to Classical Arabic happens at school. Moroccan students’ mother tongue is either Arabic Dialect (Darija) of one of the Amazigh Dialects. • The EGRA report showed that there are only minimal achievement differences between boys and girls in the early grades. However, gender disparities in learning outcomes seem to progress as students get older and as cultural beliefs about gender roles particularly in rural areas are reinforced. The primary school dropout rate is higher for girls (2.7 percent compared to 1.2 percent for boys in 2012-2013). Girls begin to drop out in the fourth grade (1.8 percent for girls and 1.3 percent for boys), and their rate of dropout increases (9.9 percent for girls and 2.1 percent for boys in grade 6)82.The repetition rate in primary school 80 PNEA Report, May 2009, Higher Council for Education. 81 EGRA/EGMA/SSME Report, RTI, 2011. 82 Ibid. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 76 was 10.5 percent with 12.6 percent for boys and 8.2 percent for girls in 2012-201383. The data show that gender disparities exist, with more boys repeating grades in primary school and more girls permanently leaving school at a young age. Although several factors explain why students drop out, numerous studies84 have found that the most effective way to reduce the number of children who will ultimately drop out is to provide the best possible classroom instruction from the beginning of their school experience through the primary grades. The road to academic success starts early in the education system. Early grade reading proficiency has been cited as indicators of future academic success. • Reading is a foundational skill and the basis of most other academic learning. Thus, it is crucial to ensure that students acquire good reading and comprehension skills in the first three grades so that they have a more equal opportunity to master other skills and knowledge85. The gap between poor and good readers continues to widen as they advance through primary and into middle school. Children who do not read well at the primary level are often on a lifetime trajectory of limited educational progress and therefore limited economic and developmental opportunity. Having all children reading well and completing primary school is foundational for a more equitable society and lower disparities in human and economic development across Morocco. Struggling readers will experience difficulty in later years and, because catching up is nearly impossible without a strong remedial support system, may opt to drop out instead. At the primary level, 90 percent of students who start grade one complete the primary cycle (89 percent for girls)86. However, primary school dropout begins, in earnest, starting at grade four, and proceeds to the point where only 18 percent of a cohort of first graders will reach Grade 12. Despite dropout prevention efforts, over 300,000 children and youth under the age of 15 leave the formal school system every year. Approximately ten percent of primary school enrollment in 2012- 13 were students repeating at least one grade. Two-thirds of primary school repeaters were boys. While the MoE data does not indicate the numbers of students repeating a grade for the second time, officials in the School Dropout Prevention program in the MoE state that students who have repeated two grades or more are twice as likely as one-time repeaters to drop out of school completely. Development Hypothesis (es) There is growing evidence that the quality of service delivered is an important factor in enrolling and keeping children in school. A study87 on grade repetition in primary schools shows that in order to effectively reduce grade repetition, reforms need to tackle the reasons behind poor attendance, the quality and relevance of teaching, and provide sufficient support for teachers. The Reading for Success project supports the GOM Strategy for improving the quality of service delivery to citizens – by improving equity, access and quality of education service delivery by 2016. The Project will enhance educational attainment for children at the primary level in targeted areas in 83 Ibid. 84 Ekstrom et al., 1986: Who drops out of high school and why?; Goldschmidt & Wang, 1999: When can schools affect dropout behavior? ; Rumberger, 1995: Dropping out of middle school: A multilevel analysis of students and schools. American Educational Research Journal, 32, 583-625; Swanson & Schneider, 1999: Students on the move: Residential and educational mobility in America's schools. Sociology of Education, 72, 54-67; Wehlage & Rutter, 1986: Dropping out: How much do schools contribute to the problem? Teachers College Record, 87,374-392 85 Moats, 1999, Teaching Reading is Rocket Science: What Expert Teachers of Reading Should Know and Be Able to Do, American Federation of Teachers. 86 Persistence to last grade of primary is the percentage of children enrolled in the first grade of primary school who eventually reach the last grade of primary education. The estimate is based on the reconstructed cohort method. Source: United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics. 87 Jonathan Penson, “Grade repetition in primary schools in Sub-Saharan Africa: An evidence base for change,” February 2008. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 77 three ways: by supporting the MoE to increase the reading abilities of students enrolled in the early years of primary education; by increasing community support for reading; and by improving the policy environment for reading. The development hypothesis is based on the premise that if USAID supports the improvement of reading skills for students in formal and non-formal primary education, parental and community support for reading, and strengthens the MoE’s capacity and policies to support reading, then student learning outcomes will improve, repetition rates will decrease, and retention rates for students through the primary cycle will increase. These interventions will result in improved overall learning outcome for students in target areas. Helping students in the early grades to read at or above a proficient level will prepare them for future success. Through the provision of technical and material assistance, the project will develop a strong evidence base on successful interventions to improve reading outcomes in Moroccan classrooms. It will support the use of evidence-based reading materials, improved reading instruction, and improved school-based reading enrichment programs. This will be reinforced by increased family engagement in supporting reading and increased civil society organization (CSO) capacity to implement reading activities. The evidence base developed through these activities will inform policy for standards and benchmarks for early grade reading (EGR), define EGR instructional reading time, and establish national assessment protocols for EGR. Description of the Strategy/Project/Activity/Intervention to be Evaluated The Reading For Success (RFS) Project The Reading for Success project is designed to enhance educational attainment for Moroccan children at the primary level. Specifically, the project supports the Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE) in its curriculum reform initiative by developing curricula and instructional reading materials in Arabic for grades 1-4, including students with disabilities; by providing training and coaching to teachers; and by implementing a community mobilization approach tailored to support student reading. In addition to developing and implementing a new reading instruction approach, a human and institutional capacity assessment is also necessary to help identify the Ministry's needs to carry out their reform effectively. The project works with Government of Morocco (GOM) counterparts to develop an evidence base for the design of effective project interventions aimed at improving reading skills, both in targeted primary schools and in non-formal education settings. The project supports a principal objective of USAID/Morocco’s Country Development Cooperation Strategy for FY 2014-2018, “Enhanced educational attainment for children at the primary level,” as well as the Agency-wide Education Strategy. The project comes at a time when the Ministry of Education (MoE) is developing a significant education reform, including a National Strategy for Reading. The Ministry has specifically asked USAID to be its lead international partner to help find effective approaches at the community, school, and classroom level to improve the reading skills of Moroccan children. The project will be implemented over a five-year period (FY 2014-2018) at a total estimated cost of approximately $25 million. The RFS project is composed of four activities (described in more details below). These four activities are designed around three principal streams as outline below. • Improved Reading Skills in grades 1-4 • Increased Community Support for Reading • Improved Policy Environment to Support Reading See Development Objective (DO 3) Logical Framework in Folder 05, REF # 2 1- Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE), implemented by Chemonics: The activity was awarded as a 30-month Task Order (September 23, 2015 to March USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 78 22, 2018) with a total estimated cost of $5,820,752. The activity concluded March 2018. The objective was to support the Ministry in its curriculum reform initiative by developing reading instructional materials in Arabic for grades 1-2; providing training and coaching to teachers; and, implementing a community mobilization approach tailored to support student reading. In its first year, RFS-SSE focused on Result 1 – Developing Arabic Reading Materials for Grades 1 and 2. Based on international best practices and Moroccan expertise, RFS-SSE introduced the phonemic method of teaching, which uses individual syllables and sounds as building blocks for basic reading skills. RFS￾SSE engaged with CSOs through small grants to design and implement summer reading activities. The CSOs engaged students from the target provinces and schools in the reading activities. Chemonics tested via EGRA the impact of these summer programs on reducing students’ reading loss over the summer. 2- Reading for Success - NATIONAL PROGRAM FOR READING (RFS-NPR), implemented by Creative Associates International, Inc. This is a $19.2 million five-year activity (awarded May 19, 2017 through May 18, 2022) which aims to build on the data, evidence and lessons learned from the experimental reading activity (RFS-SSE) to further improve reading instruction and learning materials; expand the implementation of reading instruction practices to grades 3-4; and support the Ministry in developing a new Arabic language curriculum (grades 1-4) that the Ministry will scale up nationwide. The National Program for Reading will: a. Improve reading and writing materials used in primary schools: The materials will be aligned to an updated and improved Early Grade Reading and Writing (EGR/W) curriculum with clear standards, benchmarks and appropriately sequenced content. Textbooks will be aligned with the new EGR/W curriculum and will meet expectations for usability, complexity/readability, content and quality. In addition, the program will support the MoE to develop Arabic Language Curriculum and Pedagogical Guide in the Early Grades and establish Arabic language benchmarks with an objective method for monitoring progress of children throughout the year and governments with a method to communicate performance standards. b. Improve reading and writing instruction in primary schools: NPR is working to develop and deliver an improved training program encompassing extensive preservice and in-service training. c. Expand Reading and writing enrichment programs: In addition to improved reading and writing materials and effective instruction, students need access to reading and writing enrichment opportunities outside of formal school hours and in the home. The core package will offer enrichment reading clubs that meet multiple times a week after school, tutoring, one-on-one support, a buddy system and more. d. Improve learning and assessment systems: to support continuous learning and improvements to the curricula, teaching methods, teacher training, reading and writing materials, and enrichment programs requires a cohesive learning and assessment system. The system will provide Creative, USAID, MoE, teachers, administrators, and others to identify blockages, assess effectiveness, and generate evidenced to design, refine and improve EGRA/W package components. The system will include a research agenda, implementation of a series of evaluations, assessments, analytical tools and participatory workshops. This will allow MoE to institute an adaptive management approach to inform program revision, support scale-up, and create a support policy environment and accountability system. In addition to these goals, and as a cross cutting task, RFS-NPR is building the capacity of the Ministry of Education for sustained implementation. Local capacity-building at every level is critical for program implementation, and to sustain the ongoing use of evidence-based approaches after the activity ends. Program implementation is designed to support the role of the Ministry of Education and build its capacity at the central and provincial levels, employing innovative and practical modalities, strategies and activities that will reinforce teacher knowledge and practices that ensure the success of the program beyond the life of activity. Capacity-building, which is one of the USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 79 indispensable elements for ensuring country ownership and responsibility, can be only reached if the activity is carried out hand-in-hand with MoE key officials at all levels. In this sense, RFS-NPR capacity-building approach goes beyond simple organization of and participation in capacity-building workshops and trainings. RFS-NPR is seeking ways of providing the MoE with capacity-building opportunities in real settings and in a practical and comprehensive manner. RFS-NPR will develop a milestone plan with the MoE and use this, alongside regular feedback loops, to ensure the effective delivery of this component and sustainability of results. In addition to hands on learning, RFS-NPR may include partnerships with external entities, such as national and international universities, to provide MoE officials and technical staff with focused and intensive training (including training with certification) in key areas such as early grade reading and child literacy, curriculum development, teacher training, coaching, assessment, and other. This evaluation is expected to coincide with the last quarter of the second year of NPR implementation (Jan-Feb 2019). By then, most of the instructional materials would have been designed, training of trainers would have taken place, and Grade 1-4 curricula would have been tested and implemented in the experimental schools. The detailed workplan of NPR will be provided for reference. Quarterly and Annual reports will be shared with the winning contractor as they become available. 3- Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): This activity was awarded to Chemonics in October 2015, with a total estimated cost of $1,563,935 and a revised end date of October 2018. The objective of this activity is to improve the availability and quality of reading materials and instruction available for deaf children in Grades 1-3, and ultimately to improve their literacy through: the development of software that helps teachers and parents easily create and publish Moroccan Sign Language￾supported educational materials; training of teachers; and assessment (Early Grade Reading Sign Language Assessment, EGR-SLA). A cost extension of the Cooperative Agreement was signed in April 2017 extending the project until October 2018 and increasing the budget to accommodate the added activities. These activities primarily cover another year of EGRA-SLA testing, more teacher training, and the expansion of MSL Clip and Create to include more functionality and database expansion, including Chemonics’ first and second grade national curriculum words, and regional variations. 4- Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): This $2,112,724 award to The Kaizen Company aimed to ensure that USAID and the Ministry have a full understanding of the human and institutional challenges that may hinder the implementation of the Ministry’s reform efforts, and that the Ministry would receive a package of actionable solutions to face identified challenges. The purpose of the RFS-HICD Analysis of Morocco’s MoE was to support the implementation of the 15-year reform entitled “Vision 2015-2030”, especially the components that will be supported by both USAID/Morocco and the Millennium Challenge Corporation (MCC). The activity was procured in June 2016 and concluded June 30, 2017. Activity Monitoring, Evaluation, and Learning (MEL) Plans Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE) (Reference: FOLDER 01 RFS-SSE, SUB-FOLDER 01.2, REF# 1): The purpose of this MEL Plan is to collect data, and assess accurately the progress of RFS-SSE towards its goals. The RFS-SSE team uses the collected data to measure the success of project efforts to identify first and second graders’ reading competencies via EGRA and Snapshot of School Management Effectiveness tools in order to provide data that will assist in identifying interventions to improve reading skills as well policy environment to support reading. The MEL Plan describes the RFS-SSE project methodology for activity monitoring and evaluation and is used to manage the collection of performance data. The MEL Plan includes a performance indicator summary table (pages 16-22) that includes performance indicators, data sources and collection methods, timing of data collection, disaggregation, and units of measure. The USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 80 results are used to regularly update and engage relevant stakeholders, including USAID, to analyze progress and quality. In addition to the MEL Plan, both DevResults (Chemonics) and TraiNet (USAID) are used to store the collected data. The DevResults Monitoring and Evaluation (M&E) software are used for the project’ M&E data collection, storage, management, and reporting. Its web-based interface can be used from any internet-enabled device (tablet, mobile phone, desktop). Where internet access is unavailable or inconsistent, data can be captured via formatted Excel spreadsheets and other offline tools and later uploaded. Features include Geographic Information System (GIS) functions, data visualizations, results dashboards, and document and photo storage capabilities. All documents will be provided upon award by the Education Office and Chemonics. Reading for Success - NATIONAL PROGRAM FOR READING (RFS-NPR): RFS-NPR employs several highly consultative and participatory research and evaluation methods to test activities and support scale-up. The “stepped” approach (1) tests the effectiveness and viability of the EGR/W instructional and enrichment interventions in schools; (2) provides the opportunity to identify and alleviate policy, procedural, and resource blockages in the delivery and support/ supervision systems at local, provincial, and regional levels; and (3) supports the MoE roll-out of its EGR/W program nationally. With the Assessment Quality Assurance Working Group (AQA-WG), led by CNEE, Creative is developing a research agenda, reviewing SSE results, detailing key research questions and identifying the types of studies required. The agenda includes methods, sampling, and periodicity. The research agenda should be updated annually to reflect emerging areas of inquiry. Some of the research questions that RFS-NPR may address include: • What are the major policy factors that significantly contribute to or inhibit the project effectiveness in the Morocco context? • What should successful teacher profile for teaching reading look like and how should teachers be supported in terms of training, in-school support and community support? • Which one of the four enrichment models that significantly contribute to student reading outcomes and to what extent? • What types of correlations exist between teachers’ classroom practices and student reading outcomes and to what extent? The performance indicator summary table is available in (Reference: FOLDER 02, SUB-FOLDER 02.2, REF # 1) and outlines performance indicators, data sources and collection methods, timing of data collection, disaggregation, and units of measure. Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): The proposal for Improving Deaf Children’s Reading Through Technology (IDCRT) was written using a logic model structure. It contains hypotheses, goals, objectives, tasks, inputs, outputs and outcomes. These are incorporated both into the body of the proposal and into the Life of Activity Performance Plan. The purpose of this structure is to clearly set the course of the project’s implementation, explicitly define those measures and products that provide evidence of completion and quality, and to monitor progress on an ongoing basis so as to provide for timely corrective actions and sustaining efforts that bolster positive outcomes. Monitoring of project implementation is specifically addressed within Objective 12 of IDRT’s workplan, is highly structured, and has a number of measures incorporated to facilitate month-by￾month analysis and assertion of compliance. Each of the 48 tasks is tethered to the timeline, so that at any given moment it can be ascertained whether the project is staying on task and on time. (Reference: FOLDER 03, SUB-FOLDER 03.2 REF # 1 and 2) Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): The Monitoring Evaluation & Learning System (MELS) provided the overall framework for Monitoring and Evaluation (M&E) and Learning in the Morocco HICD Assessment Activity. It served as the basis for generating, analyzing and disseminating a basic set of information USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 81 useful to set broad activity targets and to chart the progress of Morocco HICD in the attainment of these. (Reference: FOLDER 04, SUB-FOLDER 04.2, REF # 1) C.4 EVALUATION QUESTIONS The evaluation questions for the RFS project evaluation below may be further developed and finalized in collaboration with USAID prior to the finalization of the evaluation design. The Evaluator will have access to student reading performance data as well as performance monitoring data for all activities collected by the Implementing Partner (IP) (as described in above). For the ongoing RFS-NPR, EGRA baseline data was collected in May 2018 and raw data as well as analysis should be readily available to the Evaluator. Midline and endline data will not be available as midline data is expected to be collected in May 2020, and endline data in May 2021. The Evaluator will be expected to use the IP’s monitoring data (student reading performance); and will not be expected to collect primary reading assessment data. Additional sets of quantitative data may be available through the host country Ministry of Education website, and could be useful to analyze by the evaluation team: https://www.men.gov.ma/en/Pages/Statistics.aspx. The Evaluator is expected to disaggregate results by gender, as relevant and whenever possible. As relevant and necessary, the Evaluator should consider and suggest other variables to disaggregate the data by, such as by milieu (rural vs. urban) and/or by activity. This evaluation will focus on outcomes in classical Arabic only and will not look at any aspects related to the use of mother tongue or other languages for instruction. RFS Project Draft Evaluation Questions Note: Evaluation questions are separated into two categories; (1) project implementation and management and (2) lessons learned and sustainability, which are described in more detail below. Offerors should not consider these categories or the order of the questions as a prioritization and give each category and questions equal weighting. Project Implementation and Management: Currently, the IPs for the activities under this project are addressing related questions using the EGRA data and are looking at possible correlation of observed factors that may influence performance of students. Using robust, representative qualitative data, the evaluation questions under this dimension aim to enrich and deepen the IPs’ analysis and results by helping to shed light on what and how factors affect achievement of educational outcomes outlined in the project. 1. To what extent have the activities under the RFS project contributed to the overall goal of the RFS project of enhancing educational attainment for Moroccan children at the primary level? a. How and why did students under the RFS-SSE, RFS-IDCRT, and RFS-NPR activity realize the observed learning results? (In addressing this question, discuss the strengths and the gaps of the project activities in contributing to the objectives of RFS.) b. How did RFS activities intersect with, provide learning about, and result in progress towards each of the 5 Ts of reading instruction? c. How have the activities performed against the higher-level indicators at the project level? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 82 d. To what extent have the activities effectively used (or are currently using) activity￾level M&E data to make necessary and relevant adjustments to program design, implementation and management? 2. How have the structure and management of the project and its activities facilitated or hampered the realization of the project goal? 3. Which populations of children currently in (or not in) the educational system are not benefiting from the current RFS project activities and how can they be better served in future projects/activities? Lessons Learned and Sustainability: Sustainability is a key component of success for the RFS project as many of the activities will be managed and implemented by the GOM moving forward. The following questions seek to better understand whether the project has contributed to sustainable outcomes to date and inform any future programming changes. 4. To what extent (and how) has the Ministry taken over responsibilities of project activities and ownership of the national system for reading and literacy? 5. How did the MoE’s lead role in the reading and literacy project activities influence the implementation, management, and results of the activities? 6. What key strengths or constraints affected the project activities’ ability to contribute to establishing a sustainable national system for reading and literacy? 7. What key lessons and recommendations in reading and literacy can be drawn from the successes and limitations of the RFS project in order to more effectively achieve the Education Development Objective (Enhanced Educational Attainment for Children at the Primary Level) as well as help inform Morocco's new CDCS? C.5 EVALUATION APPROACH In the subsections below, general guidance is provided on methodology, selection of respondents/sampling, data sources and collection, data analysis, and methodological limitations. In their responses, offerors should propose a specific evaluation strategy for each evaluation that specifically addresses the needs and requirements of each evaluation. Evaluation Design: The Evaluator is expected to employ a primarily qualitative evaluation approach to address the research questions outlined above. Whenever feasible and relevant, the Evaluator should consider and suggest the use of quantitative data to help strength and enrich the qualitative findings. The overarching analytical strategy should be clearly summarized in an evaluation matrix that includes the following information: evaluation questions, sub-evaluation questions, indicators or assessment criteria, sampling/selection criteria, data sources and collection methods, and data analysis methods. The evaluation team, in collaboration with USAID, will finalize the overall evaluation methodology for the evaluation before fieldwork begins. Evaluation Design Workshop: An evaluation design workshop is a key component of the evaluation and must be conducted as part of the evaluation design component. It is considered a best practice in the field of evaluation and is highly recommended. The purpose of such a workshop is to help further develop/ finalize the evaluation design plan and ground the evaluation to the context and realities on the ground. During the workshop, questions will be refined and finalized, evaluation approach will be further refined/finalized based on the realities of the implementation of the activities, potential issues related to sampling, data sources, timeline are discussed and addressed. It must be led in country by the Evaluation team, and should bring together relevant Mission staff, IPs, and as appropriate, host-country/ministry stakeholders. The Evidence Team in the E3/ED office of USAID/Washington will provide technical support and will help facilitate the workshop. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 83 Selection of Respondents/Sampling Strategy: The Evaluator should carry out relevant data collection with a representative sample of purposively selected USAID project beneficiaries, partners, and host government counterparts at appropriate levels; and other key program stakeholders as appropriate. The Evaluator should propose a robust strategy for selecting respondents that will enable adequate geographical, language representation; cross-site comparison; and the capture a diversity of perspectives, among other key considerations. Final criteria for sample selection should be developed by the Evaluator in consultation with USAID. Data sources and collection methods: The Evaluator must collect both primary data and secondary data. Primary data may include, but is not limited to key informant interviews, focus group discussions, observations, and questionnaires. For secondary data, and whenever relevant, the Evaluator must consider complementary sources of secondary data, such as the IP’s performance monitoring data, activity documents, MoE national statistics88, etc. Prior to any field work, the Evaluator is expected to carry out a targeted review and analysis of recent pertinent published literature on educational program evaluations in Morocco and MENA as well as relevant project documents and performance data provided by USAID to: 1) gain a better and more in-depth understanding of the implementation of the RFS project and, 2) assess the quality and relevance of the data to help answer the evaluation questions. Primary data should only be collected if the data does not already exist via the IP’s monitoring data, school administrative data, census data, etc. The Evaluator is expected to collect at least one round of data collection and should outline a clear plan and specify which data exactly would be collected and why. The Evaluator should use, whenever possible, pre-existing data collection instruments with documented strong measurement validity and internal reliability and with a suitable plan for cultural adaptation. For new instruments, the Evaluator should propose a plan for the development, cognitive testing (to test reliability and validity of instruments), and piloting of the new data collection instruments. It should be noted that: • Prior to beginning field work (including field testing), the evaluation team is required to share data collection instruments with USAID for review and feedback before they are used in the field. • Issues of data confidentiality must be addressed as part of the IRB process. • The use of electronic data collection for quantitative data is required, whenever relevant. • Data collection methods must consider contextual factors to maximize data reliability and minimize unintended consequences. Data Analysis: The Evaluator must analyze both primary and secondary data. All secondary data must be assessed for quality and relevance in answering the research questions. The Evaluator must provide a data analysis plan for the qualitative data, and for the quantitative data (if quantitative data is used) that directly addresses each evaluation question with specific methods for analyzing the data. As part of the plan, the Evaluator should specifically discuss how the qualitative and quantitative data will be weighted and integrated (whenever relevant) to address the evaluation questions. Data should be triangulated across multiple sources and data triangulation should be evident in the evaluation findings. Methodological limitations: As part of USAID’s evaluation policy, the Evaluator is required to discuss and document any issues potentially affecting the quality of evaluation data (including the data validity, integrity, timeliness, precision, and reliability) in the evaluation planning stage, and to be assessed on an ongoing basis during implementation. The Evaluation team and USAID should discuss all limitations and measures to address or overcome limitations in the implementation phase and 88 https://www.men.gov.ma/en/Pages/Statistics.aspx. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 84 these should be detailed in the reports. The evaluation reports must be clear and transparent about any notable limitations and if and how they may affect the evaluation findings, conclusions, and recommendations. [END OF SECTION C] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 85 SECTION F - DELIVERIES OR PERFORMANCE F.1 REPORTS/DELIVERABLES All reports and deliverables under this Contract must be submitted to USAID at the times indicated in the table below. The Contractor must, however, promptly notify the Contracting Officer’s Representative (COR) of any problems, delays or adverse conditions which materially impair the Contractor’s ability to meet the requirements of the contract. The Contractor must follow the language requirements listed in the table below, as needed. Required reports and deliverables are described as follows: A. Schedule of reports and deliverables for RFS project performance evaluation 1. Draft Work Plan: Contractor must submit a draft workplan within two weeks of award that describes in detail how the team intends to implement all tasks specified in the contract and provide a detailed timeline for the completion of all tasks. The workplan must include the anticipated schedule and logistical arrangements and delineate roles and responsibilities of members of the evaluation team. 2. Consultations & desk review report: Within three weeks after award, the contractor must discuss the draft work plan and present a summary of the desk review report. The discussion can be held by conference call. 3. Evaluation Design Report: The draft Evaluation Design Report (EDR) must be submitted within five weeks of award. The EDR must describe the overarching analytical strategy. The report must incorporate the final work plan and desk review report (deliverables 1 and 2 above) and include at minimum the following information: evaluation questions, sub￾evaluation questions, indicators or assessment criteria, sampling/selection criteria, data sources and data collection instruments as well as methods, and data analysis and triangulation methods. A separate summary of the report must be provided in French (to be shared with the Ministry of Education). 4. Evaluation Design Validation Workshop89: An evaluation design workshop is a key component of the evaluation and must be conducted by the evaluation team in Morocco within seven weeks of award. Feedback provided during the workshop is to be incorporated into the final EDR and must be approved by USAID before the evaluation team begins actual field work (i.e. data collection). The purpose of such a workshop is to help further develop/finalize the evaluation design plan and ground the evaluation to the context and realities on the ground. During the workshop, questions will be refined and finalized, evaluation approach will be further refined/finalized based on the realities of the implementation of the activities, potential issues related to sampling, data sources, timeline are discussed and addressed. It must be led in-country by the Evaluation team, and should bring together relevant Mission staff, IPs, and as appropriate host-country/ministry stakeholders. 5. Interim-Briefing: Within 14 weeks of award (and following approximately eight weeks of field work), the contractor must have completed the data collection process of the RFS project performance evaluation. The evaluation team will hold an interim-briefing meeting, in Rabat, with USAID to present, in PowerPoint format, a brief overview of the preliminary findings and confirm completion of data collection. This briefing may include IPs or other stakeholders, as determined by USAID. 6. Draft evaluation report: Within 20 weeks of award (approximately six weeks after completion of data collection), Contractor must submit a draft evaluation report for the RFS 89 The Evidence Team in the E3/ED office of USAID/Washington will provide technical support and will help facilitate the workshop. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 86 project performance evaluation. The draft must be in English and must be submitted in electronic copy to the COR for review. 7. Presentations of findings & Infographic: Within 24 weeks of award, the evaluation team must make a presentation in Morocco of the evaluation findings to USAID and provide a two-page (i.e. front and back of one piece of paper) infographic. After discussion of results and recommendations, any changes suggested by USAID must be included in the final report and presentation and infographic. An Arabic version of the presentation as well as a French and Arabic version of the infographic must be prepared and approved by USAID. The presentation will be presented to the Ministry of Education officials at a date and time agreed on with USAID and MoE. 8. Final report in English, with French and Arabic Executive Summaries: The contractor must submit the evaluation final report within 27 weeks of award. The report must include an executive summary; introduction; background of the local context and the projects being evaluated; the evaluation questions; the methodology; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable). The executive summary must be less than five pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned. Submission of draft French and Arabic executive summaries will take place only after the COR has approved the final English version of the executive summary. Deliverables Table Key Deliverables Due Date for Submission 1. Draft Work Plan Within week 2 of award 2. Consultations & desk review report Within week 3 of award 3. Evaluation Design Report (EDR) Within week 5 of award 4. Evaluation Design Validation Workshop Within week 7 of award 5. Interim-Briefing Within week 14 of award 6. Draft evaluation report Within week 20 of award 7. Presentations of findings & Infographic Within week 24 of award 8. Final report in English, with French and Arabic Executive Summaries Within week 27 of award USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 87 Deliverables Timeline F.2 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and ADS 201, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 1. The evaluation report must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why. 2. Evaluation reports shall address all evaluation questions included in the scope of work. 3. The evaluation report must include the scope of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline must be approved by the Contracting Officer and agreed upon beforehand by the COR. 4. The evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an Annex in the final report. 5. Evaluation findings must assess outcomes and impact on males and females. 6. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 7. Evaluation findings must be presented as analyzed facts, evidence, and data, and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence. 8. Sources of information need to be properly identified and listed in an annex. 9. Recommendations need to be supported by a specific set of findings. 10. Recommendations must be action-oriented, practical, and specific, with defined responsibility for the action. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 88 11. Recommendations must be detailed separate from findings and conclusions. For further guidance on USAID quality evaluation policies, please visit http://www.usaid.gov/evaluation. F.3 OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, nonproprietary formats as required by USAID’s Open Data policy (see ADS 579). The data must be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions must be updated in the SOW that is included as an annex to the Evaluation Report. F.4 EVALUATION TEAM COMPOSITION The Evaluator must propose a staffing plan that fully supports the requirements of this SOW. At minimum, the staffing plan must: (1) list the proposed team members and clearly articulate each team member’s responsibilities within the team and their relevant qualifications to meet these responsibilities (2) describe how the team members will work together, and their anticipated LOE for the duration of the evaluation activities. As part of this response, the evaluation team will need to complete the table below and provide CVs and references for all team members proposed. Evaluation Team Experience Team Member Name Title Team Member Responsibilities Team Member Relevant Experience Written disclosures of conflicts of interest are required from all evaluation team members. Key Personnel a) The key personnel that the Contractor must furnish for the performance of this contract is as follows: Position Title Name 1. Evaluation Team Leader TBD 2. Education Evaluation Specialist TBD b) The key personnel specified in this Contract are considered to be essential to the work being performed. Prior to replacing the above specified individuals, the Contractor must notify both the CO and the COR reasonably in advance and must submit written justification (including proposed substitutions) in sufficient detail to permit evaluation of the impact on the program. No replacement will be made by the Contractor without the written consent of the CO. c) Minimum qualifications for the key personnel are as follows: Evaluation Team Leader: USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 89 Role: The Evaluation Team Leader will be responsible for overall and day-to-day management of contract activities, involving multiple tasks across multiple locations. S/he is the principal interlocutor between the Contractor team members and USAID and is responsible for the overall compilation of all deliverables under this contract. The Team Leader is expected to be the technical leader for the evaluation and therefore is expected to be involved in the dissemination of findings. S/he is responsible for staffing, project planning, project financials, and staff direction and oversight, and maintains and manages the relationship with USAID. S/he collaborates closely with technical evaluation staff to ensure quality of design, methodologies, tools, data collection processes and data analysis. Minimum requirements: • Graduate degree or higher in international development or related field. • Minimum seven years of experience conducting qualitative research. • Education sector expertise required, with experience in Morocco and/or MENA region preferred. • Has led previous education evaluations. • Strong background and training in qualitative research methodologies • Experience and knowledge of data analysis and data management techniques • Experience managing and coordinating complex evaluations, including evaluations of international projects. • Strong organizational, oral and written communication skills. • Proven ability in writing evaluation reports, and ability to use research findings to evaluate program implementation and operations. • Evaluation Team Leader must be fluent in English, spoken and written. • Candidates speaking French and/or Arabic in addition to English preferred Education Evaluation Expert: Role: The Senior Evaluation Expert will be responsible for supervising the quality of the evaluation design and execution, including qualitative data collection and analysis, identifying and disseminating of findings, lessons learned, and evidence from evaluations. Minimum requirements: • Graduate degree in sociology, anthropology, education or related field with relevant experience in ethnographic research preferred. • Minimum seven years of experience managing and conducting qualitative and mixed methods evaluations. • Strong background in qualitative research methodologies. • Experience managing or conducting evaluation of education and early grade reading programs (in the Moroccan context preferred) • Experience conducting site visits, key informant interviews, and focus group discussions • Experience in successful dissemination, such as writing evaluation reports, delivering presentations and providing consultation to a diverse range of stakeholders, and effectively tailoring messages to the audience. • Candidates speaking French and/or Arabic in addition to English preferred D. Overall the Evaluator will ensure that: • All evaluation team members provide signed statements attesting to a lack of conflict of interest or describing any existing conflict of interest. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 90 • The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 201. F.5 TECHNICAL APPROVAL REQUIREMENTS The Contractor is required to seek the following approvals/Technical concurrence from the COR: • Approval of reports described in Section F.4 and F.5. • Technical concurrence on subcontracts, key personnel and other approvals submitted to the CO • Approval of international travel not explicitly identified in the approved annual work-plan • Events, workshops, stakeholder meetings not explicitly identified in the approved annual work-plan [END OF SECTION F] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 91 ANNEX II: EVALUATION METHODS AND LIMITATIONS This section describes the approach as planned in the study’s original evaluation design report. EVALUATION QUESTIONS Six evaluation questions and their related sub-questions will guide the RFS performance evaluation. After an initial set of questions was drafted for the Project Appraisal Document (PAD) Evaluation Statement of Work, they were revised in collaboration with USAID/Morocco and MoE partners at an evaluation design workshop held in Rabat and concurrent interviews with RFS IPs. This independent, whole-of-project performance evaluation will answer questions that each address a separate theme—project outcomes, implementation, inclusion, sustainability, and lessons learned: Project Outcomes 1. To what extent have the activities under the RFS project contributed to the overall development objective of the RFS project of enhancing educational attainment for Moroccan children at the primary level? 1.1. How and why did students under the RFS-SSE, RFS-IDCRT, and RFS-NPR activities realize the observed learning results? 1.2. To what extent did RFS activities contribute to progress towards each of the 5 Ts of reading instruction?90 1.3. How have the activities performed against the higher-level indicators91 at the project level? Implementation 2. How and why did factors impede and/or facilitate achievement of the RFS project's development objective? 2.1. How and why did factors impede the achievement of the project's development objective? 2.2. How and why did factors facilitate achievement of the project's development objective? 2.3. To what extent have the activities responsively used (or are currently using) activity-level M&E data to make necessary and relevant adjustments to program design, implementation, and management? Inclusion 3. Which primary-school-aged populations of children are not benefiting from the current RFS project activities, and how can they be better served in future projects/activities? (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) 90 USAID’s approach to improving reading and literacy revolves around five goals also knowns at the 5 Ts. According to USAID these 5 Ts are the key to reading success. The 5 Ts are: (1) more time devoted to teaching reading, (2) better techniques for teaching reading, (3) more texts in the hands of children, (4) teaching children in the mother tongue (a language they speak and understand), and (5) testing children’s reading progress. 91 These indicators are defined as: 3.b Percentage of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with United States Government (USG) assistance 3.1.a Number of learners reached in reading programs at the primary level 3.1.b Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance 3.2.a Increased provision of reading activities outside of instructional time (disaggregated by public / private) 3.3.a Number of laws, policies, regulations or guidelines developed or modified to improve primary reading programs USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 92 3.1. Which primary-school-aged populations of children (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) in schools targeted by RFS are not benefiting from the current project activities? 3.2. What are the barriers to reaching those students who did not benefit from RFS and measures by which these students can be better served in future projects or activities? Implementation / Sustainability 4. What was the Ministry's role in leading project activities, and how did this role influence the implementation, management, and results of the activities? 4.1. What was the Ministry's role in leading RFS project activities? 4.2. How did the Ministry's role influence the implementation, management, and results of the activities? 4.3. How did the role of the Ministry vary in implementation, management, and results at the central, provincial, regional, and local levels? Sustainability 5. What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? Lessons Learned 6. What key preliminary lessons learned and recommendations can be gleaned from implementation of the program to date? METHODOLOGY In order to answer the six primary RFS performance evaluation questions and related sub-questions, the evaluators will primarily use a qualitative research approach.92 The team will conduct field work over the course of three weeks in September and October 2019, collecting data from a variety of RFS stakeholder respondent groups in order to gain understanding of multiple perspectives/experiences across a variety of levels (national, regional, provincial, and school/community) and to enable triangulation of data and findings. Triangulation will also include using both qualitative and quantitative methods to gain a more holistic and deeper understanding of the RFS project as well as mitigate potential bias in the evaluation data and findings. Moreover, while the research will focus primarily on the regions and provinces where RFS activities have been implemented—namely, the RFS-SSE and RFS-NPR pilot regions, provinces, and schools—key MoE stakeholders from non-pilot regions and provinces will also be sampled to gain additional perspectives on the broader scale-up. Qualitative data collection methods such as semi-structured key information interviews (KIIs) will capture a variety of views, experiences, and perspectives from key stakeholders at national, regional, provincial, and school levels. In addition, focus group discussions (FGDs) with beneficiaries at the school and community levels will be conducted through school/learning site visits across multiple regions and provinces. Where possible, existing RFS project data (including EGRA results and other 92 Please note that the current assumption is that the proposed team will conduct all qualitative data collection and analysis and will not use local a data collection firm. For the purposes of budgeting, it is therefore assumed that all quantitative data will be drawn from secondary, existing data and reports. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 93 monitoring and evaluation data) as well as administrative and/or performance data from MASSAR— will be analyzed to supplement qualitative findings using the approach described in Figure 2. This methodology is informed by the RFS evaluation document review, evaluation design workshop, and stakeholder mapping exercise. Figure 2. Triangulation Using Parallel Combinations for Mixed-Methods Evaluation93 The qualitative and quantitative analyses proposed in this design report will employ a “parallel combinations” analytical approach, illustrated below, through which each method is applied separately from the other and integrated or triangulated following independent analysis: As appropriate, a “sequential combinations” approach will also be employed, enabling findings from qualitative field work to inform the design and implementation of secondary quantitative analysis. To support this purpose, qualitative data collection tools will be structured so that the research team can capture information from key stakeholders on relevant and available secondary data sources that could further shed light on the evaluation question(s) being examined. The evaluators will integrate a gender-sensitive evaluation approach, drawing on resources such as USAID’s How-To Notes on Engendering Evaluation and Gender Integration in Education Programming and USAID’s Gender-sensitive Evaluation: Best and Promising Practices for Engendering Evaluation. This approach will include ensuring all data are disaggregated and reported by sex where appropriate, engaging female and male researchers, and striving for gender balance among respondents. Gender frameworks will be considered in the analyses as well, such examining how the interventions may have affected boys and girls differently. The evaluators will also incorporate social inclusion in their evaluation approach, including consideration of geographical or regional variations, ethnicity, language, socio-economic status, disability status, and age in the design, sampling, and analytic approaches. As such, the evaluation team will include members with prior experience with deaf education in Morocco and plans to engage sign language interpreters and minority language translators as needed. Geographic and regional variations in RFS target zones informs the evaluation’s stratified sampling approach across pilot regions and provinces. These areas also include variations in language groups, including both Arabic and Amazigh languages. The range of respondents, including high-level government officials and teachers and parents from purposively selected rural disadvantaged areas, incorporates an array of socioeconomic statuses. Individuals with disabilities will be purposively selected as respondents 93 Adapted from USAID Technical Note on Conducting Mixed Methods Evaluation (2013). USAID Bureau of Policy, Planning and Learning. Retrieved from https://www.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations_Technical_Note.pdf. Synthesis or triangulation Qualitative data collection Qualitative data analysis Quantitative data analysis USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 94 for the RFS-IDCRT activity in particular. Age bracket data will be collected from respondents to enable disaggregation of data and inform analysis. This overall approach is outlined in the performance evaluation matrix summary in Table 7, followed by sections detailing the sampling strategy, data sources and collection methods, and analysis plan. In the matrix, a preliminary list of associated data sources, sampling/selection criteria, and data collection methods has been identified for each evaluation question and sub-question. This matrix was informed by the document review and stakeholder mapping conducted in Morocco in May 2019. This evaluation matrix will form the basis of a more detailed data collection tool map to be developed prior to field work. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 95 Table 7. RFS Performance Evaluation Matrix # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods Project Outcomes 1 To what extent have the activities under the RFS project contributed to the overall development objective of the RFS project of enhancing educational attainment for Moroccan children at the primary level? USAID/Morocco XXXXXXXXXXXX KII Qualitative data and debriefing notes will be captured mainly through typed up FGD and KII field notes (in French or English). Electronic field notes will be imported into NVivo for first￾order and second-order qualitative data analysis. This analysis will then be examined in relation to the research questions, the literature review, and existing quantitative data to develop key findings. MoE central office officials XXXXXXXXXXXXXXXX X XXXXXXXXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXX KII School Directors XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX XXXXXXXXX KII MoE AREF/Regional Officials XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX KII MoE Provincial Officials XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXX KII CRMEF XXXXXXXXXXXXXXXX XXXXXX KII Teachers XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX FGD Parents XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX FGD NPR Baseline EGRA XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX Secondary data Treatment and control contrast using multivariate regression analysis to assess SSE’s medium-term impact on pupil reading performance following two full years of exposure to the SSE program USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 96 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods MASSAR data on repetition/dropouts as well as pupil and teacher characteristics XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXX Secondary data Pre/post-test using multivariate regression analysis—controlling for time invariant teacher and pupil characteristics—to assess the extent to which the NPR grade 1 scale-up affected pupil repetitions or dropouts 1.1 How and why did students under the RFS-SSE, RFS￾IDCRT, and RFS￾NPR activity realize the observed learning results? USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 Existing project data XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX Document review Content analysis to summarize IP results related to research questions 1.2 To what extent did RFS activities contribute to progress towards each of the 5 Ts of reading instruction? USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX FGDs See Qualitative Data Analysis Methods from Q1 Existing project data XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX Document review Content analysis to summarize IP results related to research question 1.3 How have the activities performed against the higher￾level indicators at the project level? USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 USAID/Morocco XXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 97 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods Existing project data XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX Document review Content analysis to summarize IP results related to research question Implementation 2 How and why did factors impede and/or facilitate achievement of the RFS project's development objective? USAID/Morocco XXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 USAID IP subcontractors XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE central office officials XXXXXXXXXXXXXXXX XXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX XXX XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Inspectors/Provincia l Coordinators XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 MoE AREF/Regional officials XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 School Directors XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXX XXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 Publishers XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 98 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods NPR Baseline EGRA and SSE Midline 1 EGRA XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX Secondary data T-test measuring statistical improvements in reading performance between a cross-section of Grade 1 SSE pupils in May 2016 and May 2018. While SSE was in effect for grade 1 in May 2016, it had only been operating for four months therefore a comparison after two years could offer insights on implementation efficacy over the longer term. 2.1 How and why did factors impede the achievement of the project's development objective? see Q2 (same data sources) XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 2.2 How and why did factors facilitate achievement of the project's development objective? see Q2 (same data sources) XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 2.3 To what extent have the activities responsively used (or are currently using) activity-level M&E data to make necessary and relevant adjustments to program design, implementation and management? USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX X KII See Qualitative Data Analysis Methods from Q1 USAID IP subcontractors XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Inclusion 3 Which primary school aged populations of children are not benefiting from the current RFS project activities and how can they be better MoE central office officials XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Other Government Officials XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX KII See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 99 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods served in future projects/activities? (considering gender, students who are disadvantaged/margi nal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) Teachers XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX X FGD See Qualitative Data Analysis Methods from Q1 Parents of students with special needs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 CRMEF XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 USAID IPs XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID IP subcontractors XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 AREF XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Disabled persons organizations XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 Education Committee Members of Parliament or Staff XXXXXXXXXXXXXXXX XX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 100 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods 3.1 Which primary school aged populations of children (considering gender, students who are disadvantaged/margi nal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) currently in the educational system and being reached by RFS are not benefiting from the current project activities? see Q3 (same data sources) XXXXXXXXXXXXXXXX XXXXX see Q3 (same collection methods) See Qualitative Data Analysis Methods from Q1 3.2 What are the barriers to reaching those students who did not benefit from RFS, and measures by which these students can be better served in future projects or activities? see Q3 (same data sources) XXXXXXXXXXXXXXXX XXXXX see Q3 (same collection methods) See Qualitative Data Analysis Methods from Q1 Implementation/Sustainability 4 What was the Ministry's role in leading project activities, and how did this role influence the implementation, management, and results of the activities? MoE central office officials XXXXXXXXXXXXXXXX X XXXXXXXXXXXXXXXX XXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 USAID IPs XXXXXXXXXXXXXXXX XX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX XXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID/Morocco XXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 4.1 What was the Ministry's role in leading RFS project activities? see Q4 (same data sources) XXXXXXXXXXXXXXXX XXXXX XXXXX see Q4 (same collection methods) See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 101 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods 4.2 How did the Ministry's role influence the implementation, management, and results of the activities? see Q4 (same data sources) XXXXXXXXXXXXXXXX XXXXX XXXXX see Q4 (same collection methods) See Qualitative Data Analysis Methods from Q1 4.3 How did the role of the Ministry vary in implementation, management, and results at the central, provincial, regional, and local levels? MoE AREF/Regional officials XXXXXXXXXXXXXXXX XXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE Provincial Officials XXXXXXXXXXXXXXXX XXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX XXXXXXXXXXXXXXXX X KII See Qualitative Data Analysis Methods from Q1 School Directors XXXXXXXXXXXXXXXX XXXX XXXXXXXXXXXXXX XXXXXXXXX XXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE central office officials XXXXXXXXXXXXXXXX X XXXXXXXXXXXXXXXX XXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX XXX XXXXXXXXXXXXXXXX XXXXXXXXX XXXXXXXXXXXXXXXX XXX XXXXXXXXXXXXXXXX X KII See Qualitative Data Analysis Methods from Q1 CRMEF XXXXXXXXXXXXXXXX XXXXX KII See Qualitative Data Analysis Methods from Q1 Inspectors/Provincia l Coordinators XXXXXXXXXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX XXXXXXXXXX XXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXX X XXXXXXXXXXXXXXXX XXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX FGD See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 102 # Evaluation Questions Data Source Sampling and Selection Criteria Collection Methods Data Analysis Methods Sustainability 5 What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? see Q2 (same data sources) XXXXXXXXXXXXXXXX XXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 NPR Baseline EGRA XXXXXXXXXXXXXXXX XX XXXXXXXXXXXXX Secondary data Statistical contrast of performance at SSE “high touch” schools with performance at NPR scale-up “low touch” schools at single cross section. This analysis may provide evidence for whether observed results can be sustained at scale. Lessons Learned 6 What key preliminary lessons learned and recommendations can be gleaned from implementation of the program to date? see Q2 (same data sources) XXXXXXXXXXXXXXXX XX XXXXXXXXXXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 103 SAMPLING STRATEGY/SELECTION OF RESPONDENTS Primary respondents sampled for this evaluation will include school-level respondents; MoE and non￾MoE government officials at the provincial, regional, and national levels; personnel of USAID/Morocco and RFS IPs; and other RFS partner organizations. Evaluators primarily sampled respondents from the geographic areas where RFS has targeted its interventions. These areas include eight provinces in four regions that have participated in both RFS-SSE and RFS-NPR, as these are the respondents who will be best able to speak to RFS. Details about each of these groups are presented in the following section. School Selection For the performance evaluation, evaluators aim to visit ten schools—one school in each of the eight provinces where RFS-SSE and RFS-NPR conducted pilots and two schools in different regions that participated in RFS-IDCRT. Ten schools serve as a logistically feasible number of schools to travel to and collect data within the planned data collection schedule and provides a considerable amount of coverage of the RFS pilot (eight out of 90) and IDCRT (two out of ten) populations. Evaluators will use existing secondary data from NPR to purposively select the eight RFS pilot schools to ensure representation of a range of school characteristics such as urbanicity,94 predominant mother language of students,95 school performance,96 school population size, and socio-economic status, as measured by available proxies in existing NPR datasets. Of the eight RFS-IDCRT schools which participated in the project evaluation, two have been selected based on the range of grades offered, teacher backgrounds, as well as student performance on EGRA at endline.97 School-Level Respondents At each of the ten schools, a KII will take place with the school director, and two FGDs will be conducted—one with a group of four to ten teachers (grades 2-4), and one with a group of four to ten parents with students enrolled in grades 2-4. As part of the pre-site visit mobilization, school directors will be asked to identify and invite ten parents of students in grades 2-4 who have been identified as lagging behind or as struggling readers to participate in the focus group discussion.98 Only one parent per family will be recruited. If possible, an equal number of male and female parents and teachers will be interviewed. If possible, student data will be gathered from existing project-level data, including EGRA data, survey results, and evaluation reports. 94 For example, the demographic criteria may distinguish between urban, peri-urban, rural, and deep rural schools. 95 Such as Darija, the Moroccan Arabic dialect, or Amazigh languages, such as Tarifit, Tashelhit and Central Atlas Tamazight. 96 Ideally, “high performing” and “low performing” schools may be determined by existing MASSAR, EGRA data, or other RFS-SSE or RFS￾NPR project data and/or experience. 97 The two RFS-IDCRT schools that will be visited are Al Fath Association in Meknes Prefecture in the region of Fes-Meknes and Fondation Lalla Asma in Rabat Prefecture in the region of Rabat-Salé-Kenitra. 98 Ideally, students who are ‘lagging behind’ or struggling readers may be identified through NPR’s forthcoming Early Warning System set to be integrated in its enrichment programs. However, if that is not feasible, struggling students may be identified by their participation in remedial programs or through their classroom teachers. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 104 MoE Officials and Other Stakeholders Evaluators will conduct KIIs with a host of MoE and government officials at the provincial, regional, and national levels. The provincial KIIs will be conducted in ten provinces in six regions, including all eight provinces where RFS-SSE and RFS-NPR experimental schools are located and two provinces where RFS-NPR has scaled up its activities. The two proposed scale-up areas are the urban Tangier Prefecture99 in Tanger-Tetouan-Al Hoceima100 and rural Al Haouz province in Marrakesh Safi region. In each of the ten provinces, two MoE officials will be interviewed—the provincial director and the inspector/provincial coordinator—as well as the Arabic teacher trainer at the province’s teacher trainer center. At the regional level, interviewees will include the MoE regional director in each of the four regions where RFS-SSE and RFS-NPR schools are located as well as two regional directors of teacher training centers—one in Fes-Meknes and one in Rabat-Salé-Kenitra. National-level interviews will be conducted with five officials from the central MoE and one official each from the Ministry of Health and the Ministry of Solidarity, Women, Family and Social Development. Within the MoE, interviewees will include the directors of five divisions—the Directorate of Strategy, Statistics, and Planning; the Curriculum Directorate; the Directorate of Evaluation; the National Center for Pedagogical Innovations and Experimentation (CNIPE); and the National Center for Evaluation and Examinations (CNEE). In addition, evaluators will seek to interview two representatives from disabled persons organizations, including the Deaf Association Steering Committee, and one education committee member of Parliament or an education specialist, as recommended in the design workshop. USAID/Morocco and RFS IPs Evaluators will conduct KIIs with representatives from USAID/Morocco and all four RFS activities. The USAID/Morocco officials to be interviewed include three members of the Education Office—the Director, Contracting Officer’s Representative (COR) for the activities, and the Education Specialist. Evaluators will conduct at least three interviews with IPs from each RFS activity. At a minimum, evaluators will conduct one KII with the Chief of Party (COP) and Deputy Chief of Party (DCOP), one with technical staff including monitoring and evaluation specialists, and for projects with subcontractors or similar partners, at least one with a subcontractor identified by evaluators together with the Mission and IPs. Specific subcontractors include subject matter experts engaged by Management Systems International, Creative Associates International, and Al Akhawayn University under RFS-HICD; two of the six publishers of RFS-NPR materials; technical staff working on RFS￾NPR from University of Oregon, Management Systems International, and Al Akhawayn University; and two of the ten civil society organizations (CSOs) that have implemented summer enrichment reading activities under RFS-SSE. This wide net of respondents will enable evaluators to collect data from multiple perspective without greatly increasing the scope of data collection. Interviewing more than two partner organizations is not expected to yield additional useful data given the focused scope of each partner organization relative to other respondents. 99 The proximity to Rabat makes Tangier one of the most feasible non-pilot regions for inclusion in the sample. Given its closeness and ease of transportation to and from Rabat, the inclusion of Rabat fits within the evaluation team’s broader time and logistical constraints around data collection. 100 Alternative options for regions for inclusion include Béni Mellal-Khénifra, Casablanca-Settat, or Drâa-Tafilalet. The remaining three regions—Guelmim-Oued Noun; Laâyoune-Sakia El Hamra; and Dakhla-Oued Ed-Dahab—have been excluded from the sampling frame due to anticipated logistical constraints for evaluation team’s travel to these regions related to their distance from the capital. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 105 DATA SOURCES AND COLLECTION METHODS In order to respond to each evaluation question, several tools will be developed. This section details the process for tool development, review, and approval by USAID. It also provides additional information on the two primary collection methods—KIIs and FGDs—as well as sources for secondary quantitative analysis. Tool Design, Field Test, and Review Process The evaluators will develop seven new qualitative tools101—five KII guides and two FGD guides—to use during field work. During the design phase, evaluators will clearly map in Excel each item on each tool to relevant evaluation questions and sub-questions to ensure that sufficient information is elicited from all appropriate respondents to answer the evaluation questions. These detailed tool maps will be used to produce versions of each tool in Word that are structured for appropriate interview and discussion flow with respondents. Each tool will be aligned with one specific type of respondent, including: • USAID/Morocco officials; • RFS implementers and subcontractors; • MoE officials; • Non-MoE government officials; and • School-level respondents, including school directors, teachers, and parents While the same tool will be utilized for subgroups within a respondent type, additional sub-sections or targeted items may be incorporated for specific subgroups within the main tool, where relevant. For example, MoE officials at different levels will have certain sections of the core MoE tool removed based on their level of involvement in the project. IP staff will be asked different sets of questions depending on the activity in which they participated. Based on the finalized and approved evaluation design and questions, the data collection tools will first be developed in English and submitted for initial USAID review and approval in June. Upon USAID approval of the English tools, the tools will be translated into Arabic and subsequently submitted to USAID for review and approval in July. At this time, tools will also be submitted for review as part of the IRB approval process. In August, two Research Coordinators will be hired in Morocco to administer a limited pre-test of the tools in and around Rabat. The pre-test will provide an opportunity to field-test the appropriateness of question items for relevant respondents, identify potential practical issues with the tools’ administration and field procedures, as well as surface any flaws or limitations in the tool design. Based on the learnings from the pre-test, evaluation team will modify the tools as needed and submit them to USAID/Morocco for final review and approval prior to operational data collection in September. The tools will also include informed consent scripts/forms and clearly defined protocols for respondents’ participation in the evaluation, including an outline of risks/benefits, the right to decline participation, and confidentiality. NORC’s Institutional Review Board (IRB) will review the evaluation design and tools to ensure proper procedures and protocols are in place.102 MoE approval will be obtained prior to any school-based data collection. Evaluators will submit a list of proposed schools and respondents to USAID/Morocco. Through the Mission’s existing 101 No electronic data capture software or methodologies are anticipated given the qualitative focus of the tools. 102 The IRB review will either approve the study protocol or determine that the activity is classified as exempt. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 106 relationship with MoE, approval to collect data will be requested in the weeks prior to data collection. Key Informant Interviews (KIIs) The evaluators will conduct 67 KIIs, including 16 KIIs with USAID/Morocco and its IPs, 41 KIIs with Moroccan government officials and other stakeholders, and ten KIIs with school directors. Additional details on respondent types and subgroups anticipated for participation within each of the broader categories—as well as associated targets—are detailed in Table 8. Focus Group Discussions (FGDs) A total of 20 FGDs are targeted for completion across ten school visits. As noted in Table 8, the evaluation team will complete ten FGDs with primary school teachers (one FGD per school visit) and ten FGDs with parents of primary school students (one FGD per school visit). The majority of the FGDs (16 out of 20) will be conducted in pilot schools and the remaining will be conducted in schools that participated in the IDCRT project. FGDs will aim for gender-balanced participation where possible; for example, half of the parents of students in early primary grades participating in FGDs will be fathers/male guardians and half will be mothers/female guardians.103 Table 8. Respondents by targeted number, tool, subgroup and sampling/selection criteria # Respondents (N) Tool Subgroups Sampling/Selection Criteria 1 USAID Morocco Officials (3) USAID KII Guide N/A XXXXXXXXXXXXXXXX XXXXX 2 USAID IPs and subcontractors (13) USAID IPs KII Guide XXXXXXXXX XXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXX XXXXXXXXXXX XXXXXXXX XXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX 3 MoE government officials (41) MoE Official KII Guide XXXXXXXXX XXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXXX XXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXX 103 Only one parent or guardian is expected per family. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 107 # Respondents (N) Tool Subgroups Sampling/Selection Criteria XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX XXXXXX XXXXXXXXXX XXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXX 4 Non-MoE stakeholders (5) Non-MoE Stakeholder KII Guide XXXXXXXXX XXXX XXXXXXXXX XXXXXX XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXX XXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXX XXXXXX XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXX XXXX XXXXXXXXX X XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX 5 School directors (10) School Director KII Guide XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXX XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXX 6 Teachers (up to 100)104 Teacher FGD Guide XXXXXXXXX XXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXX XXXXXXXXXXXXXXXX 7 Parents (up to 100) 105 Parent FGD Guide XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX 104 This assumes ten persons per FGD. 105 Ibid. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 108 # Respondents (N) Tool Subgroups Sampling/Selection Criteria XXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXX Inclusion Considerations during Data Collection In selecting or recruiting respondents, the evaluators will aim to be as gender-sensitive and inclusive as possible by selecting equal numbers of men and women and by seeking inclusion of deaf and hard of hearing respondents as appropriate. In addition, all KII and FGD notes will include respondents’ sex; this information will be monitored throughout data collection to ensure both women and men’s perspectives and experiences are being captured in this evaluation. To facilitate broader inclusion, the evaluation team will be sensitive to respondents’ language preferences. For example, if a respondent prefers Amazigh language, a translator may be used. In addition, a Moroccan sign language interpreter will be engaged for FGDs with respondents who are deaf or hard of hearing, e.g., parents at the two IDCRT schools. As part of the mobilization process, the list of potential KII respondents will also aim to capture the respondent’s preferred interview language to ensure the deployed evaluation team’s language skills match the preferred language of the target respondent.106 Secondary Data Sources For secondary quantitative data analysis, the evaluation team will work with USAID/Morocco, IPs, and MoE to secure the following datasets: • School-, teacher-, and pupil-level data for the NPR baseline study conducted in May 2018 • School-, teacher-, and pupil-level data for four of the five waves of data collection conducted as part of the SSE impact evaluation (May 2016, September 2016, May 2017, and September 2017) • Student- and school-level performance data from MASSAR for the 2016/17 and 2018/19 school years, including data on retention, repetition, dropouts, and academic performance (where available) Requested data will be de-identified at the individual level but identified at the school-level to facilitate meaningful analysis. In addition, the evaluation team will use qualitative field work as an opportunity to collect reliable information on the availability and relevance of further secondary data sources—including internal M&E data from IPs—that could support supplementary analysis and further triangulate the evaluation questions. FIELD WORK The evaluation team will collect qualitative data in Morocco in September and October 2019. 106 As noted previously, all teams will include two fluent Arabic speakers. In addition, it is also anticipated two members of each team will also be fluent in French. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 109 Data Collection Teams Three teams will collect data over approximately three weeks in Morocco. Each team will consist of up to three data collectors. A qualitative evaluation expert from STS or NORC will lead each team and provide quality assurance. One or two Arabic-speaking interviewers recruited locally within Morocco will join each team. The composition of the teams will ensure collection of data in Arabic, French, English, or Amazigh languages. Each data collection team will have at least one female and one male researcher. Data Recording and Notes Capture In advance of each FGD or KII, the team lead will identify a data collector/interviewer and a notetaker. The interviewer will lead the interview or discussion while digitally recording it. The notetaker will keep detailed notes during the FGD or KII.107 Permission to audio-record will be requested as part of the consent process prior to the start of the KIIs. For cases where a KII or FGD participant declines permission to be audio-recorded, the team will respect their wishes and proceed with the interview without the audio-recording while taking note of the refusal to be recorded. Field notes will be typed up as soon as possible to capture details from the interview while it is still fresh. Each evening, immediately after the FGD or KII, the notetaker will listen to the day’s recording and type complete notes in either English or French. The team lead will review these electronic notes daily for quality and completeness and submit them to the evaluation home office. One remote evaluator will review these notes daily for monitoring purposes and track progress against the field work schedule. The evaluation team will conduct research in accordance with international norms on the ethical conduct of human subjects research, including obtaining consent108 from respondents. Qualitative data (including audio recordings) will be stored on a secured, password-protected server. Once field notes are saved electronically, the audio files will also be uploaded to a secure, password-protected server and deleted from the audio-recording device. Field Work Preparation, Communication, and Coordination In the weeks prior to field work, the data collection teams will prepare remotely, including a one-day remote session conducted via video teleconference to orient all data collectors to their assigned regions and responsibilities. Teams will also participate in a two-day training in Morocco before data collection begins. Each data collector will receive a handbook describing the procedures for facilitating FGDs, conducting KIIs, and capturing full, detailed, and accurate notes. During field work, the three data collection teams will communicate regularly with one another to ensure consistency of procedures. During the first two weeks of data collection, all three teams will speak on the phone nightly to debrief together. During the third week, individual teams will continue to debrief nightly, and two inter-group phone calls will take place. 107 Handwritten notes can be in any language, but daily typed notes will be in either English or French. For example, if an interview is conducted in Arabic, the interview will be recorded to supplement the interviewer’s field notes (to be compiled at the end of the interview). The evaluation team anticipates the majority of interviews will be conducted in Arabic, enabling one team member to conduct the interview while the other takes notes. 108 The consent procedures will clearly explain the purpose, use, treatment, and destruction plan for any audio recorded data so that respondents can make an informed decision. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 110 Data Collection Schedule Before beginning data collection, the evaluation team will meet in person with USAID for an in￾briefing, tentatively scheduled for September 11. The team will then participate in a two-day data collection training led by STS to ensure all data collectors fully understand the study design, tools, and procedures. Data collection will begin on September 16 after schools open. Initially, all three teams will remain in the Rabat-Salé-Kenitra region to establish common protocols and ensure coordination. Team 1 will primarily remain in the Rabat area to conduct high-level KIIs with limited travel to Casablanca and Fes-Meknes. Team 2 will work in Fes-Meknes before traveling to Souss Massa and Marrakesh-Safi. Team 3 will also travel to Fes-Meknes before collecting data in Oriental and returning to Rabat during the final week of data collection. Table 9. Summary field work schedule Team Week 1 (first half) Week 1 (second half) Week 2 Week 3 1 Rabat-Salé-Kenitra Rabat-Salé-Kenitra Rabat-Salé-Kenitra and Casablanca Fes-Meknes and Rabat-Salé-Kenitra 2 Rabat-Salé-Kenitra Fes-Meknes Souss Massa Marrakesh-Safi 3 Rabat-Salé-Kenitra Fes-Meknes Oriental Rabat-Salé-Kenitra Following the completion of data collection, all teams will return to Rabat for an in-person debriefing and identification of initial themes from data collection. The trip will conclude with an out-briefing with USAID, tentatively scheduled for October 4. Please see Annex VI for a draft of the field work schedule. ANALYSIS PLAN Qualitative Analysis Preliminary data capture and coding will begin during field work with the team debriefing sessions that take place each night within each of the three data collection teams. In addition, all three teams will join a group call twice per week during data collection. These debriefing sessions and associated notes will provide a way to document teams’ preliminary findings in real time during data collection. These notes will inform the out-briefing session with USAID and supplement the finalized field notes. Finalized debriefing and field notes will be imported into NVivo 12, a qualitative data analysis software package. The qualitative data analysis methodology will incorporate an iterative approach and include thematic content analysis of narrative data to identify and validate emerging themes. A codebook will be developed at two stages—when the evaluation design is finalized, and during data analysis when additional themes emerge. In addition to immediate quality assurance of notes during field work, a limited number of field notes will be double coded to assess inter-coder reliability. Coders will meet to discuss any discrepancies identified and agree on definitive codes to apply. After the coding process generates meaningful patterns from the field notes, analysts will use these codes to perform second-order analyses. These analyses will examine coded data in NVivo 12 to identify salient themes across a range of respondents that answer each evaluation question. Responses from different individuals and respondent groups will be compared to identify discrepancies, if any exist, and to offer explanations for such discrepancies. The relationships between responses will also be USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 111 examined to learn about linkages across themes. Hypotheses that explain finding will be tested by examining codes for disconfirming patterns or alternative explanation. Multiple analysts working in a single software package will ensure that data, codes, and analysis findings are stored in a common, shared location for reference during reporting. Quantitative Analysis Quantitative data analysis will be conducted using the StataSE 15 statistical software package. Data will be imported and reviewed to ensure completeness and consistency in variable naming and coding, particularly when multiple datasets are merged. All relevant variables will be examined for missingness and outliers prior to analysis and appropriate strategies will be employed to address any observed irregularities (e.g., robustness checks and multiple imputation). Multiple regression analysis will be the primary analytical approach, allowing for statistical cross￾group comparisons (e.g., pre/post, treatment/comparison) while controlling for confounders and/or increasing statistical precision of the estimates. Where appropriate, basic descriptive and summary statistics and cross-tabulations will also be presented. The current proposed secondary quantitative analysis includes: • SSE treatment and control contrast of grade 2/3 pupils using NPR baseline EGRA data collected in May 2018. Because schools were randomly assigned to treatment, this analysis will allow for estimation of the causal impact of the SSE program following a full two years of program participation for the target pupils. • Cross-sectional pre/post comparison of SSE grade 1 pupil EGRA scores in May 2016 to SSE grade 1 pupil scores in May 2018. This comparison will help USAID/Morocco understand how reading scores in the early days of the SSE rollout to compare to scores after 2 full years of implementation. Time series analysis using grade 1 scores collected in May 2017 can shed further light on how pupil performance is trending over time.109 • Cross-sectional pre/post comparison of (non-SSE) grade 1 pupils targeted by the NPR national scale-up. Comparing MASSAR data on retention, repetition, dropouts, and performance (where available) for grade 1 students in the 2016/17 (pre) and 2018/19 (post) school years can offer early evidence on the efficacy of the scaled version of the NPR program. • Cross-sectional statistical contrast of grade 1 EGRA performance at SSE “high touch” schools with performance at NPR scale-up “low touch” schools using May 2018 EGRA data. This analysis may provide evidence as to whether observed results can be sustained at scale. As described in previous sections, the evaluation team will use qualitative field work as an opportunity to collect reliable information on the availability and relevance of further secondary quantitative data sources—including internal M&E data from IPs—that could support supplementary analysis and further triangulate the evaluation questions. Qualitative and quantitative findings will be integrated while the final report is written. For relevant research questions, secondary quantitative analysis will be utilized to contextualize, supplement, and/or triangulate the qualitative findings. If divergent findings emerge during analysis, the evaluators will re-examine the raw qualitative data through a more in-depth content analysis to determine if any other factors or issues were missed when data was first being organized in order to gain a deeper 109 For cross-sectional comparisons, it is important to use data from collected at the same time point during the school year. As such, there is no comparable “post” measure for the true SSE baseline, which was conducted in January 2016. While the SSE program was already in effect during the May 2016 EGRA, it had only been operating for 3-4 months at the time of data collection; in May 2018, the program would have been in effect for 2 full years. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 112 understanding of the data. In addition, disaggregated data may be reanalyzed by characteristics of respondents—such as locale of the respondent, respondent type, etc.—to help provide additional explanations for the variances the findings. REPORTING This evaluation includes the following deliverables and submission dates: # Deliverable Estimated Delivery 1 Evaluation work plan March 29, 2019 2 Evaluation design report May 31, 2019 3 Draft evaluation report (English) November 8, 2019 4 In-country validation presentation December 5, 2019 5 Final evaluation report (English) December 20, 2019 6 Evaluation executive summary (Arabic and French) January 15, 2020 7 Final remote presentation January 28, 2020 The primary audience of the full evaluation report is the USAID/Morocco Education Office in Rabat. The full evaluation report will adhere to the USAID Office of Education’s reporting templates and USAID guidance for evaluation reports. The draft report will be submitted for feedback in November. After the draft report is submitted, the evaluation team will share and validate the evaluation findings at a workshop in Morocco with Mission staff, IPs, and MoE officials. NORC will submit the final evaluation report to USAID in December following written feedback from USAID and validation of findings at the workshop. The draft and final evaluation reports will include executive summaries in English. The evaluation team will translate the executive summary into French and Arabic in January. The primary audience of the executive summary is the Moroccan MoE and related stakeholders in the education system. Evaluators will complete all reporting and dissemination activities by the end of January 2020. LIMITATIONS OF THE PROPOSED APPROACH Evaluation Design As a primarily qualitative performance evaluation, the majority of activities under this study will not be able to speak directly to the causal impact of RFS activities. Rather, the evaluation will describe ways in which RFS was understood to be successful, both from written documentation (e.g., project reports) and from the perspectives of program implementers, stakeholders, and beneficiaries. Linguistic and Cultural Complexity USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 113 The evaluation will review program success across diverse languages and cultural contexts, including deaf communication. In order to minimize misunderstandings and maximize learning, the evaluation teams will include researchers with French and/or Arabic language skills as well as previous experience working in the education sector in Morocco. The teams will also keep running records of linguistic and cultural issues that arise during data collection to help further contextualize findings. Fidelity of Implementation In some cases, it may be challenging to disentangle program effectiveness (or lack thereof) from implementation fidelity issues. For example, the desk review found that the scale-up of NPR was limited by strikes of inspectors who were designated to carry out training in NPR scale-up regions. While this development may undermine program effectiveness, it is prima facie due to implementation challenges rather than flaws in the project’s underlying theory of change. Representativeness of Sample Respondents for this primarily qualitative evaluation are not randomly selected and may not be representative of the full range of experiences among participants in the RFS project. The small sample of qualitative respondents may not reflect the perspective of the groups they were selected to represent, and systematic sampling error or bias might be present. This evaluation also does not target the most highly marginalized populations, who may not be in school at all. The study team will critically evaluate qualitative responses and triangulate among populations to discern accuracy. Risk of Divergent Findings Different individual respondents and groups of respondents may have different views on program effectiveness. While responses will be triangulated, determining whose information is correct in case of conflict will be challenging. In addition, qualitative and quantitative findings may diverge in some cases. Evaluators will acknowledge diverse perspectives in the final report as well as re-examine the raw qualitative data through more in-depth content analysis to determine if any other factors or issues were missed when data was first being organized in order to gain a deeper understanding of the data. In addition, disaggregated data may be reanalyzed by characteristics of respondents—such as locale of the respondent, respondent type, etc.—to help provide additional explanations for variances in findings. Availability of Respondents RFS-SSE, RFS-HICD, and RFS-IDCRT have already closed, which will limit the availability of personnel for interviews. In addition, it may be difficult to get teachers to participate in interviews outside of school hours if they are not compensated. When possible, the evaluators will conduct interviews with stakeholders and personnel no longer residing in Morocco via Skype or telephone. Special efforts will also be made to accommodate respondents’ schedules or preferred methods of communication. Based on prior experience, a reasonable response rate is expected. Accuracy of Qualitative Responses Respondents may be biased or not feel comfortable speaking freely. Data collectors will explain confidentiality and anonymity of responses during consent procedures to ensure respondents are comfortable providing frank, accurate data. Within FGDs, facilitators will remind participants to respect the privacy of their fellow participants and that the information discussed needs to remain confidential—i.e., not to repeat what is said in the focus group to others. In addition, the consent procedures will clearly articulate that there are no direct benefits for participating in the study so that respondents do not provide misleading information in the hopes of receiving some material benefit in the future. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 114 ANNEX III: DATA COLLECTION INSTRUMENTS Tool 1: Key Informant Interview Guide: USAID Qualitative Instruments: USAID RFS Performance Evaluation Background A. Respondent type:_____________________________________________________ B. Respondent role/position:_______________________________________ C. Sex: _______________________________________ D. Age: _______________________________________ E. Interviewer name: ____________________________________________________________ F. Note taker name: _____________________________________________________________ G. Date: _____________________________________________________________ H. Start Time: _____________________________________________________________ I. Introduce team members and obtain consent: _____________________________________ Introduction and Consent [NOTE TO INTERVIEWER: please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make A note about this tool: Purpose: This interview guide will enable you to gather information from USAID key informants to get an overview of USAID’s involvement in the RFS project. The objective is to gain insights and understanding of USAID’s role with RFS, project implementation and outcomes, engagement with the Ministry of Education and project sustainability, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with members of the USAID/Morocco Education Office—the Director, the Contracting Officer’s Representative for the activities, and the Education Specialist. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 115 sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. ROLE WITH RFS 1. Can you please tell me about your role working with RFS? [NOTE TO INTERVIEWER: Probe the following: a. Are you currently involved in NPR? If yes, how so? b. Were you previously involved in SSE? c. Were you previously involved in HICD? d. Were you previously involved in IDCRT, which worked with children who are deaf and hard of hearing?] B. OUTCOMES 2. Do you feel that the RFS activities are/were effective in helping improve primary school children’s readings skills? In what ways? Please describe why and provide examples. 3. Do you feel that the RFS activities are/were effective in increasing community support for reading? In what ways? For all responses, please describe why and provide examples. a. What about engaging families to support reading? b. What about building CSO's capacity to implement reading initiatives? 4. Do you feel that the RFS activities are/were effective in improving the policy environment around reading? a. Can you provide specific examples about why or why not? b. Can you provide examples of any laws, policies, regulations or guidelines that have been developed and/or successfully implemented to improved reading at the primary level? 5. Do you feel the HICD recommendations were useful for the MoE? a. Has the MOE taken any action in response to these recommendations? Can you provide specific examples? C. IMPLEMENTATION Now I’d like to ask you about RFS implementation. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 6. What have been the most important and valuable contributions of the RFS activities to improving children's reading in Morocco? a. Why do you consider these the most important or valuable? 7. What factors helped facilitate achievement of these accomplishments? a. How did they help? b. Were there any other factors that helped achieve the RFS project objectives? 8. Have there been some aspects of the RFS activities that have not met your expectations? Please describe and provide examples. a. How would you suggest these shortcomings be addressed? 9. What kinds of concerns or obstacles, if any, have impeded implementation of the RFS [or NPR/SSE/IDCRT/HICD] activities? a. Why did these obstacles impede implementation? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 116 10. Thinking specifically about the project's objectives, what kinds of obstacles limited achievement of the objectives? a. Why? b. Do you have any suggestions on how to address these obstacles? D. ENGAGEMENT WITH MINISTRY/SUSTAINABILITY I’d also like to ask you about USAID’s engagement with the Ministry of Education and sustainability of RFS. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 11. How did USAID/Morocco and the RFS implementers seek engagement from the Ministry? a. How effective was this engagement? b. What do you think made engagement effective (or ineffective)? 12. Do you feel that the Ministry's involvement has been not enough, limited, about right, or too much? Can you provide specific examples? [NOTE TO INTERVIEWER: Probe for differences between Ministry bodies at the national, provincial, and regional levels] 13. Can you comment on the MoE’s level of engagement and leadership on the design, implementation and roll-out of the new reading curriculum and materials under RFS-NPR? 14. Are the outcomes of the RFS activities likely to continue after the NPR activity ends in 2022? a. If yes, which ones? Why? b. If no, why not? 15. What aspects of the RFS project do you feel will have the most lasting effects? Please describe, with specific examples if possible. [NOTE TO INTERVIEWER: Probe by key areas, including: a. on children's reading skills b. on teacher's reading instruction c. on CSO/community/family support and engagement in children's reading d. on MOE capacity to implement national reading] [NOTE TO INTERVIEWER: If not already addressed by respondent, ask specifically: Do you think the MOE will be likely to continue using the early grade Arabic curriculum/materials in the future (after the project ends)? Why or why not?] 16. What strengths or capacity exist that will support continuation of RFS activities after RFS closes? Can you provide specific examples? a. Within the MOE? b. Within the CSOs? c. Within the communities? 17. What challenges do you think the MOE faces in creating a sustainable national reading system? [NOTE TO INTERVIEWER: Allow respondent to reply unprompted first. If not discussed, probe for: a. Speed of RFS-NPR scale-up and national rollout b. Teacher coaching c. Time available for teacher training d. Breadth of support for RFS-NPR across multiple MOE subunits e. MOE support to education for students with disabilities f. Anything else that has arisen earlier in the interview] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 117 18. What additional support could USAID provide that would best help the MOE or other stakeholders continue to build and sustain the current national reading system? E. INCLUSION I’d also like you ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 19. Based on your experience, what factors help children thrive in their education at the primary level? Can you provide specific examples? [NOTE TO INTERVIEWER: Examples of factors might include materials, quality teaching, school environment, encouragement and support from home and community, etc.] 20. Which students in Morocco face challenges in meeting expectations in their education at the primary level? [NOTE TO INTERVIEWER: Allow respondent to answer unprompted first, but if they have difficulty answering you may prompt them to ask about the following: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls; e. Children who are orphans; f. Children in poor households; g. Children who are migrants (e.g., nomads, refugees); h. Children engaged in child labor; i. Children in areas experiencing insecurity] 21. What are the barriers to reaching those children and supporting their education? a. What about for children with disabilities specifically? 22. How could these children be better supported in their education in the future? F. LESSONS LEARNED The last set of questions I’d like to discuss with you are about lessons learned. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 23. Is there anything you would have done differently with RFS to increase its effectiveness? Please describe, with specific examples if possible. 24. What specific activities do you think USAID can undertake to best improve reading education for children in Morocco in the future? [NOTE TO INTERVIEWER: For each response, ask: "Why do you think this activity would be the most effective way for USAID to improve reading education?"] 25. How about for improving reading for all children? a. What about... i. Children with disabilities; ii. Children whose mother language is not Arabic; iii. Children in rural areas or other geographically marginalized areas; iv. Girls; v. Any other marginalized or vulnerable children? 26. Were there any unexpected outcomes or results (positive or negative)? G. CLOSING 27. Are there any additional comments that you would like to share with us? 28. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 118 H. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the Ministry. End Time: ___________________________ Total length: ________Hours _________Minutes POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 119 Tool 2: Key Informant Interview (KII) Guide: Implementation Partners Final Version September 23 Qualitative Instruments: USAID RFS Performance Evaluation Background A. Name: _______________________________________ B. Respondent's Role/Position: ________________________________________________ C. Participating Organization and Department: __________________________________ D. Sex: _______________________________________ E. Age: 21-30, 31-40… F. Interviewer name: ____________________________________________________ G. Note taker name: ____________________________________________________ H. Team Leader: ____________ I. Date: ______________________________________________________________ J. Start Time: __________________________________________________________ K. Introduce team members and obtain consent: _____ YES ______NO L. State purpose of evaluation: _____ YES ______NO About this tool: Purpose: This interview guide will enable you to gather information from implementing partner key informants to get an overview of their involvement in the RFS project. The objective is to gain insights and understanding of implementing partners’ work and their engagement with the Ministry of Education, the new Arabic language curricula introduced as part of RFS, inclusion, sustainability, and lessons learned. Recommended Sources: Information should be collected through at least three interviews with implementing partners from each RFS activity. At a minimum, evaluators will conduct one KII with the Chief of Party (COP) and Deputy Chief of Party (DCOP), one with technical staff including monitoring and evaluation specialists, and for projects with subcontractors or similar partners, at least one with a subcontractor identified by evaluators together with the Mission and implementing partners. Specific subcontractors include subject matter experts engaged by Management Systems International, Creative Associates International, and Al Akhawayn University under RFS-HICD; two of the six publishers of RFS￾NPR materials; technical staff working on RFS-NPR from University of Oregon, Management Systems International, and Al Akhawayn University; and two of the ten civil society organizations (CSOs) that have implemented summer enrichment reading activities under RFS-SSE. If more than one official is interested and available to participate at the planned date, time and place, the KII can be conducted as a group interview. However, the possible impact of any power dynamic (including due to age, gender and the level of authority within the organization or the ministry) must be considered and mitigated during the interview. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 120 Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained for each person, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participate. Make sure to also ask for permission to record the interview on your audio recorder. If the participant answers in the affirmative, please proceed. If participants answer NO, agree to not record the conversation and proceed without turning on your audio recorder.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. I. ROLE IN YOUR ORGANIZATION AND IN RFS [SSE, NPR, IDCRT] 1. Can you describe your job's main duties? a. How long have you been in this job? 2. Were you previously involved in the following initiatives? If yes, how were you involved? i. SSE ii. NPR iii. IDCRT iv. HICD 3. Have you worked with the Ministry of National Education? If yes, with which departments, centers or individuals? How frequently? 4. For COPs/Project Leadership] How involved was the MoE in designing, implementing and managing your RFS activities? Can you give specific examples? a. At regional level? At provincial level? At school level? b. Do you think that the involvement of the MoE was sufficient, limited, just enough or excessive? Can you give specific examples? 5. Under the project, how would you describe the relationship between your organization and the MoE? [Preferably, probe for an adjective that describes the relationship. As appropriate, prompt them with their statements about their previous work.] 6. How has your project use monitoring and evaluation (M&E) data to adjust your activities? a. Can you give examples? What about EGRA (reading)/EGRSLA (sign language) specifically? b. What, if any, adjustments to your project implementation came from M&E data? J. PERSPECTIVES ABOUT THE RFS PROJECT [SSE, NPR, IDCRT] 7. Do you think that your RFS activity succeeded in improving the reading skills of primary school students? Can you give examples of activities that helped achieve this? 8. [Prompt respondents to describe which ones, indicate why and give examples.] Do you think that your RFS activity (e.g., SSE, NPR ...) succeeded in increasing family or community support for reading? Can you give examples of activities that helped achieve this? [Prompt respondents to describe which ones, indicate why and give examples of activities in USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 121 which they were involved—e.g., in summer courses.] 9. Do you think that your activity has enhanced SCO skills thus helping them implement reading initiatives? Can you give examples of activities that helped achieve this? [Prompt respondents to describe which ones, indicate why and give examples of skills that were developed.] [For the main partners (prime contractors: Creative, Chemonics, IDRT, Kaizen)] 10. Do you think that your RFS activity succeeded in improving the policy environment to support reading? Can you give examples of activities that helped achieve this? (E.g., laws, policies, regulations or guidelines) [SSE, NPR, IDCRT] 11. In your opinion, which are the most important contributions of your RFS activities? [Probe for the greatest contribution or success, not a full list. Look for specific examples—i.e., not “learning to read”, but more specifically...] 12. Which challenges have you or your staff encountered during while implementing your RFS activity? a. Do you have any suggestions on how to address them? K. CURRICULA [Main implementation partners: SSE (Chemonics), NPR (Creative)] Now I’d like to ask you about the new Arabic language curriculum developed as part of RFS. 13. Can you comment on the MoE's level of engagement and leadership on the design, implementation and roll-out of the new reading curriculum and materials? Please give examples. Can you comment on the quality and relevance of the new reading curriculum and training materials? For example, are they appropriate for the learning context of Morocco? 14. Can you describe any successes related to the implementation of the new curriculum? Things that have been particularly relevant or useful? [E.g., a. Amount of time devoted to reading instruction b. Techniques used for reading instruction c. Texts in the hands of children d. Teaching children in their mother language (a language they speak and understand) e. Testing children's reading progress 15. Can you describe the various issues that emerged in implementing the new curriculum or in using the new materials? [If answered in question xxx, jump to the next question.] 16. How did the Ministry of Education support teachers and administrators in implementing the new curriculum and materials? a. Do you feel your RFS activity has built the capacity of the teachers and administers to improve reading instruction at the primary level? Can you provide specific examples about why or why not? b. Can you comment of the quality and effectiveness of the support being provided to teachers on reading instruction? c. Can you describe some of the challenges or factors that had an impact on the quality of training or support? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 122 d. Can you describe some of the challenges or factors that can impact the mainstreaming of the curriculum? L. INCLUSION [SSE, NPR, IDCRT] I would now like to ask you questions about inclusion. 17. Based on your experience, which types of children in Morocco still find it difficult to access and thrive in primary school? [Allow participants to spontaneously answer initially; if they say nothing, give 1 or 2 examples as follows (pick randomly)]: a. Children with disabilities b. Children whose mother language is not Arabic; c. Children in rural or other geographically marginalized areas d. Girls e. Other vulnerable or marginalized children] 18. Which obstacles prevent these children from receiving a quality education? [Refer to answers to the previous question.] a. What about specifically children living with disabilities? [Prompt about children who are deaf or hearing impaired in particular] 19. How could these children be better supported in the future through their education? M. SUSTAINABILITY [SSE, NPR, IDCRT] I would now like to ask you questions about sustainability. 20. Do you think that the Ministry of Education is likely to keep using the Arabic language curriculum in the future? a. If so, how? b. If not, why not? [If needed, mention: • Teacher training • School materials for teachers and students • Methods for supporting teachers—e.g., coaching/support • New approach to teaching with the Arabic language curriculum • Support for policy development • Activities to involve parents and communities] 21. What challenges does the MOE face in creating a sustainable reading instruction approach within the Arabic language curriculum? 22. What capacities exist that will facilitate continuation of RFS activities after RFS closes? Please give examples for the following levels: a. At MoE level b. At SCO level c. At community level 23. What support is needed to continue to build and sustain the current reading instruction approach within the Arabic language curriculum? N. LESSONS LEARNED [SSE, NPR, IDCRT] I would finally like to ask you questions about lessons learned. 24. What specific activities do you think USAID can undertake to best improve reading education for children in Morocco in the future? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 123 [For each response, ask: “Why do you think this activity would be the most effective way for USAID to improve reading education? “ a. And for children who are often excluded? E.g., with disabilities, whose mother language is not Arabic, living in rural or other geographically marginalized areas, girls, vulnerable or marginalized? 25. In general, have you noticed unexpected effects or results (positive or negative)? If so, which ones? O. CLOSING We have reached the end of our questions. Do you wish to share additional comments? Do you have any questions for our team? Thank you for your time. End Time: ___________________________ Total length: ________Hours _________Minutes Language of interview: __ Arabic __ French __Amazigh __ Other: Specify Turn off the recorder. POST-KII NOTES: Please comment on: • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate. • If more than one person participated, the various insights that have emerged through disagreements during the interviews. Any additional insight or comment that should be included. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 124 Tool 3: Key Informant Interview (KII) Guide: Ministry of Education Officials Final Version September 23 Qualitative Instruments: USAID RFS Performance Evaluation Background A. Name: _______________________________________ B. Respondent's Role/Position: ________________________________________________ C. Division within Ministry: ___________________________________________________ D. Sex: _______________________________________ E. Age: 21-30, 31-40… F. Interviewer name: _______________________________________________________ G. Note taker name: ________________________________________________________ H. Team Leader: ____________ I. Date: _____________________________________________________________ J. Start Time: _________________________________________________________ K. Introduce team members and obtain consent: _____ YES ______NO L. State purpose of evaluation: _____ YES ______NO A note about this tool: Purpose: This interview guide will enable you to gather information from Ministry of Education key informants to get an overview about their involvement in the RFS project. The objective is to gain insights and understanding of their engagement with the RFS activities and implementers, the new Arabic language curricula introduced as part of RFS, inclusion, and sustainability. Recommended sources: Information should be collected through with a host of MoE officials at the provincial, regional, and national levels. The provincial KIIs will be conducted in ten provinces in six regions, including all eight provinces where RFS-SSE and RFS-NPR experimental schools are located and two provinces where RFS-NPR has scaled up its activities In each of the ten provinces, two MoE officials will be interviewed—the provincial director and the inspector/provincial coordinator—as well as the Arabic teacher trainer at the province’s teacher trainer center. At the regional level, interviewees will include the MoE regional director in each of the four regions where RFS-SSE and RFS-NPR schools are located as well as two regional directors of teacher training centers—one in Fes-Meknes and one in Rabat-Salé-Kenitra. National-level interviews will be conducted with the directors of five divisions—the Directorate of Strategy, Statistics, and Planning; the Curriculum Directorate; the Directorate of Evaluation; the National Center for Pedagogical Innovations and Experimentation (CNIPE); and the National Center for Evaluation and Examinations (CNEE). If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 125 Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with each participant. If agreement to participate voluntarily is obtained for each person, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participate. Make sure to also ask for permission to record the interview on your audio recorder. If the participant answers in the affirmative, please proceed. If participants answer NO, agree to not record the conversation and proceed without turning on your audio recorder.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. ROLE WITHIN THE MoE AND IN RFS 1. Can you describe your job's main duties? a. How long have you been in this job? [MoE at regional and provincial levels only] 2. During this discussion, we will ask you questions about your experience with USAID's RFS project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment in this field. This initiative is currently implemented in collaboration with the MoE under the RFS project, and with Small Scale Experimentation projects (SSE), and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT), which helps improve the reading skills of deaf and hearing-impaired children using technology.] What involvement, if any, have you had with any of these activities? a. SSE b. HICD c. IDCRT d. Are you currently involved in the NPR activity? If so, what do you do? [MoE at central level only] 3. During this discussion, we will ask you questions about your experience with USAID's RFS project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. Can you describe your role in the following phases of the project? a. Preparation and design? b. Experimentation (SSE)? c. Implementation? d. Monitoring and evaluation? B. PERSPECTIVES ABOUT THE RFS PROJECT [MoE at all levels] 4. Do you think that RFS activities succeeded in improving the reading skills of primary school students? Please give examples. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 126 [Prompt respondents to describe which ones, indicate why and give examples of activities or activity components.] 5. Do you think that RFS activities succeeded in increasing family and community support for reading? Please give examples. [Prompt respondents to describe which ones, indicate why and give examples.] 6. Do you think that RFS activities succeeded in improving the policy environment to support reading? Please give examples. [E.g., laws, policies, regulations or guidelines] e. Are you familiar with recommendations from the RFS-HICD project? If so, do you think that these recommendations are useful? i. Has the MoE undertaken certain approaches as per the recommendations? Can you give specific examples? 7. Do you think that RFS activities succeeded in developing the skills of civil society organizations (CSOs) to implement reading initiatives? If so, please give examples. 8. In your opinion, what are RFS’ greatest successes? Please give examples. [If the respondent does not give examples, ask if he/she has seen any changes in: • Amount of time devoted to reading instruction • Techniques used for reading instruction • Texts in the hands of children • Teaching children in their mother language (a language they speak and understand) • Testing reading progress/benchmarks 9. In your opinion, what are the greatest challenges faced by RFS? Please give examples. C. RELATIONSHIP WITH USAID 10. How would you describe the relationship between your office and USAID on the one hand, and RFS’ implementation partners on the other hand: insufficient, limited, just enough or too close? Can you give specific examples? [Probe for an adjective that describes the relationship.] [Reminder: Chemonics (RFS-SSE), Creative (RFS-NPR), Kaizen Company (HICD) and Institut de recherche et de formation sur les handicaps (IDCRT implementer).] D. CURRICULA [AREF, Directorate for Curricula, Teacher Training Centers, “Super 8” Inspectors, MoE at central level] I would now like to ask you questions about the new Arabic language curriculum. 11. As you may know, the RFS project has worked with the MoE to introduce a new reading instruction method focused on the phonetic aspect of the language. Based on your experiences, what do think of this new method? Which successes or obstacles have you observed? 12. In your opinion, which factors facilitate or prevent the mainstreaming of the curriculum? E. STAFF TRAINING [AREF, Directorate for Curricula, Teacher Training Centers, “Super 8” Inspectors] 13. Do you think that RFS activities were effective in enhancing the skills of teachers and administrators? If so, which ones? 14. In your opinion, which factors facilitate or prevent training or support? USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 127 F. INCLUSION [AREF, Directorate for Curricula, Teacher Training Centers, “Super 8” Inspectors, MoE at central level] I would now like to ask you questions about inclusion. 15. How could these, often excluded, children be better supported in the future through their education? • Children with disabilities • Children whose mother language is not Arabic; • Children in rural or other geographically marginalized areas • Girls • Any other vulnerable or marginalized children] G. SUSTAINABILITY [AREF, Directorate for Curricula and Teacher Training Centers, “Super 8” Inspectors, MoE at central level] I would now like to ask you questions about sustainability. 16. Do you think that the Ministry of Education is likely to keep using the Arabic language curriculum and the new materials in the future? c. If so, how? d. If not, why not? [If needed, mention: • Teacher training • School materials for teachers and students • Methods for supporting teachers—e.g., coaching/support • New approach to teaching with the Arabic language curriculum • Support for policy development • Activities to involve parents and communities] H. OVERALL [AREF, Directorate for Curricula, Teacher Training Centers, “Super 8” Inspectors, MoE at central level] 17. What would you change in the RFS project to make it more effective? Please describe. 18. Do you have any recommendations for USAID, the MoE or the schools to help improve children's reading skills in the future? 19. In general, have you noticed unexpected RFS related effects or results (positive or negative)? If so, which ones? I. CLOSING We have reached the end of our questions. Do you wish to share additional comments? Do you have any questions for our team? Thank you very much for taking the time to answer our questions. End Time: ___________________________ Total length: ________Hours _________Minutes Language of interview: __ Arabic __ French __Amazigh __ Other: Specify Turn off the recorder. POST-KII NOTES: Please comment on: USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 128 • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate. • If more than one person participated, the various insights that have emerged through disagreements during the interviews. • Any additional insight or comment that should be included. _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 129 Tool 4: Key Informant Interview (KII) Guide: Non-Ministry of Education Official Qualitative Instruments: USAID RFS Performance Evaluation Background A. Respondent type: ________________________________________________ B. Respondent role/position: _________________________________________ C. Respondent Ministry or Organization: _________________________________ D. Sex: _______________________________________ E. Age: _______________________________________ F. Interviewer name: _______________________________________________ G. Note taker name: _______________________________________________ H. Date: ________________________________________________________ I. Start Time: ____________________________________________________ J. Introduce team members and obtain consent: ___________________________ K. State purpose of evaluation: ________________________________________ A note about this tool: Purpose: This interview guide will enable you to gather information from non-Ministry of Education key informants to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of their engagement with RFS, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with various non￾Ministry of Education officials, including one representative from the Ministry of Solidarity and one representative from the Ministry of Health, two representatives from disabled persons organizations, and one education specialist from Parliament. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 130 [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. ROLE IN ORGANIZATION AND RELATIONSHIP WITH MOE 1. Can you please tell me about your role in this Ministry [or organization]? a. Title? b. Which directorate or division? c. How long have you been in this role for? B. ENGAGEMENT WITH RFS 2. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading and improving the policy environment to support reading. This is currently being done by working with the MOE on the NPR activity as well as SSE and HICD in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called IDCRT. Have you had any involvement with any of these four USAID education activities? If yes, how so? 3. How would you describe your and your Ministry's [or organization’s] relationship with USAID education activities and their implementers? You might know them as IDCRT, SSE, NPR, and HICD; or Chemonics (RFS-SSE prime), Creative (RFS-NPR prime), Kaizen Company (HICD prime), and Institute for Disabilities Research and Training (IDCRT prime). C. INCLUSION Now I’d like to ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 4. As we've mentioned, USAID/Morocco is working with the Ministry of Education to enhance educational development for primary-school aged children (i.e. age 6-12). Based on your experience, which students in Morocco face challenges in thriving in their education at the primary level? [NOTE TO INTERVIEWER: allow respondent to answer unprompted first, but if they have difficulty answering you may prompt them to ask about the following: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls e. Children who are orphans; f. Children in poor households; g. Children who are migrants (e.g., nomads, refugees); h. Children engaged in child labor; i. Children in areas experiencing insecurity] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 131 5. What are the barriers to reaching those children and supporting their education? D. LESSONS LEARNED Now I’d like to ask you about lessons learned. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 6. What type of support currently exists for these children in their education, such as policies, strategies, organizations or activities? [NOTE TO INTERVIEWER: Probe for all groups of children mentioned by respondent above.] a. Which of these have been particularly effective? Why? b. Which of these have been ineffective? Why haven’t they been successful? 7. How could these children be better supported in their education in the future? 8. Do you have any other recommendations for USAID, the MOE, or schools for helping improve children's reading in the future? 9. How about for improving reading for all children? a. What about children with disabilities? b. What about children whose mother language is not Arabic? c. What about children in rural areas or other geographically marginalized areas? d. What about girls? e. What about any other marginalized or vulnerable children? 10. Were there any unexpected outcomes or results (positive or negative)? E. CLOSING 11. Those are all of my questions. Are there any additional comments that you would like to share with us? 12. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? F. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the Ministry. End Time: ___________________________ Total length: ________Hours _________Minutes USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 132 POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 133 Tool 5: Key Informant Interview (KII) Guide: School Directors Final Version 2 - September 23 Qualitative Instruments: USAID RFS Performance Evaluation Background A. Name(s) (e.g., Director and Deputy): _______________________________________ B. Role(s): ______________________________ C. Number of years in this position: ____ D. Sex: _________________________ E. Age: 21-30, 31-40… F. Name of the school: ___________________________________________________ G. Province: _______________________________________ H. Region: _______________________________________ I. Name of Facilitator: ___________________________________________________ J. Note taker name: _____________________________________________________ K. Date: ______________________________________________________________ L. Start Time: _________________________________________________________ M. Introduction of the team member and consent obtained YES NO Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained for each person, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participate. Make sure to also ask for permission to record the interview on your audio recorder. If the participant answers in the affirmative, please proceed. If participants answer NO, agree to not record the conversation and proceed without turning on your audio recorder.] A note about this tool: Purpose: This interview guide will enable you to gather information from school director key informants to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of general perceptions of quality of education, social outcomes, RFS project implementation, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with school directors who are actively working on early grade reading and/or inclusive education. Twelve school directors will be interviewed overall, including one apiece in each of the eight provinces where RFS-NPR pilot schools are located, two from RFS scale-up schools, and two with former directors of RFS-IDCRT schools. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 134 **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. PERSPECTIVES ABOUT THE RFS PROJECT During this discussion, we will ask you questions about your experience with USAID's RFS project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment to support reading through initiatives such as RFS/SSE/NPR. [Reference: This work is currently taking place through the National Program for Reading (NPR) activity as well as Small Scale Experimentation (SSE) and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT), which helps improve the reading skills of deaf and hearing-impaired children using technology.] [SSE, NPR, IDCRT] 1. What involvement, if any, have you and your school had with this program? Please describe. [Probe for information about • teacher and director training • received materials • community support • summer activities] 2. Since the inception of the RFS program in your school, have you noticed gaps in your students’ reading in your school? [Ask for explanations of the answers so that they are substantiated by concrete examples] a. If so, on which aspect? b. What was RFS doing during this time that may have contributed to this difference? 3. Since the inception of the RFS program, have you noticed a difference in the quality of reading instruction delivered by the teachers? a. If so, on which aspect? b. What was RFS doing during this time that may have contributed to this difference in teachers' reading instruction? 4. In your opinion, which RFS interventions have been most effective in helping teachers improve their reading instruction? [Probe for strategies specifically used in the intervention (phonemic awareness, increased reading time, new reading materials). a. What made these interventions more effective? Please give examples. What could the project have done better to support teachers’ reading instruction? b. What could the project have done better to support teachers’ reading instruction? 5. What are some of the challenges that your school’s teachers have faced in reading instruction? [Probe for examples that help them improve their teaching method.] a. Did RFS provide support to address these issues? Please explain. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 135 6. What do you think about the quality of materials provided by RFS? (Teaching guides, manuals, other) Their relevance? 7. Since the inception of the RFS program, have you noticed a difference in family and community support for children's reading? a. If so, on which aspect? b. What was RFS doing during this time that may have contributed to this difference? 8. [If these students participated in the NGO-led summer activities] Have you noticed any gaps between their performance level and that of students who did not take part in summer programs? [NPR and SSE] 9. RFS introduced a new reading instruction method—the syllabic method. What do you think about this new method compared to the overall approach? Which successes or obstacles have you observed? 10. What changes, if any, have you seen in any of the following areas in your school since 2015? a. Amount of time devoted to reading instruction b. Techniques used for reading instruction c. Number of texts in the hands of children d. Teaching children in their mother language (a language they speak and understand) e. Testing children's reading progress [If a change is mentioned, ask for an example] [If they have not been part of the project since 2015, ask]: f. Which changes have you noticed since you arrived? B. ROLE OF THE DIRECTOR [SSE & NPR] 11. Have you played a role in coaching/supporting teachers (e.g., class observation, feedback)? a. If so, have you been trained for this role by RFS? Describe your training. b. As a support, what have you done? How frequently? c. Have you noticed differences in how your teachers deliver their classes following this support? If so, please give examples. C. INCLUSION [SSE, NPR, IDCRT] 12. Are there certain types of children in your school who did not benefit from the project or some of its components? [If the director cannot answer, suggest the following examples: • Children with disabilities • Children whose mother language is not Arabic; • Children in rural or other geographically marginalized areas • Girls • Other vulnerable or marginalized children] a. Can you explain why these students did not benefit from this project? [E.g., frequently absent? Home or work responsibilities?] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 136 13. Do certain types of children still struggle with learning to read? Please describe. 14. How could these children be better served in future education and reading activities? [This question should be asked separately after each group that is mentioned in previous questions] E. LESSONS LEARNED [SSE, NPR, IDCRT] 15. As a school director, do you think that you could benefit from additional training sessions? If so, which ones? 16. Do you have any recommendations for USAID, the MoE or the schools to help improve children's reading skills in the future? 17. What about improving teaching for all children? a. What about ... 1. Children with disabilities? 2. Children whose mother language is not Arabic? 3. Children in rural areas or other geographically marginalized areas? 4. Girls 5. Other vulnerable or marginalized children? 18. Were there any unexpected outcomes or results (positive or negative)? F. CLOSING [SSE, NPR, IDCRT] Do you wish to share additional comments? Do you have any questions for our team? Thank you very much for taking the time to answer our questions. End Time: ___________________________ Total length: ________Hours _________Minutes Language of interview: __ Arabic __ French __Amazigh __ Other: Specify USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 137 POST-KII NOTES: Please comment on: • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate. • If more than one person participated, the various insights that have emerged through disagreements during the interviews. • Any additional insight or comment that should be included. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 138 Tool 6: Focus Group Discussion Guide: Teachers Final Version September 23 Qualitative Instruments: USAID RFS Performance Evaluation Introduction and Consent [Please review and complete the FGD Consent form with each participant. If agreement to participate voluntarily is obtained for each person, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participate. While reviewing and completing the consent form, also make sure you ask for permission to record the interview on your audio recorder. If all participants answer in the affirmative, please proceed. If participants answer NO, agree to not record the conversation and proceed without turning on your audio recorder.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. If you have not received the consent of all participants, do not record the discussion. Indicate in your notes if the discussion was not audio-recorded. Consent obtained: ___ YES ___ NO PERCEPTION OF THE RFS PROJECT [SSE, NPR, IDCRT schools] 1. During this discussion, we will ask you questions about your experience with USAID's RFS project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. a. What do you know about the project? [Allow open answers from participants initially.] b. What involvement, if any, have you had with this project? [Probe their role: training they took, materials they received, their activities with teachers ...] [If some are not well acquainted with the program, briefly describe to them the main activities of the project and the approach used to enhance education and reading skills among Moroccan children by citing the following ... USAID/Morocco works with the Ministry of Education (MoE) to support children's primary education in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment in this field. This work is currently taking place through the National Program for Reading (NPR) activity as well as Small Scale Experimentation A note about this tool: Purpose: This focus group discussion (FGD) guide will enable you to gather information from teachers to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of RFS project outcomes, project implementation, inclusion, and lessons learned. Recommended sources: Information should be collected through FGDs with teachers at eight RFS pilot school, two RFS scale-up schools, and two RFS-IDCRT schools. For the RFS pilot schools, participants should be grade 2-4 teachers. Up to ten teachers should participate in each FGD. FGDs will aim for gender-balanced participation where possible; for example, half of the teachers participating in FGDs will be male and half will be female. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 139 (SSE) and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT), which helps improve the reading skills of deaf and hearing-impaired children using technology.] 2. Which strategies/interventions have, in your opinion, been most effective in helping your students learn to read? [Probe for strategies specifically used in the intervention (phonemic awareness, increased reading time, new reading materials). Then ask if they wish to mention other strategies.] 3. What are some of the challenges that you, as teachers, and your colleagues face as soon as you must teach children how to read or improve their skills in this area? [Probe for examples that help them improve their teaching method.] 4. What are some of the challenges faced by your students as they were learning to read? 5. Are you aware of any other educational projects in your communities? a. If so, what can you tell me about them? A. PROJECT IMPLEMENTATION AND RESULTS [SSE, NPR, IDCRT schools] 6. If you participated in RFS-related training, did it change your teaching method? If so: a. Can you give examples of what you are now able to do, which you were unable to do previously? b. Can you comment on the effectiveness (strengths) and the challenges (weaknesses) of training? c. What type of improvements should, in your opinion, be made to training [duration, structure, content, location, etc.]? 7. Have you received new materials [for teachers] to teach reading in Arabic? If so: a. Who provided you with these materials? b. Can you comment on the effectiveness (strengths) and the challenges (weaknesses) of this approach? c. In your opinion, what type of improvements could be made to these new materials? 8. Have your students received new materials [for students] to learn reading in Arabic? If so, what kind of materials? Please describe. a. Who provided you with these materials? b. Can you comment on the effectiveness (strengths) and the challenges (weaknesses) of this approach? c. In your opinion, what type of improvements could be made to these new materials? 9. Did you receive support for teaching reading in Arabic? If so: a. Who provides you with this support, and how frequently? b. Can you comment on the effectiveness (strengths) and the challenges (weaknesses) of this approach? c. In your opinion, what type of improvements could be made to this support? 10. Have you used or participated in new evaluations for reading in Arabic? If so: a. Who introduced these evaluations? [E.g., RFS? MoE? Other?] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 140 b. Can you comment on the effectiveness (strengths) and the challenges (weaknesses) of this approach? c. In your opinion, what type of improvements could be made to these new evaluations? 11. Do you know if some of your students take part in summer activities managed by civil society organizations (CSOs)? a. If so, have you noticed any gaps between their performance level and that of students who did not take part in summer programs? B. CURRICULUM [NPR and SSE] The RFS project introduced a new method and new teaching materials, which were recently used with students attending grades 1-4 of primary school. This approach emphasizes phonetics, or helps readers understand how letters match sounds and identify some common pronunciation structures. 12. Based on your experiences, what do you think about the use of this method in your teaching, compared to the previous approach? a. Can you describe the various issues that you experienced in using the new curriculum and/or the new materials? [E.g., unnecessary or insufficiently relevant aspects; some issues such as the lack of adequate training; additional teacher requirements/increased work burden; or the pace of lessons plans] 13. Have you noticed any differences in your students’ capacity to read in Arabic during the implementation of the RFS project? If so, please give examples. a. Have you noticed different progress levels in the following areas? 1. Decoding (The ability to map letters to sounds) 2. Reading out fluently (capacity to read orally, with speed, precision and an adequate expression) 3. Reading with comprehension (The ability to actively engage with, and derive meaning from, the texts that are read)? 14. [This question must be asked of teachers working in the school for at least three years. If the teachers have not taken part in the project since 2015, try to find out if they have noticed changes since they joined the project and indicate the appropriate duration in your field notes.] For those of you who have worked in this school for at least three years, have you noticed any changes in the following areas since 2015? a. Amount of time devoted to reading instruction b. Techniques used for reading instruction c. Number of texts in the hands of children d. Teaching children in their mother language (a language they speak and understand) e. Testing children's reading progress [If changes were noticed] f. Which ones? g. Do you think that the RFS project has led to these changes? How so? C. INCLUSION [For SSE, NPR, IDCRT schools] USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 141 15. Have you noticed different progress levels for different types of students, such as: a. Children with disabilities? b. Girls or boys? c. Disadvantaged or marginalized students? d. Less successful students? e. Students who do not speak Arabic at home? 15. [If they know the project or not] Are there some types of students in your community who still struggle to learn reading in primary school? Which ones in particular? [Refer to the types of children in the previous question to seek examples.] a. Are some students frequently absent? b. Are some students not focused on their studies? c. For all these students, what are the hurdles? 16. Are some students not at all enrolled? Which ones? [Refer to the types in the previous question to seek examples.] In your opinion, why? 17. How could these children be better served within educational and reading activities in the future? [Refer to the types of struggling children mentioned by the parents.] D. LESSONS LEARNED [EVERYONE] 18. Do you have any other recommendations to improve the children's reading skills in the future? 19. Were there any unexpected outcomes or results (positive or negative)? E. CLOSING We have reached the end of our questions. Do you wish to share additional comments? Do you have any questions for our team? Thank you very much for taking the time to answer our questions. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 142 Tool 7: Focus Group Discussion Guide: Parents Final Version September 23 Qualitative Instruments: USAID RFS Performance Evaluation Introduction and Consent [Please review and complete the FGD Consent form with each participant. If agreement to participate voluntarily is obtained for each person, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participate. While reviewing and completing the consent form, also make sure you ask for permission to record the interview on your audio recorder. If all participants answer in the affirmative, please proceed. If participants answer NO, agree to not record the conversation and proceed without turning on your audio recorder.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. If you have not received the consent of all participants, do not record the discussion. Indicate in your notes if the discussion has not been audio-recorded. Consent obtained: ___ YES ___ NO A. PERCEPTION OF THE RFS PROJECT [SSE, NPR, IDCRT schools] Introductory Part: USAID/Morocco works with the Ministry of Education (MoE) to support children's primary education in Morocco. This work aims to improve children’s reading skills and is implemented in schools, within the community and through government policy. USAID's support is provided through the Reading for Success project to improve the reading skills of Moroccan students. We are going to ask a few questions about this program. 1. During the last year, have you noticed changes in your children's attitude towards school? Please describe. 2. During the last year, have you noticed changes in your children's attitude towards reading? Please describe. About this tool: Goal: This focus group discussion (FGD) guide will enable you to gather information from teachers to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of RFS activities, project outcomes, the relevance and effectiveness of specific activities, project implementation, inclusion, and lessons learned. Recommended Sources: Information must be collected by conducting FGDs with parents in eight RFS project pilot schools, two schools included in scaling up the project, and two RFS-IDCRT schools. For RFS pilot schools, participants must be parents of students enrolled in primary school (grade 2-4), including a crosscutting section of parents of highly performing and low performing students. Up to ten parents can take part in each FGD. FGDs will aim for gender-balanced participation where possible. For example, half of parents of primary school students participating in FGDs will be fathers/male custodians, while the other half will be mothers/female custodians. Only one of the parents or legal custodians will be expected from each family. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 143 3. During the last year, have you noticed a change in the time your primary school child spent reading? Please describe. 4. Have you or another family member spent time reading to your children? (E.g., follow up, reads to the children, asks children to read) Why or why not? [Is there any difference compared to the previous year?] 5. In your opinion, what factor is most helpful for your children to learn to read? 6. What are the biggest challenges faced by your children in learning to read? 7. Are you familiar with some projects related to children's reading in you school or community? If so, which ones? The Reading for Success project is made up of a variety of activities, including the National Program for Reading (NPR) activity, as well as Small Scale Experimentation (SSE) and, previously, the Human and Institutional Capacity Development (HICD) initiative. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT), which helps improve the reading skills of deaf and hearing-impaired children using technology.] 8. Have you heard about the Reading for Success (RFS) project or the National Program for Reading (NPR)? 9. Did your children attend the summer program? (Reading classes and activities during the long summer break) [IDCRT schools] 10. Did your children attend the IDCRT program? If so, what do you think about: a. The use of technology to develop a software (named Clip and Create) to help teachers and parents easily create and publish educational materials in the Moroccan sign language? b. The training of teachers, schools, directors, parents, members of the Association for the Deaf and Hearing Impaired, and other stakeholders in terms of handling the software and educational reading strategies for deaf and hearing-impaired students starting primary school? [For SSE, NPR] 11. Did you receive information about the new reading instruction method in this school? If so, which ones? B. RELEVANCE/EFFECTIVENESS [For SSE, NPR] 12. [If they are familiar with the project] Do you think that RFS affects the approach used by your school to teach reading? a. Alternatively: For those of you who have children, has this school's capacity to teach reading to the children changed? b. Why or why not? Can you give specific examples? C. INCLUSION [For SSE, NPR, IDCRT schools] 13. [If they are familiar with RFS] Were these effects the same for all the following students? [Probe: Mention the following categories: Children with disabilities Children whose mother language is not Arabic Children in rural or other geographically marginalized areas Girls Other vulnerable or marginalized children USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 144 14. [Whether they are familiar with the project or not] Are there some types of students in your community who still struggle to learn reading in primary school? Which ones in particular? [Refer to categories of children in the previous question to seek examples.] d. Are some students frequently absent? e. Are some students not focused on their studies? f. For all these students, what are the obstacles? 15. Are some students not at all enrolled? Which ones? [Refer to categories in the previous question to seek examples.] (If there is enough time), Why? 16. How could these children be better served in future education and reading activities? [Refer to the categories of struggling children mentioned by the parents.] Are any extra-curricular activities closed to certain students or not attended by them? For example, summer educational programs. If so, why? E. LESSONS LEARNED 17. Do you have any other recommendations to improve children's reading skills in the future? 18. Were there any unexpected outcomes or results (positive or negative)? D. CLOSING We have reached the end of our questions. Do you wish to share additional comments? Do you have any questions for our team? Thank you very much for taking the time to answer our questions. F. Summary and Conclusion End Time: ___________________________ Total length: ________Hours _________Minutes Language of interview: __ Arabic __ French __Amazigh __ Other: Specify POST FOCUS GROUP NOTES: Please comment on: • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate. • Any additional insight or comment that should be included. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 145 ANNEX IV: SOURCES OF INFORMATION NUMBER OF INDIVIDUALS INTERVIEWED In order to maintain the confidentiality of respondents, the total number of interviewees by relationship to the RFS project is provided in place of the name and specific titles of individuals interviewed. Table 10. KII qualitative sample by respondent type, subgroup, and number of respondents by sex Respondent type Subgroups Total number of respondents (by sex) USAID/Morocco officials N/A 3 (m: 1; f: 2) USAID IPs and subcontractors XXXXXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 1; f: 2) XXXXXXXXXX 5 (m: 4; f: 1) XXXXXXXXXXXXXXXXXX 6 (m: 3; f: 3) MoE officials XXXXXXXXXXXXXXXX 9 (m: 7; f: 2) XXXXXXXXXXXX 10 (m: 10; f: 0) XXXXXXXXXXX 10 (m: 8; f: 2) XXXXXXXXXXXXXXXXXXXX 16 (m: 13; f: 3) XXXXXXXXXXXXXXXXXXXXX 3 (m: 3; f: 0) XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX 8 (m: 5; f: 3) School directors XXXXXXXXXXXXXXXXXXXXXXXX 8 (m: 8; f: 0) XXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 2; f: 1) XXXXXXXXXXXXXXXXXXXXXXXX 3 (m: 2; f: 1) Total number of key informant interviews 87 (m: 67; f: 20) USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 146 NUMBER OF FOCUS GROUP DISCUSSIONS (FGDS) COMPLETED Table 11. FGD participants by respondent type, subgroup, number completed, and number of participants by sex Respondent type Subgroups Number of FGDs Completed Approx. # participants Teachers XXXXXXXXXXXXXXXXXXXXXXXXX 8 58 (m: 26; f: 32) XXXXXXXXXXXXXXXXXXXX 2 12* (m: 4; f: 2) XXXXXXXXXXXXXXXX 2** 2 (m: 0; f: 2) Total teacher FGDs 12 71* (m: 30; f: 35) Parents XXXXXXXXXXXXXXXXXXXXXXXX 8 67 (m: 23; f: 44) XXXXXXXXXXXXXXXX 0*** 0 (m: 0; f: 0) Total parent FGDs 8 67 (m: 23; f: 44) Total FGDs 20 138* (m: 53; f: 79) *Please note the discrepancy between the total number of respondents and sex disaggregated figures is due to missing data on six participants’ sex from one teacher scale-up FGD. The teacher respondents from RFS-NPR scale-up scales is due to missing information. ** In some cases, only one teacher was available during both of the IDCRT school visits. While these may be understood as interviews, the figures are provided here to align with respondent types. *** At one visited RFS-IDCRT school, the only parent in attendance at the FGD was the parent of an 18-year old student who was too old to have participated in the activity. At another visited RFS-IDCRT school, no parents attended. At a third attempted visit, evaluators interviewed an additional president of a Deaf association, who is counted among the KIIs above, in lieu of meeting with parents. SOURCES REVIEWED Banyan Global. USAID/Morocco Gender Analysis (Final) 2018, (2018). Bensoukas, Karim, Language Policy and Mother Tongues in Morocco: A Linguistic Human Rights Perspective. In El Kirat, Y. (ed.), (2010). Globalization and Mother Tongues in Africa, 135- 152. Rabat: Publications of FLSH. Chemonics International, Inc., and School-to-School International, USAID Reading for Success-Small Scale Experimentation Early Grade Reading Assessment Endline Report, (2017). Chemonics International, Inc., USAID Reading for Success-Small Scale Experimentation Final Report. (2018). Creative Associates International, Inc., USAID Reading for Success-National Program for Reading Annual Report Year 2, Including Quarterly Report 4, Fiscal Year 2018, (2018). —, USAID Reading for Success-National Program for Reading Monitoring, Evaluation, and Learning Plan (MELP)—Draft for Discussion, (2018). —, USAID Reading for Success-National Program for Reading Quarterly Report 1, Fiscal Year 2018, (2018). —, USAID Reading for Success-National Program for Reading Quarterly Report 1, Fiscal Year 2019, (2019). USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 147 —, USAID Reading for Success-National Program for Reading Quarterly Report 3, Fiscal Year 2018, (2018). —, USAID Reading for Success-National Program for Reading Quarterly Report 4, Year 1, (2017). —, USAID Reading for Success-National Program for Reading Quarterly Report 4, Year 1, (2017). —, USAID Reading for Success-National Program for Reading Quarterly Report—May 19, 2017, to June 30, 2017, (2017). Democracy International. USAID Morocco Development Context Assessment 2018, (2018). High Commission for Planning (Morocco). Morocco Population and Housing Census 2014. High Commission for Planning (Morocco). Haut-Commissariat Au Plain, Indicateurs RGPH2014. Accessed November 2019 http://rgphentableaux.hcp.ma/Default1/ Gove, Amber and Peter. Early Reading: Igniting Education for All. A report by the Early Grade Learning Community of Practice, (2010). Institute for Disabilities Research and Training, Inc. Reading for Success-Improving Deaf Children’s Reading Through Technology Final Report, (2018). Kim, Young-Suk G., Boyle, Helen N., Simmons, Stephanie, & Nakamura, Pooja. Landscape Report on Early Grade Literacy, (2016). Kingdom of Morocco Ministry of Culture and Communication—Department of Communication, “Royal Speeches: HM the King addresses a speech to the Nation on the 59th anniversary of the Revolution of the King and the People,” accessed April 2019, http://www.maroc.ma/en/royal-speeches/hm-king-addresses-speech-nation-59th-anniversary￾revolution-king-and-people Kingdom of Morocco Ministry of Culture and Communication—Department of Communication, “Royal Speeches: Speech of His Majesty the King to Nation on Occasion of 60th Anniversary of Revolution of King and People,” accessed April 2019, http://www.maroc.ma/en/royal￾speeches/speech-his-majesty-king-nation-occasion-60th-anniversary-revolution-king-and￾people Kingdom of Morocco Ministry of National Education, Vocational Training, Higher Education, and Scientific Research, Vision 2030, accessed April 2019, https://www.men.gov.ma/Fr/Documents/ Vision_ strateg_CSEF16004fr.pdf Mullis, I.V.S., Martin, M.O., Foy, P., & Hooper, M. PIRLS 2016 International Results in Reading. (Chestnut Hill, Mass.: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement, 2017). RTI International and Al Akhawayn University, Research on Reading in Morocco: Analysis of the National Education Curriculum and Textbooks—Final Report (Component 1)—Part 1 (Curriculum Analysis), (2015). —, Research on Reading in Morocco: Analysis of the National Education Curriculum and Textbooks—Final Report (Component 1)—Part 2 (Textbook Analysis, Part A), (2015). USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 148 —, Research on Reading in Morocco: Analysis of the National Education Curriculum and Textbooks—Final Report (Component 1)—Part 2 (Textbook Analysis, Part B), (2015). RTI International and Varlyproject, Research on Reading in Morocco: Analysis of Initial Teacher Training— Final Report (Component 2), (2014). RTI International, Student Performance in Reading and Mathematics, Pedagogic Practice, and School Management in Doukkala Abda, Morocco, (2012). RTI International, Situation and Needs Assessment for Students who are Blind/Low Vision or Deaf/Hard of Hearing in Morocco, (2016). School-to-School International, Evaluation Report: Moroccan Sign Language Assistive Technology for Reading Improvement of Children who are Deaf/Hard of Hearing, (2018). The Kaizen Company, Reading for Success-Human and Institutional Capacity Development Monitoring, Evaluation, and Learning Plan (MELP)—Final Report, (2017). The Kaizen Company, Reading for Success-Human and Institutional Capacity Development—Executive Summary for the Department of National Education, (2017). The Kaizen Company, Reading for Success-Human and Institutional Capacity Development—Final Report for the Department of National Education, (2017). The World Bank, “Education Statistics (EdStats): Country at a Glance—Morocco,” accessed April 2019, http://datatopics.worldbank.org/education/country/morocco. The World Bank, “Program Info Document (PID)—Morocco Education Support Program,” accessed April 2019, http://documents.worldbank.org/curated/en/992621537725308548/pdf/Concept￾Stage-Program-Information-Document-PID-Morocco-Education-Support-Program￾P167619.pdf. World Economic Forum. The Global Gender Gap Report, (2017). UNICEF, “UNICEF Global Databased—Education: Adult Literacy Rates—Percentage,” accessed April 2019, https://data.unicef.org/wp-content/uploads/2016/05/education_table-youth-and￾adult-literacy-rate-updated-oct.-2015.xlsx. USAID/Morocco, USAID/Morocco Country Development Cooperation Strategy—2013-2017, (2013). —, Reading for Success-National Program for Reading Request for Task Order Proposal no, SOL-608-16- 000006: Section C—Description/Specifications/Statement of Work, (2016). —, USAID/Morocco/Education Project Appraisal Document, (2013). USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 149 ANNEX V: DETAILS ON CROSS-COUNTRY EVALUATION RESULTS PRESENTED IN FINDINGS110 Country Program (evaluation start year) Evaluation length (school years) Effect size Assessment language(s) Grade(s) Donor Democratic Republic of Congo PAQUED (2012) 1 0.13 French 4 USAID Tonga PEARL (2015) 1 0.24 Tongan 2 GPE Kyrgyz Republic USAID Reading Quality Project (2014) 1 0.27 Russian, Kyrgyz 2 and 4 USAID Papua New Guinea Reader Booster (2011, 2013) 2, 3 0.3 English 3 and 4 GPE Mali RLL (2009) 3 0.33 Bamanankan, Bomu, Fulfulde, Songhai 1 and 2 Hewlett Uganda SHRP (2013) 4 0.35 Luganda, Leblango, Ateso, Ruyankor e-Rukiga, English, Runyoro-Rutooro, Acoli, Lugbarati, Lumasaaba 1-3 USAID Kenya PRIMR (2012) 2 0.41 English, Kiswahili 1 and 2 USAID Jordan National Early Grade Literacy and Numeracy Survey (2013) 1 0.46 Arabic 2 USAID Egypt GILO (2009) 3 0.55 Arabic 2 USAID Nigeria RARA (2014) 1 0.66 Hausa 2 USAID Liberia EGRA Plus (2008) 2 0.8 English 1-3 USAID South Africa SMRS (2009) 0.5 0.8 Sepedi, Zulu, Setswana 1 USAID 110 Graham, J., & Kelly, S. (2018). How effective are early grade reading interventions? A review of the evidence. The World Bank. Retrieved from https://openknowledge.worldbank.org/handle/10986/29127. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 150 ANNEX VI: RFS EVALUATION SUMMARY MATRIX Recommendations Source of Recommendation Relevant Findings for Each Recommendations LESSON 1: MOE OWNERSHIP: Additional measures should be taken to strengthen MoE’s ownership of the reform, including clarification of roles, improved information sharing, and negotiation of priorities within contractual requirements. CONCLUSION 1: Evaluators observed a high degree of MoE ownership with RFS, including USAID promoting the role of the MoE as the driver of the project; however, if the MOE is the principal driver of RFS, not all strategies worked as intended, and information was not always shared sufficiently. 1.1 As RFS extends to all schools in Morocco and upward through grades 5 and 6, spell out the roles more clearly to be played by MoE personnel and departments at central and provincial levels in areas of program design, development, and implementation, and ensure these roles are performed. Evaluators determined this recommendation in response to gaps identified in rollout and planning, concerns from implementers and other stakeholders such as teacher trainers around the rate and speed at which roll out/expansion was taking place on the quality of implementation as well as in response to local-level MoE officials recommendations for increased engagement and clarity of their role with RFS. Factors identified as contributing to MoE ownership included: • Charging various elements and levels of the MoE with rollout and monitoring of RFS, • Working through existing MoE structures, • Involving several MoE departments in ongoing program design, • Establishing an RFS Steering Committee to provide quality assurance. • Provincial MoE offices reported uneven participation in RFS operations, noting that their role was to “to take their orders from the top.” 1.2 Identify, mobilize, and deploy resources required to roll-out RFS nationally at each grade level, and the role to be played by each MoE personnel and departments in coordinating training and managing resources. Evaluators determined this recommendation in response to observed gaps in understanding and clarity about roles, ownership and implementation of activities by the MoE, especially at the provincial and regional levels. This will be critical for supporting MoE ownership, quality of roll out, and sustainable impact in the future. Factors identified as contributing to MoE ownership included: • Charging various elements and levels of the MoE with rollout and monitoring of RFS, • Working through existing MoE structures, • Establishing an RFS Steering Committee to provide quality assurance • Provincial MoE offices reported uneven participation in RFS operations, noting that their role was to “to take their orders from the top.” 1.3 Enable the RFS Steering Committee to provide program design review, monitoring, and quality assurance. Evaluators determined this recommendation in recognition that in order for the RFS activities and outcomes to be truly sustainable, there is a need for the MoE centers involved in the committee to go beyond a consultative role, and instead provide a more active and consistent role in the program design, monitoring and quality assurance. While the Steering Committee played a central role early on, it had not met for some months prior to this evaluation. Charging various elements and levels of the MoE with rollout and monitoring of RFS was identified as a factor contributing to MoE ownership. 1.4 Establish a mechanism for sharing information with MoE personnel at central and provincial levels on a timely basis. Evaluators determined this recommendation in recognition that for MoE to truly be in the driver seat, they need access to information, reports and data from activities in a consistent, and timely manner (a gap observed by evaluators during data collection and analysis). Some research reports produced by RFS, including EGRA reports of reading progress, failed to reach the MoE. Involving the MoE in initial RFS research was identified as a factor contributing to MoE ownership. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 151 Recommendations Source of Recommendation Relevant Findings for Each Recommendations 1.5 Ensure that any additional institutional capacity development efforts reflect the evolving systemic nature of the MoE and include provisions for involving the MoE in developing strategies to strengthen these systems. Evaluators determined this recommendation in response to the gaps, weaknesses, and successes of the HICD activity identified by select MoE officials. The HICD assessment, though useful to some, was seen by some as too “siloed” and not sufficiently reflective of the MoE’s evolving, systemic structure. LESSON 2: IMPLEMENTATION: Respondents universally recognized the excellent quality of the RFS reading program; however, as the program expands to grades 5–6 in all schools in Morocco, careful consideration must be given to how it is scaled. CONCLUSION 2: Stakeholders praised new teaching methodology and materials for transforming teaching, animating classrooms, and producing impressive reading results; however, evaluators found considerable variation between the degrees of support received by schools across provinces as well as other obstacles in delivering the new methodology and materials as effectively as possible. 2.1 Define a minimum package of support to be provided to schools in each activity, including modalities for teacher and school director support and minimum levels of training and materials. Evaluators provided this recommendation based on the lack of clarity or clear definition of the type and level of support or package schools were receiving around the 5 Ts, for example, lack of clarity around number and type of texts distributed as well as number and type of trainings conducted—both prior to and during the SSE and NPR intervention. It's essential to have a clear understanding of the package of support in order to be able to measure the impact of an activity. This recommendation was also informed by evaluator expertise and experience in global education sector. While SSE teachers and schools had received extensive training and material support, many respondents from the NPR scale-up schools reported having received little—or in some cases no—support. 2.2 Develop and sustain a dynamic plan for scaling that is updated and complied to by all relevant stakeholders; delineates the roles of key actors by grade, year, and province; and identifies resources required for scaling, including transportation, communication, finance and technical assistance. Evaluators determined this recommendation in response to observed gaps in understanding and clarity about roles, ownership and implementation of activities by the MoE, especially at the provincial and regional levels. This will be critical for supporting MoE ownership, quality of roll out, and sustainable impact in the future. This recommendation was also informed by evaluator expertise and experience in global education sector. While SSE teachers and schools had received extensive training and material support, many respondents from the NPR scale-up schools reported having received little—or in some cases no—support. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 152 Recommendations Source of Recommendation Relevant Findings for Each Recommendations 2.3 Establish indicators for donors and the MoE to use to determine the extent to which the objectives of the scale-up are being met. This recommendation was mentioned by some of the MoE central office respondents. One respondent noted that while the MoE has their own strategic plan and indicators, they would like help in developing a more cohesive approach to measuring impact. The evaluators provided this recommendation in recognition that providing increased accountability and relevancy of indicators would also strengthen the MoE's role as the driver of the activity, and in turn, support ownership and sustainability. This recommendation was also informed by evaluator expertise and experience in global education sector. While SSE teachers and schools had received extensive training and material support, many respondents from the NPR scale-up schools reported having received little—or in some cases no—support. 2.4. Conduct further research on the reasons underlying the limited impacts of SSE. This recommended is derived from the quantitative comparison of SSE impact effect sizes relative to other projects. The SSE program’s impact on oral reading fluency—considered “the bridge between decoding and comprehension”—is considerably below impacts conventionally considered to be substantively meaningful. 2.5 Ensure that sustainability is at the center of discussions. Evaluators provided this recommendation in response to concerns raised by implementers that the speed and pace required for roll-out of activities impeded their ability to sufficiently engage the leadership of MoE and ensure their approach systematically engaged and utilized existing MoE systems in a way that implementation could be sustained by the MoE in the future. This recommendation was also informed by evaluator expertise and experience in global education sector. Informed by evaluator expertise and experience in global education sector. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 153 Recommendations Source of Recommendation Relevant Findings for Each Recommendations LESSON 3: INCLUSION: Important strides ensure all children are included in public education. However, vulnerable populations and children with special needs will not have sufficient access to quality education until (1) strategy defines how to address each child’s needs, (2) National effort mobilizes support for inclusive education at school, in the community, and at home CONCLUSION 3: Impressively high levels of access at the primary level across the country were noted at the primary level; however, obstacles continue to hinder greater participation of vulnerable populations, including children with learning needs. 3.1 Develop a national strategy for increasing the MoE’s capacity to provide inclusive education over time. Evaluators provided this recommendation in response to implementers citing the lack of a cohesive strategy— including purposeful and/or actionable plan for implementing new inclusion policies within schools and communities—from the MOE as a bottleneck for implementation of inclusive programming as well as MoE ownership and sustainability. Obstacles identified as hindering greater access included: • Children with special needs and vulnerable children are often stigmatized, and no official strategy has yet been developed to counter these attitudes. • Meeting the needs of each of these populations can be complex and daunting. • MoE ‘s policy requiring all schools to provide access to children with special needs does not require schools to staff the classroom with personnel specifically trained to teach children with special needs, nor does it require that schools equip these classrooms with appropriate equipment, materials, or infrastructure. 3.2 Develop an inclusive education plan to be shared with development partners ensuring political, financial, and technical support for the rollout of the national strategy, including needs￾specific staffing and training and the provision of appropriate infrastructure and material support. Evaluators provided this recommendation in response to implementers citing the lack of a cohesive strategy—including purposeful and/or actionable plan for implementing new inclusion policies within schools and communities—from the MOE as a bottleneck for implementation of inclusive programming as well as MOE ownership and sustainability. Obstacles identified as hindering greater access included: • Children with special needs and vulnerable children are often stigmatized, and no official strategy has yet been developed to counter these attitudes. • Where learning centers do exist for children with special needs, they are often privately managed, and no mechanism yet exists for the MoE to provide oversight or ongoing support, or for the MoE to assume the role of provider. • MoE ‘s policy requiring all schools to provide access to children with special needs does not require schools to staff the classroom with personnel specifically trained to teach children with special needs, nor does it require that schools equip these classrooms with appropriate equipment, materials, or infrastructure. • Meeting the needs of each of these populations can be complex and daunting. 3.3 Develop an inclusive education communication strategy to enable the MoE and its partners to implement effective outreach strategies. Evaluators provided this recommendation in response to stakeholders—including across MoE levels from the school to central offices—citing the need for awareness raising to reduce stigmatization of children with disabilities as well as the need for increased sensitization for support the education of children with disabilities—including parents. Obstacles identified as hindering greater access included: • Children with special needs and vulnerable children are often stigmatized, and no official strategy has yet been developed to counter these attitudes. • The MoE established a policy requiring all schools to provide access to children with special needs and created an Inclusive Education Unit to oversee the development if initiatives to include vulnerable populations. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 154 Recommendations Source of Recommendation Relevant Findings for Each Recommendations 3.4 Strengthen the technical understanding and capacity of the MoE to implement its inclusive education strategy. Evaluators provided this recommendation based on discussions with stakeholders— including requests for support, as well as findings and recommendations from existing evaluations and reports on inclusion in Morocco. This recommendation was also informed by evaluator expertise and experience in global education sector. Obstacles identified as hindering greater access included: • MoE’s policy requiring all schools to provide access to children with special needs does not require schools to staff the classroom with personnel specifically trained to teach children with special needs, nor does it require that schools equip these classrooms with appropriate equipment, materials, or infrastructure. • Meeting the needs of each of these populations can be complex and daunting. LESSON 4: RESEARCH, MONITORING & EVALUATION: USAID and the MoE have conducted research on their programs and tracked progress based on results frameworks and national indicators. However, to use the information from these efforts to inform decision-making routinely, M&E systems and research processes require strengthening. CONCLUSION 4: Positive examples identified in the evaluation study of RFS utilizing research, monitoring and evaluation over the course of the project; in spite of these efforts, however, the sharing of data has often been limited, and potential avenues for discussing results across donors and MoE departments have not been fully exploited. 4.1 Develop an information strategy to be used by USAID/Morocco and the MoE to establish a common research, monitoring, and evaluation framework. Evaluators provided this recommendation in consideration of stakeholders citing the need to better integrate project data with MOE as well as in recognition of the need to strengthen MOE engagement in quality assurance and monitoring to support ownership and sustainability. • Mechanisms often did not exist to ensure results of studies (e.g., EGRA) were made available in a timely manner or shared in French or Arabic. • Systems have not yet been linked, including USAID’s indicators and data and those of the MASSAR system. 4.2 Where appropriate, support efforts to “collaborate, learn, and adapt” (CLA) among USAID implementers, the MoE, and partners. Evaluators provided this recommendation in response to IPs citing the need for more extensive sharing of the findings from implementation, experimentation phases with MOE as well as best practices to support MOE ownership and sustainability. This recommendation was also informed by evaluator expertise and experience in global education sector. • Mechanisms often did not exist to ensure results of studies (e.g., EGRA) were made available in a timely manner or shared in French or Arabic. 4.3 Strengthen the capacity of the MoE and IPs to conduct research, monitoring, and evaluation. Evaluators provided this recommendation in consideration of stakeholders citing the need to better integrate project data with MOE (and their existing systems such as MASSAR), as well as in recognition of the need to strengthen MOE engagement in quality assurance and monitoring to support ownership and sustainability. • While some interviewees noted that RFS had provided opportunities to learn about research—e.g., how to understand analyses and interpret EGRA data—such opportunities have been rare. • Shifts in SSE and NPR project priorities and scope have limited the amount of time available for purposeful engagement and reflection. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project–Morocco | 155 Recommendations Source of Recommendation Relevant Findings for Each Recommendations LESSON 5: COMMUNITY ENGAGEMENT: At both central and provincial levels, respondents stressed the importance of education outside the classroom, including the need for materials and activities to engage parents and communities. However, RFS’s scope limited support for parental and community engagement, raising questions about the importance and effectiveness of such initiatives. CONCLUSION 5: The activities geared toward community engagement that RFS implemented, notably summer reading enrichment activities, were successful, but their impact was narrow due to limited implementation and little focus from project implementers and the MoE; overall, respondents expressed desire for more community engagement in schools and with parents. 5.1 Conduct a landscape review to identify effective strategies for supporting early grade reading learning outside the classroom, both within and outside Morocco. While some specific recommendations for approaches to engage community members were provided by stakeholders, the evaluators recommended a more thorough landscape review be conducted. This recommendation was also informed by evaluator expertise and experience in global education sector. While the summer enrichment program supported by RFS aimed to address this need and, by many reports, was successful in motivating children to try to read, the overall impact was limited. Several factors were identified as limiting the success of these programs, including: • Misalignment with MOE priorities, • Limited size of the beneficiary population, and 5.2 Investigate the political will of the MoE and potential partners— including parents associations, CSOs, private sector—to support extracurricular activities to support early grade reading. Evaluators provided this recommendation for further investigation given increased parental/community engagement was recognized across stakeholders—especially school￾level and local MOE officials—as an integral component to supporting reading and education—while at the same time a lack of MOE engagement on RFS community-based activities was observed as well as other stakeholders perceptions that it is a low priority for the MOE. This recommendation was also informed by evaluator expertise and experience in global education sector. Several factors were identified as limiting the success of these programs, including: • Misalignment with MOE priorities, • Competing demands for time and resources on the part of implementers. • Implementation (of the summer enrichment programs) through local CSOs reduced the likelihood that efforts like these could be sustained over time is limited. Interviewees from school directors to teachers to parents frequently cited the need for libraries, extracurricular activities and the need to educate parents about supporting their children in reading and equipping them with materials and strategies to do so. 5.3 If needed, develop a model in collaboration with the MoE for parent￾and community-based support for early grade reading that can be deployed at the provincial level. Evaluators provided this recommendation in response to parents and teachers expressing the need for more direct engagement from parents and community members in supporting program activities and their children's education—for example, increasing or leveraging the support of PTAs in programming. Evaluators also note drawing on existing school support structures are more likely to be sustainable. This recommendation was also informed by evaluator expertise and experience in global education sector. Interviewees from school directors to teachers to parents frequently cited the need for libraries, extracurricular activities and the need to educate parents about supporting their children in reading and equipping them with materials and strategies to do so. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 156 ANNEX VII: EVALUATION STAFF QUALIFICATIONS VARUNI DAYARATNA (NORC) As Project Director of the USAID Reading & Access contract under which the present evaluation was undertaken, Varuni Dayaratna was responsible for managerial oversight of the project and served as the point of contact to the Reading & Access COR. She also served as the senior reviewer of all deliverables produced as part of the present evaluation. Ms. Dayaratna is Vice President and Associate Director for International Programs at NORC. She is a seasoned program manager, with over 20 years of experience working in the international development field and over 15 years of experience managing research, evaluation, and technical assistance projects. Ms. Dayaratna has extensive experience and expertise in international education and early grade reading, having managed evaluations in countries such as Ethiopia, Zambia, Nepal, Liberia, Tanzania, Malawi, Nepal, Peru, and South Africa. She also served as Project Director for the performance and impact evaluation of USAID’s School Health and Reading Program in Uganda. Ms. Dayaratna holds a Master’s degree in international affairs from Princeton University. SAMANTHA DOWNEY (NORC) Samantha Downey contributed to the coding and analysis of the qualitative data collected in the field, and oversaw the 508 compliance process for all deliverables. As a Research Analyst at NORC, she provides technical and logistical support to discrete evaluation projects as well as to Mission-wide M&E platforms, including research design, data collection and analysis, programming of web- and tablet-based surveys, quantitative and qualitative instruments development, planning and implementing a wide range of data collection activities and contributing to the preparation of proposals, reports, and presentations. Ms. Downey holds a B.A. in Cultural Anthropology from the University of Pennsylvania and speaks intermediate French. ERIKA KEAVENEY (NORC) Erika Keaveney provided technical and quality oversight to the evaluation including design report development and reviewing key deliverables. In addition, she led supplemental quantitative data analysis. Ms. Keaveney is a Research Scientist at NORC and an impact evaluation and quantitative research expert. She has provided technical and managerial oversight to large, high-profile research projects for clients/donors including USAID, United States Department of Labor, Millennium Challenge Corporate (MCC), and the World Bank and has led several large-scale evaluations of USAID education programs in Africa. Ms. Keaveney holds an M.A. in International and Comparative Education from Stanford University and a B.A. in Political Science from University of California, San Diego. MARK LYND (STS) Mark Lynd was responsible for supervising the quality of the evaluation design and execution, including qualitative data collection and analysis, identifying and disseminating of findings, lessons learned, and evidence from evaluations. He also supervised a team of data collectors during fieldwork. With over 30 years’ experience in international education and development, Dr. Lynd brings in-depth skills and knowledge for the design and implementation of large-scale education initiatives. He has served as the education and assessment advisor on numerous USAID Basic Education Projects in countries across sub-Saharan Africa. His experience includes the design and implementation of social research and survey tools, Early Grade Reading Assessment (EGRA), impact USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 157 evaluations, quasi-experimental studies, monitoring and evaluation systems, and fidelity of implementation research in multiple and varied contexts across the globe. He is a former educator and skilled in working with various levels of stakeholders to build capacity in research methods and data utilization for policymaking. He is a native English speaker and fluent in French. CASEY MCHUGH (STS) Casey McHugh was responsible for advising during the evaluation design and the data analysis and report writing process on qualitative components and gender issues. Ms. McHugh is an international development professional with over ten years’ experience in gender, international education, and monitoring and evaluation for international donor-funded education activities. Ms. McHugh is an applied anthropologist with experience in social research, survey design, indicator development, monitoring and evaluation system development for USAID and DFID. She is skilled in working with stakeholders to identify strength and weakness in existing M&E systems and programming around gender and social inclusion. Ms. McHugh holds an M.A. in Anthropology with a concentration in International Development from The George Washington University. ALEXANDRE MONNARD (NORC) Alexandre Monnard co-led the training for data collectors and supervised a team of data collectors during the evaluation fieldwork. He also substantially contributed to the development of KII and FGD guides, the coding and analysis of qualitative data, and the writing of the present report. Mr. Monnard is a Research Scientist at NORC and an evaluation specialist with more than ten years of experience designing, implementing, and overseeing mixed-methods evaluations across a broad range of themes, including economic development and small business support, entrepreneurship and livelihood development, food security, and combating child labor and gender-based violence. He holds a Master’s in Public Policy from the College of William & Mary and is a native French speaker. MATTHEW MURRAY (STS) Matthew Murray leads the overall management of the evaluation. Mr. Murray is a program manager and evaluator with over eight years of experience in monitoring, evaluation, and learning in development settings. He has managed education and early childhood development projects funded by DFID, USAID, the World Bank, and education agencies in sub-Saharan Africa, the Middle East, and Central Asia. Mr. Murray has managed and provided technical advice on study design, sampling, electronic data collection, analysis, and reporting for quantitative and qualitative learning assessments, performance evaluations, and cost-effectiveness analysis. Mr. Murray holds an M.A. in International Affairs from George Washington University, studied Arabic in Rabat, Morocco under a State Department-funded Critical Language Scholarship, and speaks intermediate Arabic, French, and Turkish. SELENE RANGEL (STS) Selene Rangel was responsible for support during project startup and evaluation design. Ms. Rangel is an international education professional with four years of experience in the management of large￾scale evaluations for USAID-funded projects in the Democratic Republic of the Congo, Djibouti, Mali, Morocco, and Niger. Ms. Rangel is skilled in supporting literacy assessments in multiple country and linguistic contexts, including EGRA tool adaption in Arabic, piloting, enumerator training in French, and data collection oversight. She also brings her experience in project management providing critical contractual, financial, managerial support to multi-million-dollar projects. She holds a Bachelor of Science in Foreign Service from Georgetown University with a focus on Regional and USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 158 Comparative Studies between Latin America, North Africa, and the Middle East. She is a native English and Spanish speaker, fluent in French, and has a working knowledge of Modern Standard Arabic, Moroccan Arabic, and Italian. DREW SCHMENNER (STS) Drew Schmenner was responsible for qualitative analysis and writing. Mr. Schmenner is a former journalist with over a decade of professional experience. He is a skilled problem solver, talented report writer, has excellent attention to detail, and masters complex logistics. Mr. Schmenner has provided expert troubleshooting and training support to early grade reading data collection software for STS for the last four years. He has direct experience training data collectors in the field in key protocols for EGRA data collection and in the functionality of Tangerine, Ona, and SurveyCTO electronic data collection software. Mr. Schmenner holds a Master of Art in International Studies from the University of San Francisco. Mr. Schmenner is a native English speaker with basic proficiency in French. KRISTINA SOLUM (STS) Kristina Solum was the technical lead for the evaluation, bringing her technical expertise to the design and execution of the evaluation design workshop, operational data collection, and report writing. Ms. Solum has over 15 years’ experience designing, managing, implementing, and evaluating education projects, including pioneering support to assessing learning outcomes for children with disabilities. Ms. Solum is a former educator and teacher trainer and routinely designs and delivers capacity-building workshops for stakeholders in countries around the globe. STEPHANIE TEMPLETON (STS) Stephanie Templeton lead data collector training and supervised a team of data collector during the evaluation fieldwork. Ms. Templeton is a researcher with over nine years’ experience in supporting partner governments and development agencies in assessing the quality of their education systems and programs. Stephanie is experienced in communicating technical concepts to non-expert and expert audiences alike through written documentation including guidelines, process manuals, results reports, and technical reports, as well as through in-person and remote training. Stephanie has also developed innovative results communications using interactive dashboards and graphic memos. Stephanie holds a Master of Arts in International Education Development from Teachers College, Columbia University. She is fluent in English, with advanced proficiency in French, and intermediate proficiency in Spanish. AUTUMN THOMAS (STS) Autumn Thomas was responsible for data collection support. Ms. Thomas is an international development professional with experience providing project support to international education and youth, curriculum development and democracy programs. During her time at STS, Ms. Thomas has supported numerous large-scale data collection activities in various countries including Tajikistan, Kyrgyzstan, and Afghanistan. Thomas holds an M.S. in International and Comparative Education from Florida State University. USAID.GOV Whole-Of-Project Performance Evaluation of the Reading For Success Project – Morocco | 159 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Memorandum Regarding Organizational Conflict of Interest for School-to-School International (STS) NORC and School-to-School International (STS) do not foresee an Organizational Conflict of Interest (OIC) resulting from STS’s role as a subcontractor on Reading for Success – Small Scale Experimentation (RFS-SSE) and Reading for Success – Improving Deaf Children’s Reading Through Technology (RFS-IDCRT) as outlined by the terms in Contract Information Bulletin 99-17 (CIB). According to the CIB, high potential for OIC occurs in the following scenarios:111 1. Where a firm that designs a USAID activity under contract with USAID wants to be eligible for the competition to implement the activity; 2. Where a firm that evaluates an activity or contractor under contract with USAID wants to provide services that are requested as a result of the evaluation; and 3. Where a firm that audits USAID contractors under contract with USAID seeks to do consulting work under contract with USAID (sometimes in competition with the firms audited). None of the above-mentioned scenarios outlined in the CIB apply to STS’s subcontractor role on RFS-SSE or RFS-IDCRT. In both instances, STS acted strictly in an evaluation role having no role in project implementation. STS’s role on RFS-SSE included conducting assessments of student learning using the Early Grade Reading Assessment (EGRA) tool and measures changes in teacher and school administrator behaviors using the Snapshot of School Management Effectiveness (SSME) methodology. As part of this work, STS designed data collection tools, facilitated enumerator training, conducted student assessments and teacher surveys, and completed data analysis. On RFS￾IDCRT, STS provided technical assistance to grantee IDRT on program design and theory of change, monitoring, evaluation, and learning and fidelity of implementation approaches, and research design. STS also worked collaboratively with Institute for Disabilities Research and Training, Inc. (IDRT) and local stakeholders on the development, piloting, administration, results analysis and interpretation of one of the first ever Early Grade Reading Sign Language Assessments in Moroccan Sign Language, used to measure the MSA and MSL skills of G1 and G2 students who are deaf or hard of hearing. Though NORC and STS do not foresee any potential for impaired objectivity stemming from STS’s evaluation role on past projects, staffing will be responsive to this concern with NORC staff being included to ensure objectivity in data collection and analysis. 111 https://www.usaid.gov/sites/default/files/documents/1868/cib9917.pdf USAID.GOV USAID Impact and Performance Evaluation of the Waache Wasome Project (Tanzania) | 160 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523