EVALUATION DESIGN REPORT FOR THE WHOLE-OF-PROJECT PERFORMANCE EVALUATION OF THE READING FOR SUCCESS PROJECT – MOROCCO TASKING NO: 7617.020.01 Contract No. GS-10F-0033M / AID-OAA-M-13-00013 JULY 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NORC at the University of Chicago and School-to-School International. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | i EVALUATION DESIGN REPORT FOR THE WHOLE-OF-PROJECT PERFORMANCE EVALUATION OF THE READING FOR SUCCESS PROJECT – MOROCCO TASKING NO: 7617.020.01 JULY 2019 Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013 SUBMITTED TO: Ben Sylla, USAID E3/ED Kurt Krausse, USAID/Morocco Amina El Abdellaoui, USAID/Morocco Mariam Britel-Swift, USAID/Morocco PREPARED BY: Erika Keaveney Alexandre Monnard Kristina Solum Stephanie Templeton Matthew Murray Mark Lynd Casey McHugh Drew Schmenner SUBMITTED BY: NORC at the University of Chicago DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | ii TABLE OF CONTENTS ACRONYMS...........................................................................................................................................................IV EVALUATION PURPOSE .................................................................................................................................... 1 PROJECT BACKGROUND................................................................................................................................. 1 EVALUATION APPROACH AND DATA ...................................................................................................... 3 EVALUATION QUESTIONS ....................................................................................................................... 3 Project Outcomes.......................................................................................................................................................3 Implementation...........................................................................................................................................................4 Inclusion ........................................................................................................................................................................4 Implementation / Sustainability..............................................................................................................................4 Sustainability................................................................................................................................................................5 Lessons Learned .........................................................................................................................................................5 METHODOLOGY.......................................................................................................................................... 5 SAMPLING STRATEGY/SELECTION OF RESPONDENTS ..............................................................16 School Selection .......................................................................................................................................................17 School-Level Respondents.....................................................................................................................................17 MoE Officials and Other Stakeholders.............................................................................................................18 USAID/Morocco and RFS Implementing Partners.........................................................................................18 DATA SOURCES AND COLLECTION METHODS ..........................................................................19 Tool Design, Field Test, and Review Process...................................................................................................19 Key Informant Interviews (KIIs) ..........................................................................................................................20 Focus Group Discussions (FGDs)........................................................................................................................20 Inclusion Considerations During Data Collection ...........................................................................................22 Secondary Data Sources.......................................................................................................................................23 FIELD WORK................................................................................................................................................23 Data Collection Teams..........................................................................................................................................23 Data Recording and Notes Capture..................................................................................................................23 Field Work Preparation, Communication, and Coordination......................................................................24 Data Collection Schedule......................................................................................................................................24 ANALYSIS PLAN ..........................................................................................................................................25 Qualitative Analysis.................................................................................................................................................25 Quantitative Analysis..............................................................................................................................................25 REPORTING..................................................................................................................................................27 LIMITATIONS OF THE PROPOSED APPROACH..............................................................................27 Evaluation Design....................................................................................................................................................27 Linguistic and Cultural Complexity.....................................................................................................................28 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | iii Fidelity of Implementation ....................................................................................................................................28 Representativeness of Sample.............................................................................................................................28 Risk of Divergent Findings....................................................................................................................................28 Availability of Respondents...................................................................................................................................28 Accuracy of Qualitative Responses.....................................................................................................................29 WORK PLAN........................................................................................................................................................29 SCHEDULE.....................................................................................................................................................29 EVALUATION TEAM..................................................................................................................................29 DISSEMINATION PLAN ....................................................................................................................................30 ANNEXES ..............................................................................................................................................................32 ANNEX I: EVALUATION STATEMENT OF WORK .........................................................................32 ANNEX II: DATA COLLECTION INSTRUMENTS AND IRB MATERIALS................................. 50 ANNEX III: ESTIMATED LOE AND STAFFING PLAN......................................................................89 ANNEX IV: ROLES AND RESPONSIBILITIES MATRIX.....................................................................90 ANNEX V: TIMELINE WITH DELIVERABLES AND MILESTONES................................................92 ANNEX VI: DETAILED DRAFT FIELDWORK SCHEDULE .............................................................94 ANNEX VII: DISCLOSURE OF REAL OR POTENTIAL CONFLICT OF INTEREST FOR USAID EVALUATIONS.............................................................................................................................100 ANNEX VIII: EVALUATION DESIGN WORKSHOP AGENDA...................................................101 ANNEX IX: EVALUATION DESIGN WORKSHOP PARTICIPANTS .........................................102 LIST OF TABLES Table 1. RFS Performance Evaluation Matrix ............................................................................................ Table 2. AREF/Regions Included in the RFS Evaluation Sample ..........................................................16 8 Table 3. Respondents by targeted number, tool, subgroup and sampling/selection criteria .......20 Table 4. Summary field work schedule .....................................................................................................25 LIST OF FIGURES Figure 1. Reading for Success Project Results Framework .....................................................................2 Figure 2. Triangulation Using Parallel Combinations for Mixed-Methods Evaluation.......................6 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | iv ACRONYMS CDCS Country Development Cooperation Strategy CSO Civil Society Organization DO Development Objective EGRA Early Grade Reading Assessment ES Education and Social Services HICD Human and Institutional Capacity Development IR Intermediate Result IRB NORC’s Institutional Review Board MASSAR Moroccan national database for education statistics MELP Monitoring, Evaluation, and Learning Plan MoE Ministry of National Education, Vocational Training, Higher Education, and Scientific Research MSA Modern Standard Arabic MSL Moroccan Sign Language PIRLS Progress in International Reading Literacy Study RFS Reading for Success RFS-HICD Reading for Success-Human and Institutional Capacity Development RFS-IDCRT Reading for Success-Improving Deaf Children's Reading Through Technology RFS-NPR Reading for Success-National Program for Reading RFS-SSE Reading for Success-Small Scale Experimentation USAID United States Agency for International Development USG United States Government CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 1 EVALUATION PURPOSE The performance evaluation of the Reading for Success (RFS) project will assist the United States Agency for International Development/Morocco (USAID/Morocco) in gathering information for the following purposes: • To assess the extent to which the activities under the RFS project contribute to achieving the Education Development Objective; • To inform the implementation of ongoing or upcoming activities, including that of the RFS￾National Program for Reading (RFS-NPR); and • To inform the next Morocco Country Development Cooperation Strategy (CDCS) design, which is scheduled to be drafted in 2020. The evaluation results will be shared with broader USAID audiences, including USAID/Washington and other missions, to highlight best practices and lessons learned, and to inform upper management of the results achieved by the Morocco Mission in basic education and early grade reading. In addition, relevant results will be shared with Government of Morocco counterparts and other donors to further promote improving basic education in Morocco through enhanced reading instruction in early grades. This evaluation design report details the study’s six research questions and the approaches to sampling, data collection, and analysis that will be employed to answer the questions. This report has been informed by two evaluation tasks that have already been completed—a desk review of existing RFS-related documents and an evaluation design workshop and stakeholder mapping exercise held in Morocco in May 2019. This design workshop allowed the evaluation team to ground the evaluation in the local context and realities, refine the evaluation questions, identify the most appropriate data sources and methods for answering the questions, build consensus and buy-in on the evaluation approach, and finalize the evaluation work plan. This evaluation design report helps to ensure that evaluators and stakeholders have a shared understanding of the evaluation purpose, methods, and analysis and reporting procedures. PROJECT BACKGROUND USAID/Morocco conceived the RFS project to address Development Objective 3: enhanced educational development for children at the primary level. This objective was spelled out in the Mission’s most recent CDCS released in 2013. Citing persistent inequities in education and lagging literacy levels, USAID/Morocco shifted its focus toward primary education after having targeted other educational domains during the preceding 10 years, including middle schools, teacher training institutes, and job training for disadvantaged youth. As USAID/Morocco laid out in its CDCS, the foundational skill of literacy provides a springboard for students’ future success in the classroom, workforce, and broader civil society. The CDCS also emphasized the importance of synergies with the Moroccan government’s education agenda, including the comprehensive reform initiative Vision 2030. The education development objective would be met through attaining three Intermediate Results (IRs): improvement of reading skills, increased community support for reading, and improved policy environment to support reading, as detailed in the RFS results framework in Figure 1. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 2 Figure 1. Reading for Success Project Results Framework IR 1: Reading skills improved RFS is comprised of four activities—RFS-Small Scale Experimentation (RFS-SSE), RFS-NPR, RFS￾Human and Institutional Capacity Development (RFS-HICD), and RFS-Improving Deaf Children’s Reading Through Technology (RFS-IDCRT); all but RFS-NPR have already closed. As a whole, RFS aims to address lagging reading outcomes for Moroccan students. The country’s youth literacy rate of 82 percent is among the lowest in the Middle East and North Africa, with a notable gap between young men (89 percent) and young women (74 percent).1 While reading achievement has improved in recent years, it remains low. Moroccan students’ reading scores on the Progress in International Reading Literacy Study (PIRLS) were statistically significantly higher in 2016 than in 2011, but Morocco ranked 48th out of 50 countries in 2016 and scored well below international benchmarks.2 In addition to PIRLS results, prior research supported by USAID underscores the opportunity for improvement: in 2011, an Early Grade Reading Assessment (EGRA) administered in the Doukkala Abda region revealed that only one out of three grade 2 and 3 students could read a second grade level text well enough to comprehend it. Additionally, 33 percent of grade 2 students and 17 percent of grade 3 students could not read any words of the Modern Standard Arabic (MSA) text.3 Factors cited as contributing to poor performance on the EGRA include a lack of teacher preparation and reading materials as well as a mismatch between the language of instruction at schools and the predominant languages spoken at home.4 Launched in 2015, RFS-SSE aimed to improve early grade reading outcomes through the development and piloting of new Arabic reading lessons and materials for grades 1 and 2 and the implementation of summer enrichment reading programs. The activity’s lessons and materials ushered in a new phonics-based pedagogical approach to teaching reading in Morocco, representing a shift from the historically-dominant whole language method. Four regions were selected for 1 UNICEF, “UNICEF Global Databased—Education: Adult Literacy Rates—Percentage,” accessed April 2019, https://data.unicef.org/wp-content/uploads/2016/05/education_table-youth-and-adult-literacy-rate-updated-oct.-2015.xlsx. 2 Mullis, I.V.S., Martin, M.O., Foy, P., & Hooper, M. PIRLS 2016 International Results in Reading. (Chestnut Hill, Mass.: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement, 2017). 3 RTI International, Student Performance in Reading and Mathematics, Pedagogic Practice, and School Management in Doukkala Abda, Morocco, (2012). 4 Ibid. DO #3: Enhanced educational development for children at the primary level IR 3: Policy environment to support reading improved IR 2: Community support for reading increased IR 2: Community support for reading increased IR 2: Community support for reading increased Sub-IR 1.1: Reading materials used in primary schools improved Sub-IR 1.2: Reading instruction improved Sub-IR 1.3: Extra￾curricular reading programs expanded Sub-IR 2.1: Family to support reading increased engagement Sub-IR 2.2: CSO capacity to implement reading initiatives increased Sub-IR 3.1: Benchmarks for grades 1 to 3 established Sub-IR 3.2: Reading instruction time defined Sub-IR 3.3: National assessment protocols for reading established CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 3 piloting the new lessons and materials—Sous Massa, Rabat-Kenitra, Orient, and Fes-Meknes—based on geographic, cultural, and socioeconomic factors. RFS-NPR built on RFS-SSE’s work at a greater scale with the overall goal of enabling students in target regions to read fluently with comprehension by the end of grade 2. Started in 2017, this five￾year activity is revising and piloting RFS-SSE’s grade 1 and 2 curriculum and materials, developing new curriculum and materials for grades 3 and 4, and expanding reading enrichment activities with the ultimate goal of a national scale-up. In addition, the activity is focusing on enhancing reading instruction through better teacher training and improving systems for assessment. The activity is piloting its new materials in the same schools targeted by RFS-SSE. Scale-up began during the 2017- 18 school year with the national rollout of grade 1 materials, followed by grade 2 materials during the 2018-19 school year. The two other RFS activities—RFS-HICD and RFS-IDCRT—targeted the Ministry of Education and children who are deaf or hard of hearing, respectively. RFS-HICD aimed to support the Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE) as it began to implement its comprehensive reform plan—Vision 2030—to create a quality, equitable educational system. Through the administration of a Human Institutional and Capacity Development (HICD) assessment, RFS-HICD measured the MoE’s capacity to implement Vision 2030 and associated priority projects in nine technical areas and proposed performance solutions to address identified gaps. RFS-IDCRT aimed to improve the reading skills of students who are deaf or hard of hearing in grades 1 to 3 through the use of assistive technology. In Morocco, more than 85 percent of students who are deaf or hard of hearing cannot access education, and those who do attend school encounter a dearth of materials and properly trained educators. RFS-IDCRT’s four major components included the development of educational software for Moroccan Sign Language (MSL), training stakeholders on use of the software as well as other topics related to Deaf education, the creation of a steering committee to foster national dialogue on issues in Deaf education, and the development of an assessment to measure beneficiaries’ reading skills. EVALUATION APPROACH AND DATA EVALUATION QUESTIONS Six evaluation questions and their related sub-questions will guide the RFS performance evaluation. After an initial set of questions was drafted for the Project Appraisal Document (PAD) Evaluation Statement of Work, they were revised in collaboration with USAID/Morocco and MoE partners at an evaluation design workshop held in Rabat and concurrent interviews with RFS implementing partners. This independent, whole-of-project performance evaluation will answer questions that each address a separate theme—project outcomes, implementation, inclusion, sustainability, and lessons learned: PROJECT OUTCOMES 1. To what extent have the activities under the RFS project contributed to the overall development objective of the RFS project of enhancing educational attainment for Moroccan children at the primary level? 1.1. How and why did students under the RFS-SSE, RFS-IDCRT, and RFS-NPR activities realize the observed learning results? CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 4 1.2. To what extent did RFS activities contribute to progress towards each of the 5Ts of reading instruction?5 1.3. How have the activities performed against the higher-level indicators6 at the project level? IMPLEMENTATION 2. How and why did factors impede and/or facilitate achievement of the RFS project's development objective? 2.1. How and why did factors impede the achievement of the project's development objective? 2.2. How and why did factors facilitate achievement of the project's development objective? 2.3. To what extent have the activities responsively used (or are currently using) activity-level M&E data to make necessary and relevant adjustments to program design, implementation, and management? INCLUSION 3. Which primary-school-aged populations of children are not benefiting from the current RFS project activities, and how can they be better served in future projects/activities? (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) 3.1. Which primary-school-aged populations of children (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) in schools targeted by RFS are not benefiting from the current project activities? 3.2. What are the barriers to reaching those students who did not benefit from RFS and measures by which these students can be better served in future projects or activities? IMPLEMENTATION / SUSTAINABILITY 4. What was the MoE’s role in leading project activities, and how did this role influence the implementation, management, and results of the activities? 4.1. What was the ministry's role in leading RFS project activities? 4.2. How did the ministry's role influence the implementation, management, and results of the activities? 4.3. How did the role of the ministry vary in implementation, management, and results at the central, provincial, regional, and local levels? 5 USAID’s approach to improving reading and literacy revolves around five goals also knowns at the 5 T’s. According to USAID these 5 T’s are the key to reading success. The 5 T’s are: (1) more time devoted to teaching reading, (2) better techniques for teaching reading, (3) more texts in the hands of children, (4) teaching children in the mother tongue (a language they speak and understand), and (5) testing children’s reading progress. 6 These indicators are defined as: 3.b Percentage of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with United States Government (USG) assistance 3.1.a Number of learners reached in reading programs at the primary level 3.1.b Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance 3.2.a Increased provision of reading activities outside of instructional time (disaggregated by public / private) 3.3.a Number of laws, policies, regulations or guidelines developed or modified to improve primary reading programs CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 5 SUSTAINABILITY 5. What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? LESSONS LEARNED 6. What key preliminary lessons learned and recommendations can be gleaned from implementation of the program to date? METHODOLOGY This evaluation began with a desk review, which provided the context, background, and understanding needed for effective qualitative fieldwork along with support for the design of more in-depth and nuanced subsequent evaluation components. The evaluation team studied existing documents from USAID/Morocco and the implementers of the four RFS activities as well as external documents pertaining to early grade reading in Morocco and those from Morocco’s Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE). The review provided an overview of education and early grade reading in Morocco, including past activities by USAID and the MoE as well as the research that informed the design of RFS activities. The overall results framework and project indicators for RFS are discussed, followed by specific descriptions of the four activities, including their frameworks and indicators. The review also described cross-cutting issues from all four activities, including inclusion and disability, gender, sustainability, lessons learned, and challenges. The evaluation team then hosted a one-day evaluation design workshop in Rabat on May 7 to ground the evaluation in the local context and realities, refine the evaluation questions, and build consensus and buy-in on the evaluation approach (see Annex VIII for the workshop agenda). The design workshop included key representatives from the MoE and USAID/Morocco, who reviewed and proposed revisions to evaluation questions (see Annex IX for the list of participants) as well as provided additional sub-questions to inform the overall evaluation. Using the updated questions and sub-questions, the participants completed a stakeholder mapping exercise to identify the most appropriate data sources and methods for answering the each of the proposed evaluation questions. This map was used to drive the sample used for this evaluation. In order to answer the six primary RFS performance evaluation questions and related sub-questions, the evaluators will primarily use a qualitative research approach.7 The team will conduct field work over the course of three weeks in September and October 2019, collecting data from a variety of RFS stakeholder respondent groups in order to gain understanding of multiple perspectives/experiences across a variety of levels (national, regional, provincial, and school/community) and to enable triangulation of data and findings. Triangulation will also include using both qualitative and quantitative methods to gain a more holistic and deeper understanding of the RFS project as well as mitigate potential bias in the evaluation data and findings. Moreover, while the research will focus primarily on the regions and provinces where RFS activities have been implemented—namely, the RFS-SSE and RFS-NPR pilot regions, provinces, and schools—key MoE 7 Please note that the current assumption is that the proposed team will conduct all qualitative data collection and analysis and will not use local a data collection firm. For the purposes of budgeting, it is therefore assumed that all quantitative data will be drawn from secondary (i.e., existing) data and reports. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 6 stakeholders from non-pilot regions and provinces will also be sampled to gain additional perspectives on the broader scale-up. Qualitative data collection methods such as semi-structured key information interviews (KIIs) will capture a variety of views, experiences, and perspectives from key stakeholders at national, regional, provincial, and school levels. In addition, focus group discussions (FGDs) with beneficiaries at the school and community levels will be conducted through school/learning site visits across multiple regions and provinces. Where possible, existing RFS project data (including EGRA results and other monitoring and evaluation data) as well as administrative and/or performance data from Morocco’s National Education Management Information System (EMIS)—L’évaluation des Acquis des Élèves et du Système d’Information (MASSAR)—will be analyzed to supplement qualitative findings using the approach described in Figure 2. This methodology is informed by the RFS evaluation document review, evaluation design workshop, and stakeholder mapping exercise. Figure 2. Triangulation Using Parallel Combinations for Mixed-Methods Evaluation8 The qualitative and quantitative analyses proposed in this design report will employ a “parallel combinations” analytical approach, illustrated below, through which each method is applied separately from the other and integrated or triangulated following independent analysis: As appropriate, a “sequential combinations” approach will also be employed, enabling findings from qualitative field work to inform the design and implementation of secondary quantitative analysis. To support this purpose, qualitative data collection tools will be structured so that the research team can capture information from key stakeholders on relevant and available secondary data sources that could further shed light on the evaluation question(s) being examined. Synthesis or triangulation Qualitative data collection Qualitative data analysis Quantitative data analysis The evaluators will integrate a gender-sensitive evaluation approach, drawing on resources such as USAID’s How-To Notes on Engendering Evaluations on Engendering Evaluation and Gender Integration in Education Programming and USAID’s Gender-sensitive Evaluation: Best and Promising Practices for Engendering Evaluation. This approach will involve ensuring all data are disaggregated and reported by sex where appropriate, engaging female and male researchers, and striving for gender balance among respondents.9 Gender frameworks will be considered in the analyses as well, 8 Adapted from USAID Technical Note on Conducting Mixed Methods Evaluation (2013). USAID Bureau of Policy, Planning and Learning. Retrieved from https://www.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations_Technical_Note.pdf. 9 For example, focus group discussions with parents will aim for equal participation between men and women. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 7 including, for example, examinations of how the interventions may have affected boys and girls differently. The evaluators will also incorporate social inclusion in their evaluation approach, including consideration of geographical or regional variations, ethnicity, language, socio-economic status, disability status, and age in the design, sampling, and analytic approaches. As such, the evaluation team will include members with prior experience with Deaf education in Morocco and plans to engage sign language interpreters and minority language translators as needed. Geographic and regional variations in RFS target zones informs the evaluation’s stratified sampling approach across pilot regions and provinces. These areas also include variations in language groups, including both Arabic and Amazigh languages. The range of respondents, incorporates an array of socioeconomic statuses and includes high-level government officials in addition to teachers and parents from purposively selected rural disadvantaged areas. Individuals with disabilities will be purposively selected as respondents for the IDCRT activity in particular. Age bracket data will be collected from respondents to enable disaggregation of data and inform analysis. In addition, in order to support an inclusion lens within the analysis, qualitative tools include specific sections on inclusion and targeted questions and probes10 related to inclusion, such as identifying which kinds of children who may not have benefitted from the program, barriers to their inclusions as well as recommendations and lesson learned to better serve hard to reach children in the future. A wide-range of respondent types will provide these varied perspectives, including but not limited to parents of students with disabilities; representatives from disabled persons organization; teachers and school directors; province and AREF-level directors; AREF coordinators who focus on students with disabilities; as well central government officials and experts at the MoE and Ministry of Solidarity, Women, Family and Social Development. The overall RFS evaluation approach is outlined in the performance evaluation matrix summary in Table 1, followed by sections detailing the sampling strategy, data sources and collection methods, and analysis plan. In the matrix, a preliminary list of associated data sources, sampling/selection criteria, and data collection methods has been identified for each evaluation question and sub￾question. This matrix was informed by the document review and stakeholder mapping conducted in Morocco in May 2019. This evaluation matrix will form the basis of a more detailed data collection tool map to be developed prior to field work. 10 For example, follow-up questions may probe about more specifically about the experience of children with disabilities, children whose mother language is not Arabic, child in rural areas or other geographically marginalized areas, children who are orphans, children in poor householder, children who are migrants (e.g. nomads, refugees), children engaged in child labor, or children living in areas experiencing insecurity. CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 8 Table 1. RFS Performance Evaluation Matrix # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS PROJECT OUTCOMES 1 To what extent have the activities under the RFS project contributed to the overall development objective of the RFS project of enhancing educational attainment for Moroccan children at the primary level? USAID/Morocco XXXXXXXXXXXX KII Qualitative data and debriefing notes will be captured mainly through typed up FGD and KII field notes (in French or English). Electronic field notes will be imported into NVivo for first-order and second-order qualitative data analysis. This analysis will then be examined in relation to the research questions, the literature review, and existing quantitative data to develop key findings. MoE central office officials XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX XXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII School Directors XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXX KII MoE AREF/Regional Officials XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXX KII MoE Provincial Officials XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXX KII CMREF XXXXXXXXXXXXXXXXXXXXXX KII Teachers XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX FGD Parents XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXX FGD NPR Baseline EGRA XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXX Secondary data Treatment and control contrast using multivariate regression analysis to assess SSE’s medium-term impact on pupil reading performance following 2 full years of exposure to the SSE program MASSAR data on repetition/dropouts as well as pupil and teacher characteristics XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX Secondary data Pre/post-test using multivariate regression analysis—controlling for time invariant teacher and pupil characteristics—to assess the extent to which the NPR grade 1 scale-up affected pupil repetitions or dropouts CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 9 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS 1.1 How and why did students under the RFS-SSE, RFS- IDCRT, and RFS-NPR activity11 realize the observed learning results? USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 Existing project data XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX Document review Content analysis to summarize implementing partner results related to research questions 1.2 To what extent did RFS activities contribute to progress towards each of the 5Ts of reading instruction? USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX FGDs See Qualitative Data Analysis Methods from Q1 Existing project data XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX Document review Content analysis to summarize implementing partner results related to research question 1.3 How have the activities performed against the higher level indicators at the project level? USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 USAID/Morocco XXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Existing project data XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX Document review Content analysis to summarize implementing partner results related to research question 11 Areas of focus within this evaluation question include the following specific areas for RFS-SSE and RFS-NPR: 1- teaching methods and instruction (including training and materials) 2 - student reading materials like textbooks 3 - enrichment programs such as school and community-based extracurricular reading programs for students and family engagement in support of reading CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 10 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS IMPLEMENTATION 2 How and why did factors impede and/or facilitate achievement of the RFS project's development objective? USAID/Morocco XXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partner subcontractors XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE central office officials XXXXXXXXXXXXXXXXXXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX XXX XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE Provincial Officials XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE AREF/Regional officials XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 School Directors XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX X XXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 Publishers XXXXXXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 11 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS NPR Baseline EGRA and SSE Midline 1 EGRA XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXX Secondary data T-test measuring statistical improvements in reading performance between a cross-section of Grade 1 SSE pupils in May 2016 and May 2018. While SSE was in effect for grade 1 in May 2016, it had only been operating for 4 months therefore a comparison after 2 years could offer insights on implementation efficacy over the longer term. 2.1 How and why did factors impede the achievement of the project's development objective? see Q2 (same data sources) XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 2.2 How and why did factors facilitate achievement of the project's development objective? see Q2 (same data sources) XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 2.3 To what extent have the activities responsively used (or are currently using) activity-level M&E data to make necessary and relevant adjustments to program design, implementation and management? USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partner subcontractors XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 12 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS INCLUSION 3 Which primary school aged populations of children are not benefiting from the current RFS project activities and how can they be better served in future projects/activities? (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) MoE central office officials XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Other Gov Officials XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 Parents of students with special needs XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 CMREF XXXXXXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partners XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partner subcontractors XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE AREF/Regional officials XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Disabled persons organizations XXXXXXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Education Committee Members of Parliament or Staff XXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 13 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS 3.1 Which primary school aged populations of children (considering gender, students who are disadvantaged/marginal, less successful, or special needs, those who do not speak Arabic at home, students in rural or vulnerable areas) currently in the educational system and being reached by RFS are not benefiting from the current project activities? see Q3 (same data sources) XXXXXXXXXXXXXXXXXXXXX see Q3 (same collection methods) See Qualitative Data Analysis Methods from Q1 3.2 What are the barriers to reaching those students who did not benefit from RFS, and measures by which these students can be better served in future projects or activities? see Q3 (same data sources) XXXXXXXXXXXXXXXXXXXXX see Q3 (same collection methods) See Qualitative Data Analysis Methods from Q1 IMPLEMENTATION/SUSTAINABILITY 4 What was the ministry's role in leading project activities, and how did this role influence the implementation, management, and results of the activities? MoE central office officials XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX KII See Qualitative Data Analysis Methods from Q1 USAID implementing partners XXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 USAID/Morocco XXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 4.1 What was the ministry's role in leading RFS project activities? see Q4 (same data sources) XXXXXXXXXXXXXXXXXXXXX XXXXX see Q4 (same collection methods) See Qualitative Data Analysis Methods from Q1 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 14 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS 4.2 How did the ministry's role influence the implementation, management, and results of the activities? see Q4 (same data sources) XXXXXXXXXXXXXXXXXXXXX XXXXX see Q4 (same collection methods) See Qualitative Data Analysis Methods from Q1 4.3 How did the role of the ministry vary in implementation, management, and results at the central, provincial, regional, and local levels? MoE AREF/Regional officials XXXXXXXXXXXXXXXXXXX XXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE Provincial Officials XXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 School Directors XXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXX XXXXXXX KII See Qualitative Data Analysis Methods from Q1 MoE central office officials XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXX XXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXX XXX XXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 CMREF XXXXXXXXXXXXXXXXXXXXX KII See Qualitative Data Analysis Methods from Q1 Teachers XXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXX FGD See Qualitative Data Analysis Methods from Q1 CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 15 # EVALUATION QUESTIONS DATA SOURCE SAMPLING AND SELECTION CRITERIA COLLECTION METHODS DATA ANALYSIS METHODS SUSTAINABILITY 5 What are the key strengths and challenges that have affected the RFS project activities in creating a sustainable national reading system? see Q2 (same data sources) XXXXXXXXXXXXXXXXXXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 NPR Baseline EGRA XXXXXXXXXXXXXXXXXX XXXXXXXXXXXXX Secondary data Statistical contrast of performance at SSE “high touch” schools with performance at NPR scale-up “low touch” schools at single cross section. This analysis may provide evidence for whether observed results can be sustained at scale. LESSONS LEARNED 6 What key preliminary lessons learned and recommendations can be gleaned from implementation of the program to date? see Q2 (same data sources) XXXXXXXXXXXX XXXXXXXXXXXXX see Q2 (same collection methods) See Qualitative Data Analysis Methods from Q1 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 16 SAMPLING STRATEGY/SELECTION OF RESPONDENTS Primary respondents sampled for this evaluation will include school-level respondents; MoE and non￾MoE government officials at the provincial, regional, and national levels; personnel of USAID/Morocco and RFS implementing partners; and other RFS partner organizations. Evaluators sampled respondents from the geographic areas where RFS has targeted its interventions. These areas include eight provinces in four regions that have participated in both RFS-SSE and RFS-NPR, as these are the respondents who will be best able to speak to RFS. In addition, selected respondents are included from four provinces where RFS-NPR has supported the MoE’s scale-up of activities. Details about each of these groups are presented in the following section. The following table highlights the eight regions selected for inclusion in the overall RFS sample. Regional selection first prioritized areas that received the most intensive RFS intervention(s) – namely the four regions that include SSE and NPR pilot schools. Second, an additional four regions were selected to help capture the experiences within NPR scale-up areas, with the aim of capturing both urban and rural areas outside the four pilot regions that would also be logistical to access and conduct the data collection within the designated field work timeline. Based on these criteria, Béni Mellal-Khénifra, Drâa-Tafilalet, Marrakesh-Safi, and Tanger-Tetouan-Al Hoceima were selected for inclusion in the sample. The selection of NPR scale-up regions will be finalized in consultation with USAID.12 Table 2. AREF/Regions Included in the RFS Evaluation Sample AREF / REGION REGIONS THAT INCLUDE RFS SCHOOLS BY ACTIVITY INCLUDED IN IN RFS EVALUATION SAMPLE? INCLUDES SSE/NPR PILOT SCHOOLS INCLUDES NPR SCALE￾UP SCHOOLS INCLUDES IDCRT SCHOOLS Béni Mellal-Khénifra - ✓ - ✓ Casablanca-Settat - ✓ ✓ - Dakhla-Oued Ed-Dahab - ✓ - - Drâa-Tafilalet - ✓ ✓ ✓ Fes-Meknes ✓ ✓ ✓ ✓ Guelmim-Oued Noun - ✓ - - Laâyoune-Sakia El Hamra - ✓ - - Marrakesh-Safi - ✓ ✓ ✓ Orient ✓ ✓ ✓ ✓ 12 While three alternate scale-up regions have also been identified, Dakhla-Oued Ed-Dahab, Guelmim-Oued Noun and Laâyoune-Sakia El Hamra regions were not selected for inclusion in the RFS evaluation sample due to lower level of RFS direct interventions outside of NPR scale-up as well as anticipated logistical constraints for evaluation team’s travel to these regions related to their distance from the capital. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 17 AREF / REGION REGIONS THAT INCLUDE RFS SCHOOLS BY ACTIVITY INCLUDED IN IN RFS EVALUATION SAMPLE? INCLUDES SSE/NPR PILOT SCHOOLS INCLUDES NPR SCALE￾UP SCHOOLS INCLUDES IDCRT SCHOOLS Rabat-Kenitra ✓ ✓ ✓ ✓ Sous Massa ✓ ✓ ✓ ✓ Tanger-Tetouan-Al Hoceima - ✓ ✓ ✓ Total AREF/Regions 4 12 8 8 SCHOOL SELECTION For the performance evaluation, evaluators aim to visit 12 schools—one school in each of the eight provinces where RFS-SSE and RFS-NPR conducted pilots, two schools in different regions that participated in RFS-IDCRT, and two schools in RFS-NPR scale-up regions. 12 schools serves as a logistically feasible number of schools to travel to and collect data within the planned data collection schedule and provides a considerable amount of coverage of the RFS pilot (eight out of 90) and IDCRT (two out of ten) populations, as well as two schools from the RFS-NPR scale-up. Evaluators will use existing secondary data from NPR to purposively select the eight RFS pilot and two RFS￾NPR scale-up schools to ensure varied representation of a range of school characteristics such as urbanicity,13 predominant mother language of students,14 school performance,15 school population size, and socio-economic status, as measured by available proxies in existing NPR datasets. Of the eight RFS-IDCRT schools which participated in the project evaluation, two have been selected based on the range of grades offered, teacher backgrounds, as well as student performance on EGRA at endline.16 The qualitative data gathered from school-based sources aims not for statistical representativeness or empirical generalizability, as a quantitative study would. Rather, the sample aims for “theoretical generalizability,” in which the data can be used to explain phenomena detected in the study.17 The number of FGDs and schools is expected to yield saturation, or sufficient data to provide no new information. SCHOOL-LEVEL RESPONDENTS At each of the 12 schools, a KII will take place with the school director, and two FGDs will be conducted—one with a group of four to ten teachers (grades 2-4), and one with a group of six to ten parents with students enrolled in grades 2 to 4.18 As part of the pre-site visit mobilization, 13 For example, the demographic criteria may distinguish between urban, peri-urban, rural, and deep rural schools. 14 Such as Darija, the Moroccan Arabic dialect, or Amazigh languages, such as Tarifit, Tashelhit and Central Atlas Tamazight. 15 Ideally, “high performing” and “low performing” schools may be determined by existing EMIS, EGRA data, or other RFS￾SSE or RFS-NPR project data and/or experience. 16 The two RFS-IDCRT schools that will be visited are Al Fath Association in Meknes Prefecture in the region of Fes￾Meknes and Fondation Lalla Asma in Rabat Prefecture in the region of Rabat-Salé-Kenitra. 17 Building Evidence in Education (BE2) (2019). Guidance Note on Qualitative Research in Education: Considerations for Best Practice. Draft version, p. 42. 18 Given the limited amount of time since scale-up began, FGDs with parents are not planned in RFS-NPR scale-up schools. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 18 school directors will be asked to identify and invite six to ten parents of students in grades 2-4 who have been identified as lagging behind or as struggling readers to participate in the focus group discussion.19 Only one parent per family will be recruited. If possible, an equal number of male and female parents and teachers will be invited. If possible, student data will be gathered from existing project-level data, including EGRA data, survey results, and evaluation reports. MOE OFFICIALS AND OTHER STAKEHOLDERS Evaluators will conduct KIIs with a host of MoE and government officials at the provincial, regional, and national levels. The provincial KIIs will be conducted in 12 provinces in eight regions, including all eight provinces where RFS-SSE and RFS-NPR experimental schools are located and four provinces where RFS-NPR has scaled up its activities. The proposed scale-up areas including a rural province in Béni Mellal-Khénifra, a rural province in Drâa-Tafilalet, urban Tangier Prefecture in Tanger-Tetouan￾Al Hoceima and urban Marrakesh Prefecture in in Marrakesh Safi region. In each of the 12 provinces, two MoE officials will be interviewed—XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. At the regional level, interviewees will include the XXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX. National-level interviews will be conducted with six officials from the MoE and one official each from the Ministry of Health and the Ministry of Solidarity, Women, Family and Social Development. Within the MoE, interviewees will include the directors of six divisions—the Directorate of Strategy, Statistics, and Planning; the Curriculum Directorate; the Directorate of Evaluation; the National Center for Pedagogical Innovations and Experimentation (CNIPE); the National Center for Evaluation and Examinations (CNEE), and the Directorate of Cooperation and Teaching in Private Schools. In addition, evaluators will seek to interview two representatives from disabled persons organizations, including the Deaf Association Steering Committee, and one education committee member of parliament or an education specialist, as recommended in the design workshop. USAID/MOROCCO AND RFS IMPLEMENTING PARTNERS Evaluators will conduct KIIs with representatives from USAID/Morocco and all four RFS activities. The USAID/Morocco officials to be interviewed include three members of the Education Office XXXXXXXXXXXXXXXXXXXXXXXXX. Evaluators will conduct at least three interviews with implementing partners from each RFS activity. At a minimum, evaluators will conduct one KII with XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX one with technical staff including monitoring and evaluation specialists, and for projects with subcontractors or similar partners, at least one with a subcontractor identified by evaluators together with the Mission and implementing partners. Specific subcontractors include subject matter experts engaged by Management Systems International, Creative Associates International, and Al Akhawayn University 19 Ideally, students who are ‘lagging behind’ or struggling readers may be identified through NPR’s forthcoming Early Warning System set to be integrated in its enrichment programs. However, if that is not feasible, struggling students may be identified by their participation in remedial programs or through their classroom teachers. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 19 under RFS-HICD; two of the six publishers of RFS-NPR materials; technical staff working on RFS￾NPR from University of Oregon, Management Systems International, and Al Akhawayn University; and two of the ten civil society organizations (CSOs) that have implemented summer enrichment reading activities under RFS-SSE. This wide net of respondents will enable evaluators to collect data from multiple perspective without greatly increasing the scope of data collection. Interviewing more than two partner organizations is not expected to yield additional useful data given the focused scope of each partner organization relative to other respondents. DATA SOURCES AND COLLECTION METHODS In order to respond to each evaluation question, several tools will be developed. This section details the process for tool development, review, and approval by USAID. It also provides additional information on the two primary collection methods—KIIs and FGDs—as well as sources for secondary quantitative analysis. TOOL DESIGN, FIELD TEST, AND REVIEW PROCESS The evaluators will develop seven new qualitative tools20—five KII guides and two FGD guides—to use during field work. During the design phase, evaluators will clearly map in Excel each item on each tool to relevant evaluation questions and sub-questions to ensure that sufficient information is elicited from all appropriate respondents to answer the evaluation questions. These detailed tool maps will be used to produce versions of each tool in Word that are structured for appropriate interview and discussion flow with respondents. Each tool will be aligned with one specific type of respondent, including: • USAID/Morocco officials; • RFS implementers and subcontractors; • MoE officials; • Non-MoE government officials; and • School-level respondents, including school directors, teachers, and parents While the same tool will be utilized for subgroups within a respondent type, additional sub-sections or targeted items may be incorporated for specific subgroups within the main tool, where relevant. For example, MoE officials at different levels will have certain sections of the core MoE tool removed based on their level of involvement in the project. Implementing partner staff will be asked different sets of questions depending on the activity(ies) in which they participated. Based on the finalized and approved evaluation design and questions, the data collection tools will first be developed in English and submitted for initial USAID review and approval in June. Upon USAID approval of the English tools, the tools will be translated into Arabic and subsequently submitted to USAID for review and approval in July. At this time, tools will also be submitted for review as part of the IRB approval process. In August, two Research Coordinators will be hired in Morocco to administer a limited pre-test of the tools in and around Rabat. The pre-test will provide an opportunity to field-test the appropriateness of question items for relevant respondents, identify potential practical issues with the tools’ administration and field procedures, as well as surface any flaws or limitations in the tool design. Based on the learnings from the pre-test, evaluation team will 20 No electronic data capture software or methodologies are anticipated given the qualitative focus of the tools. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 20 modify the tools as needed and submit them to USAID/Morocco for final review and approval prior to operational data collection in September. The tools will also include informed consent scripts/forms and clearly defined protocols for respondents’ participation in the evaluation, including an outline of risks/benefits, the right to decline participation, and confidentiality. NORC’s Institutional Review Board (IRB) will review the evaluation design and tools to ensure proper procedures and protocols are in place.21 MoE approval will be obtained prior to any school-based data collection. Evaluators will submit a list of proposed schools and respondents to USAID/Morocco. Through the Mission’s existing relationship with MoE, approval to collect data will be requested in the weeks prior to data collection. KEY INFORMANT INTERVIEWS (KIIS) The evaluators will conduct 81 KIIs, including 16 KIIs with USAID/Morocco and its implementing partners, 53 KIIs with Moroccan government officials and other stakeholders, and 12 KIIs with school directors. Additional details on respondent types and subgroups anticipated for participation within each of the broader categories—as well as associated targets—are detailed in Table 3. FOCUS GROUP DISCUSSIONS (FGDS) A total of 22 FGDs are targeted for completion across 12 school visits. As noted in Table 3, the evaluation team will complete 12 FGDs with primary school teachers (one FGD per school visit) and ten FGDs with parents of primary school students (one FGD per school visit). The majority of the FGDs (16 out of 22) will be conducted in pilot schools, four will be conducted in schools that participated in the IDCRT project, and two will be in scale-up schools. FGDs will aim for gender￾balanced participation where possible; for example, half of the parents of students in early primary grades invited to participate in FGDs will be fathers/male guardians and half will be mothers/female guardians.22 Table 3. Respondents by targeted number, tool, subgroup and sampling/selection criteria # RESPONDENTS (N) TOOL SUBGROUPS SAMPLING/SELECTION CRITERIA 1 USAID Morocco Officials (3) USAID KII Guide N/A Three KIIs with USAID/Morocco Education officials 2 USAID Implementing partners and subcontractors (13) USAID Implementing Partners KII Guide RFS activity COPs/COPs Four KIIs with XXXXXXXXXXX RFS technical staff Four KIIs or group interview—one per project with up to three technical staff participants or subject matter experts in each interview RFS subcontractors and partners Five KIIs with subcontractors and partners— XXXXXXXXXXXXXX XXXXXXXXXXXXX XXXXXXX 21 The IRB review will either approve the study protocol or determine that the activity is classified as exempt. 22 Only one parent or guardian is expected per family. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 21 # RESPONDENTS (N) TOOL SUBGROUPS SAMPLING/SELECTION CRITERIA 3 MoE government officials (48) MoE Official KII Guide MoE Central Office Directors Six KIIs with Central Office Directors identified during stakeholder mapping, including officials from XXXXXXX XXXXXXXXXXX AREF/Regional Directors Four KIIs with XXXXXXX XXXXXXXXX XXXXXX from the four pilot regions Provincial Directors 12 KIIs with XXXXXXXXXXX XXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXX XXXXXXX Inspectors/ Provincial Coordinators 12 KIIs with XXXXXXXXXXX XXXXXXX XXXXXXXXXXXXXX XXXX XXXXXXXXXX CMREF Directors Two KIIs with XXXXXXXXXXXXXXX Province-level Arabic teacher trainers from CMREF 12 KIIs with provincial-level XXXXXXXXX XXXXXXXXXXXX XXXXXXXXXX XXXXXX 4 Non-MoE stakeholders (5) Non-MoE Stakeholder KII Guide Official from Ministry of Solidarity, Women, Family and Social Development; Division of Social Affairs One KII with XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX Official from Ministry of Health One KII with official XXXXXXXXXX XXXXXXXXXXXXXXXXXXX Representatives from disabled persons organizations Two KIIs, including XXXXXXXXXXXXX Education Committee Members in Parliament or Staff One KII with an XXXXXXXXXXXXX XXXXX 5 School directors (12) School Director KII Guide School Directors in RFS Pilot Schools Eight KIIs with XXXXXXXXXXXX XXXXXXX School Directors in RFS-IDCRT Schools Two KIIs with XXXXXXXXXXXXXXX School Directors in RFS Scale-Up Schools Two KIIs with XXXXXXXXXXXXXXX DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 22 # RESPONDENTS (N) TOOL SUBGROUPS SAMPLING/SELECTION CRITERIA 6 Teachers (min. 48; max. 96)23 Teacher FGD Guide Teachers in RFS Pilot Schools Eight FGDs24 with primary school teachers (grade 1-4) in RFS pilot schools Teachers in RFS￾IDCRT schools Two FGDs25 with primary school teachers in RFS-IDCRT schools in non-pilot provinces Teachers in RFS Scale￾Up Schools 2 FGDs with primary school teachers in RFS-NPR scale-up schools 7 Parents (min. 60; max. 100) 26 Parent FGD Guide Parents in RFS-Pilot Schools 8 FGDs with parents of primary school students (grade 2-4) in RFS pilot schools; including a cross-section of parents of high￾performing and low-performing students27 Parents in RFS-IDCRT schools 2 FGDs with parents of primary school students in RFS-IDCRT schools in non-pilot provinces INCLUSION CONSIDERATIONS DURING DATA COLLECTION When selecting or recruiting respondents, the evaluators will aim to be as gender-sensitive and inclusive as possible by selecting equal numbers of men and women and by seeking inclusion of deaf and hard of hearing respondents as appropriate. In addition, all KII and FGD notes will include respondents’ sex; this information will be monitored throughout data collection to ensure both women and men’s perspectives and experiences are being captured in this evaluation. To facilitate broader inclusion, the evaluation team will be sensitive to respondents’ language preferences. For example, if a respondent prefers Amazigh language, a translator may be used. In addition, a Moroccan sign language interpreter will be engaged for FGDs with respondents who are 23 This table assumes four to eight teachers per FGD. While each FGD with teachers will aim to include eight participants, the total number of teachers who participate will depend on the number of teachers available within each sampled primary school and targeted grade levels. For example, if a school only includes one teacher (or section) for each targeted grade level (i.e. grades 1, 2, 3, and 4), the maximum number of teachers available to participate in the FGD would be four. In cases where more than eight grade 1 – 4 teachers are available, the criteria for selecting teacher participants will prioritize including two teachers from each targeted grade level as well as gender balance or participants (i.e. the inclusion of equal number of male and female teachers if available). If needed, additional selection criteria may be added, such as prioritizing teachers who have been at the school for at least four years as they would have had greater exposure to the RFS activity. 24 The number of teacher FGDs to be completed in RFS pilot schools is in line with best practices identified by Building Evidence in Education (BE2) around systemizing saturation points in qualitative research; namely that for each relevant category of subjects within a study, the sample size for the number of focus groups to be completed is a minimum of three to six focus groups. Building Evidence in Education (BE2) (2019). Guidance Note on Qualitative Research in Education: Considerations for Best Practice. Draft version. 25 Two school visits are expected to be sufficient for data saturation in IDRCT and RFS-NPR scale-up schools. IDRCT data will be further supplemented by qualitative results from the IDRT Evaluation Report produced by STS in July 2018 under the All Children Reading Grand Challenge. In addition, research suggests that two to three focus groups per category of respondent can yield “more than 80% of all themes.” See Guest, Greg, Emily Namey, and Kevin McKenna. “How Many Focus Groups Are Enough? Building an Evidence Base for Nonprobability Sample Sizes.” Field Methods 29, no. 1 (February 2017): 3–22. doi:10.1177/1525822X16639015. 26 This table assumes six to ten parents per FGD. While each FGD with parents will aim to include ten participants total, the evaluators recognize it may be difficult to identify, recruitment and mobilize ten parents at each site at the designated time and date. In recognition of this constraint, and standard practice on ideal FGD group size, a minimum of six parents per FGD has been established. 27 Student performance will be utilized as part of the parent selection criteria to capture a broader range of experiences. Evaluators do not anticipate being able to disaggregate data or analysis by this specific parent characteristic. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 23 deaf or hard of hearing, e.g., parents at the two IDCRT schools. As part of the mobilization process, the list of potential KII respondents will also aim to capture the respondent’s preferred interview language to ensure the deployed evaluation team’s language skills match the preferred language of the target respondent.28 SECONDARY DATA SOURCES For secondary quantitative data analysis, the evaluation team will work with USAID/Morocco, implementing partners, and MoE to secure the following datasets: • School-, teacher-, and pupil-level data for the NPR baseline study conducted in May 2018 • School-, teacher-, and pupil-level data for four of the five waves of data collection conducted as part of the SSE impact evaluation (May 2016, September 2016, May 2017, and September 2017) • Student- and school-level performance data from MASSAR for the 2016/17 and 2018/19 school years, including data on retention, repetition, dropouts, and academic performance (where available) Requested data will be de-identified at the individual level but identified at the school-level to facilitate meaningful analysis. In addition, the evaluation team will use qualitative field work as an opportunity to collect reliable information on the availability and relevance of further secondary data sources—including internal M&E data from implementing partners—that could support supplementary analysis and further triangulate the evaluation questions. FIELD WORK The evaluation team will collect qualitative data in Morocco in September and October 2019. DATA COLLECTION TEAMS Three teams will collect data over approximately three weeks in Morocco. Each team will consist of up to three data collectors. A qualitative evaluation expert from STS or NORC will lead each team and provide quality assurance. One or two Arabic-speaking interviewers recruited locally within Morocco will join each team. The composition of the teams will ensure collection of data in Arabic, French, English, or Amazigh languages. Each data collection team will have at least one female and one male researcher. DATA RECORDING AND NOTES CAPTURE In advance of each FGD or KII, the team lead will identify a data collector/interviewer and a notetaker. The interviewer will lead the interview or discussion while digitally recording it. The notetaker will keep detailed notes during the FGD or KII.29 Permission to audio-record will be requested as part of the consent process prior to the start of the KIIs. For cases where a KII or FGD participant declines permission to be audio-recorded, the team will respect their wishes and 28 As noted previously, all teams will include two fluent Arabic speakers. In addition, it is also anticipated two members of each team will also be fluent in French. 29 Handwritten notes can be in any language, but daily typed notes will be in either English or French. For example, if an interview is conducted in Arabic, the interview will be recorded to supplement the interviewer’s field notes (to be compiled at the end of the interview). The evaluation team anticipates the majority of interviews will be conducted in Arabic, enabling one team member to conduct the interview while the other takes notes. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 24 proceed with the interview without the audio-recording while taking note of the refusal to be recorded. Field notes will be typed up as soon as possible to capture details from the interview while it is still fresh. Each evening, immediately after the conclusion of fieldwork, the notetaker will listen to the day’s recording and type complete notes in either English or French. The team lead will review these electronic notes daily for quality and completeness and submit them to the evaluation home office. One remote evaluator will review these notes daily for monitoring purposes and track progress against the field work schedule. The evaluation team will conduct research in accordance with international norms on the ethical conduct of human subjects research, including obtaining consent30 from respondents. Qualitative data (including audio recordings) will be stored on a secured, password-protected server. Once field notes are saved electronically, the audio files will also be uploaded to a secure, password-protected server and deleted from the audio-recording device. FIELD WORK PREPARATION, COMMUNICATION, AND COORDINATION In the weeks prior to field work, the data collection teams will prepare remotely, including a one-day remote session conducted via video teleconference to orient all data collectors to their assigned regions and responsibilities. Teams will also participate in a two-day training in Morocco before data collection begins. Each data collector will receive a handbook describing the procedures for facilitating FGDs, conducting KIIs, and capturing full, detailed, and accurate notes. During field work, the three data collection teams will communicate regularly with one another to ensure consistency of procedures. During the first two weeks of data collection, all three teams will speak on the phone nightly to debrief together. During the third week, individual teams will continue to debrief nightly, and two inter-group phone calls will take place. DATA COLLECTION SCHEDULE Before beginning data collection, the evaluation team will meet in person with USAID for an in￾briefing, tentatively scheduled for September 11. The team will then participate in a two-day data collection training led by STS to ensure all data collectors fully understand the study design, tools, and procedures. Data collection will begin on September 16 after schools open. Initially, all three teams will remain in the Rabat-Salé-Kenitra region to establish common protocols and ensure coordination. Team 1 will be primarily French-speaking and will mostly remain in the Rabat area to conduct high-level KIIs with limited travel to Casablanca and Fes-Meknes. Team 2 will primarily collect data in Arabic, beginning in Fes-Meknes before traveling south to Souss Massa and Marrakesh-Safi. Team 3 will also primarily collect data in Arabic, beginning in Fes-Meknes before traveling east to collect data in Oriental in the second week and returning west through Béni Mellal-Khénifra and Drâa-Tafilalet in the third week. Please see Annex VI for a detailed draft fieldwork schedule. 30 The consent procedures will clearly explain the purpose, use, treatment, and destruction plan for any audio recorded data so that respondents can make an informed decision. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 25 Table 4. Summary field work schedule TEAM WEEK 1 (FIRST HALF) WEEK 1 (SECOND HALF) WEEK 2 WEEK 3 1 Rabat-Salé-Kenitra Rabat-Salé-Kenitra Rabat-Salé-Kenitra and Casablanca Fes-Meknes and Rabat￾Salé-Kenitra 2 Rabat-Salé-Kenitra Fes-Meknes Souss Massa Marrakesh-Safi 3 Rabat-Salé-Kenitra Fes-Meknes Oriental Béni Mellal-Khénifra and Drâa-Tafilalet Following the completion of data collection, all teams will return to Rabat for an in-person debriefing and identification of initial themes from data collection. The trip will conclude with an out-briefing with USAID, tentatively scheduled for October 4. Please see Annex VI for a draft of the field work schedule. ANALYSIS PLAN QUALITATIVE ANALYSIS Preliminary data capture and coding will begin during field work with the team debriefing sessions that take place each night within each of the three data collection teams. In addition, all three teams will join a group call twice per week during data collection. These debriefing sessions and associated notes will provide a way to document teams’ preliminary findings in real time during data collection. These notes will inform the out-briefing session with USAID and supplement the finalized field notes. Finalized debriefing and field notes will be imported into NVivo 12, a qualitative data analysis software package. The qualitative data analysis methodology will incorporate an iterative approach and include thematic content analysis of narrative data to identify and validate emerging themes. A codebook will be developed at two stages—when the evaluation design is finalized, and during data analysis when additional themes emerge. Each night, the team lead will review KII and FGD notes in English or French taken by data collectors to ensure that notes are complete and sufficiently detailed. Notes will be uploaded nightly for further review by remote staff to track against the data collection schedule. In addition to this immediate quality assurance of notes during field work, a limited number of field notes will be double coded to assess inter-coder reliability. Coders will meet to discuss any discrepancies identified and agree on definitive codes to apply. After the coding process generates meaningful patterns from the field notes, analysts will use these codes to perform second-order analyses. These analyses will examine coded data in NVivo 12 to identify salient themes across a range of respondents that answer each evaluation question. Responses from different individuals and respondent groups will be compared to identify discrepancies, if any exist, and to offer explanations for such discrepancies. The relationships between responses will also be examined to learn about linkages across themes. Hypotheses that explain findings will be tested by examining codes for disconfirming patterns or alternative explanation. Multiple analysts working in a single software package will ensure that data, codes, and analysis findings are stored in a common, shared location for reference during reporting. QUANTITATIVE ANALYSIS Quantitative data analysis will be conducted using the Stata/SE 15 statistical software package. Data will be imported and reviewed to ensure completeness and consistency in variable naming and DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 26 coding, particularly when multiple datasets are merged. All relevant variables will be examined for missingness and outliers prior to analysis, and appropriate strategies will be employed to address any observed irregularities (e.g., robustness checks and multiple imputation). Multiple regression analysis will be the primary analytical approach, allowing for statistical cross￾group comparisons (e.g., pre/post, treatment/comparison) while controlling for confounders and/or increasing statistical precision of the estimates. Where appropriate, basic descriptive and summary statistics and cross-tabulations will also be presented. The current proposed secondary quantitative analysis includes: • SSE treatment and control contrast of grade 2/3 pupils using NPR baseline EGRA data collected in May 2018. Because schools were randomly assigned to treatment, this analysis will allow for estimation of the causal impact of the SSE program following a full two years of program participation for the target pupils. • Cross-sectional pre/post comparison of SSE grade 1 pupil EGRA scores in May 2016 to SSE grade 1 pupil scores in May 2018. This comparison will help USAID/Morocco understand how reading scores in the early days of the SSE rollout to compare to scores after two full years of implementation. Time series analysis using grade 1 scores collected in May 2017 can shed further light on how pupil performance is trending over time.31 • Cross-sectional pre/post comparison of (non-SSE) grade 1 pupils targeted by the NPR national scale-up. Comparing MASSAR data on retention, repetition, dropouts, and performance (where available) for grade 1 students in the 2016/17 (pre) and 2018/19 (post) school years can offer early evidence on the efficacy of the scaled version of the NPR program. • Cross-sectional statistical contrast of grade 1 EGRA performance at SSE “high touch” schools with performance at NPR scale-up “low touch” schools using May 2018 EGRA data. This analysis may provide evidence as to whether observed results can be sustained at scale. As described in previous sections, the evaluation team will use qualitative field work as an opportunity to collect reliable information on the availability and relevance of further secondary quantitative data sources—including internal M&E data from implementing partners—that could support supplementary analysis and further triangulate the evaluation questions. Qualitative and quantitative findings will be integrated while the final report is written. For relevant research questions, secondary quantitative analysis will be utilized to contextualize, supplement, and/or triangulate the qualitative findings. If divergent findings emerge during analysis, the evaluators will re-examine the raw qualitative data through a more in-depth content analysis to determine if any other factors or issues were missed when data was first being organized in order to gain a deeper understanding of the data. In addition, disaggregated data may be reanalyzed by characteristics of respondents – such as locale of the respondent, respondent type, etc. - to help provide additional explanations for the variances the findings. 31 For cross-sectional comparisons, it is important to use data from collected at the same time point during the school year. As such, there is no comparable “post” measure for the true SSE baseline, which was conducted in January 2016. While the SSE program was already in effect during the May 2016 EGRA, it had only been operating for three to four months at the time of data collection; in May 2018, the program would have been in effect for two full years. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 27 REPORTING This evaluation includes the following deliverables and submission dates: # DELIVERABLE ESTIMATED DELIVERY 1 Evaluation work plan March 29, 2019 2 Evaluation design report May 31, 2019 3 Draft evaluation report (English) November 8, 2019 4 In-country validation presentation December 5, 2019 5 Final evaluation report (English) December 20, 2019 6 Evaluation executive summary (Arabic and French) January 15, 2020 7 Final remote presentation January 28, 2020 The primary audience of the full evaluation report is the USAID/Morocco Education Office in Rabat. The full evaluation report will adhere to the USAID Office of Education’s reporting templates and USAID guidance for evaluation reports. The draft report will be submitted for feedback in November. In addition to reporting on the extent to which the activities under RFS contribute to achieving the Education Development Objective, the data and major findings will be summarized in the conclusion section of the report. This section will include a discussion of lessons learned and provide specific recommendations for the implementation of RFS-NPR and future activities, including the design of the next CDCS. Recommendations will be action-oriented, practical, and specific, and will note which actors or stakeholders are best situated to implement or take up the recommendations. After the draft report is submitted, the evaluation team will share and validate the evaluation findings at a workshop in Morocco with Mission staff, implementing partners, and MoE officials. This workshop will also provide an opportunity to further refine recommendations to help ensure that they are specific and actionable. NORC will submit the final evaluation report to USAID in December following written feedback from USAID and validation of findings at the workshop. The draft and final evaluation reports will include executive summaries in English. The evaluation team will translate the executive summary into French and Arabic in January. The primary audience of the executive summary is the Moroccan MoE and related stakeholders in the education system. Clearly articulated recommendations for appropriate audiences will be included in the translated executive summary. Evaluators will complete all reporting and dissemination activities by the end of January 2020. LIMITATIONS OF THE PROPOSED APPROACH EVALUATION DESIGN As a primarily qualitative performance evaluation, the majority of activities under this study will not be able to speak directly to the causal impact of RFS activities. Rather, the evaluation will examine and document RFS successes, challenges, and lessons learned from written documentation (e.g., DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 28 project reports) and from the perspectives of program implementers, stakeholders, and beneficiaries. Findings from the evaluation will be utilized to generate and provide recommendations to the USAID Education office on future planning in education activities and strategy development. LINGUISTIC AND CULTURAL COMPLEXITY The evaluation will review program successes, challenges, and lessons learned across diverse languages and cultural contexts, including deaf communication. In order to minimize misunderstandings and maximize learning, the evaluation teams will include researchers with French and/or Arabic language skills as well as previous experience working in the Deaf education sector in Morocco. The teams will also keep running records of linguistic and cultural issues that arise during data collection to help further contextualize findings. FIDELITY OF IMPLEMENTATION In some cases, it may be difficult to disentangle program successes, challenges or lessons learned from implementation fidelity issues. For example, the desk review found that the scale-up of NPR was limited by strikes of inspectors who were designated to carry out training in NPR scale-up regions. While this development may have undermined program effectiveness, it is prima facie due to implementation challenges rather than flaws in the project’s underlying theory of change. REPRESENTATIVENESS OF SAMPLE Respondents for this primarily qualitative evaluation are not randomly selected and may not be representative of the full range of experiences among participants in the RFS project. The small sample of qualitative respondents may not reflect the perspective of the groups they were selected to represent, and systematic sampling error or bias might be present. This evaluation also does not target the most highly marginalized populations, who may not be in school at all. The study team will critically evaluate qualitative responses and triangulate among populations to discern accuracy. RISK OF DIVERGENT FINDINGS Different individual respondents and groups of respondents may have different views on program effectiveness. While responses will be triangulated, determining whose information is correct in the event of conflict will be challenging. In addition, qualitative and quantitative findings may diverge in some cases. Evaluators will acknowledge diverse perspectives in the final report as well as re￾examine the raw qualitative data through more in-depth content analysis to determine if any other factors or issues were missed when data was first being organized in order to gain a deeper understanding of the data. In addition, disaggregated data may be reanalyzed by characteristics of respondents – such as locale of the respondent, respondent type, etc. - to help provide additional explanations for variances in findings. AVAILABILITY OF RESPONDENTS RFS-SSE, RFS-HICD, and RFS-IDCRT have already closed, which will limit the availability of personnel for interviews. In addition, it may be difficult to get teachers to participate in interviews outside of school hours if they are not compensated. When possible, the evaluators will conduct interviews with stakeholders and personnel no longer residing in Morocco via Skype or telephone. Special efforts will also be made to accommodate respondents’ schedules or preferred methods of communication. Based on prior experience, a reasonable response rate is expected. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 29 ACCURACY OF QUALITATIVE RESPONSES Respondents may be biased or not feel comfortable speaking freely. Data collectors will explain confidentiality and anonymity of responses during consent procedures to ensure respondents are comfortable providing frank, accurate data. Within FGDs, facilitators will remind participants to respect the privacy of their fellow participants and that the information discussed needs to remain confidential—i.e., not to repeat what is said in the focus group to others. In addition, the consent procedures will clearly articulate that there are no direct benefits for participating in the study so that respondents do not provide misleading information in the hopes of receiving some material benefit in the future. WORK PLAN SCHEDULE Work began in March 2019 with work planning, initial meetings, and a desk review of RFS project literature, which was submitted to USAID on April 22. From March to May 2019, the evaluation team prepared workshop agendas, presentation materials, participant invitations, and stakeholder maps. The evaluation design workshop was held on May 7 in Morocco, with related briefings with USAID/Morocco conducted on May 6 and 10. As of this writing, the evaluation team is finalizing evaluation questions and drafting data collection tools in English for initial USAID approval in June. After USAID approves the English tools, the evaluation team will translate them into Arabic for USAID approval by July as well as obtain IRB approval or exemption. In August, Research Coordinators hired in Morocco will pre-test the data collection tools in and around Rabat. The evaluation team will revise the tools to incorporate pre￾test learnings and submit revised tools to USAID in late August. Field work planning will take place during summer. STS will lead the identification of respondents for qualitative data collection. Using the sample identified by STS, USAID/Morocco will ensure that the MoE provides approval for data collection before field work begins. Data analysis and reporting will occur from September to December. The evaluation team will code qualitative data in the field during collection and continue coding and analysis into late October. The team will submit a draft report to USAID by mid-November and validate the initial findings during a workshop in Morocco at that time. The revised final report will be submitted at the end of December after incorporating feedback from USAID and input from the validation workshop. The executive summary of the final report will be translated into French and Arabic for MoE dissemination in early January. A final remote presentation of these French- and Arabic-language results will conclude all evaluation activities. EVALUATION TEAM Members of the evaluation team and their key responsibilities are summarized below. Evaluation Team Leader Kristina Solum (STS) will lead the overall management of the evaluation as well as bring key technical expertise to the evaluation design and execution of the evaluation design workshop, operational data collection, and report writing. Ms. Solum has experience working in Morocco and will lead work related to students who are deaf and hard of hearing. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 30 Education Evaluation Expert Mark Lynd, Ed.D. (STS) will supervise the quality of the evaluation design and its execution. He will provide quality control for data collection and analysis, identify themes and lessons learned, provide links to other evaluations, and disseminate the findings. Dr. Lynd is a French speaker and will participate in the field work. Evaluation Expert Erika Keaveney (NORC) will provide technical and quality oversight to the evaluation, including design workshop planning, design report development, and reviewing key deliverables. In addition, she will lead supplemental quantitative data collection and/or analysis. Ms. Keaveney is an impact evaluation and quantitative research expert and has led several large-scale evaluations of USAID’s early grade programming in Africa. Qualitative Evaluation Expert Casey McHugh (STS) will advise the team about qualitative components and gender issues during evaluation design, data analysis, and report writing. Ms. McHugh is an applied anthropologist with over 10 years’ experience in gender, international education, and M&E for USAID education activities. Evaluator Matthew Murray (STS) will support study design, data analysis, and report writing. He is an experienced development evaluator and researcher with over eight years of experience in international education. Research Analyst Alexandre Monnard (NORC) is a native French speaker with extensive experience in the collection and analysis of both qualitative and quantitative data for international performance evaluations. Mr. Monnard will support field work and data analysis. Research Coordinators (STS), to be hired, will provide Arabic and Amazigh-language data collection capacities. At least five coordinators will be recruited in Morocco to participate in data collection. All locally-recruited data collectors will be fluent in Arabic, as well English or French and Amazigh languages, as appropriate. Please see Annex IV for a description of the roles and responsibilities of USAID, NORC, and STS. Annex V includes an evaluation timeline. Annex VI details the field work schedule, which will be revised in collaboration with relevant stakeholders prior to the beginning of data collection. DISSEMINATION PLAN The evaluation team will disseminate and discuss deliverables with stakeholders throughout the evaluation process. A May 2019 evaluation design workshop with USAID and MoE representatives in Rabat informed the evaluation design. The design workshop aimed to: • Ground the evaluation in local context and realities; • Refine the preliminary evaluation questions; • Identify the most appropriate data sources and methods for answering the evaluation questions; • Build consensus and buy-in on the evaluation approach; and • Finalize the evaluation work plan. Following data collection, coding, and analysis, the evaluation team will share findings with key stakeholders, including implementing partners of the four RFS activities, at a validation workshop in Morocco. The presentation will be in English or Arabic, as determined by the evaluation team and USAID. In the validation workshop, evaluators will solicit feedback from stakeholders about the DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 31 overall findings and will update the final report accordingly. Implementing partners, in particular, may provide a written statement of differences in the event of differences of opinion, which will be appended to the final evaluation report. Evaluators will also work with stakeholders to identify methods for dissemination of the final report. After completing the final report in English and translating the executive summary into Arabic and French, the evaluation team will present their results remotely, utilizing technology platforms to provide an opportunity for stakeholders in the United States, Morocco, and other countries, as appropriate, to learn about evaluation findings. The presentation will be in Arabic or French. All products designated for public dissemination by USAID/Morocco will be publicly available on the Development Experience Clearinghouse. Any anonymized primary quantitative data gathered will be submitted to the Development Data Library. The evaluation team may present findings from the evaluation at relevant conferences such as the Comparative and International Education Society annual conference as appropriate and agreed upon with USAID. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 32 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS / STATEMENT OF WORK C.1 PURPOSE OF THE EVALUATION The purpose of the performance evaluation of the RFS project is threefold: 1) To assess the extent to which all the activities under the RFS project together contribute to achieving the Education Development Objective; 2) to inform the implementation of ongoing or upcoming activities, including that of the National Program for Reading; 3) to inform the next Country Development Coordination Strategy design. The results of the RFS evaluation will be used by the USAID/Morocco Education Office to inform the design of the new strategy and make any adjustments needed to improve the implementation of the National Program for Reading and other activities, including those related to deaf education. The results will be shared with broader USAID audiences, including USAID/Washington and other missions, to highlight best practices and lessons learned, and to inform upper management of the results achieved by the Morocco mission in basic education and early grade reading. Finally relevant results will be shared with Government of Morocco (GOM) counterparts and other donors to further promote improving basic education in Morocco through enhanced reading instruction in early grades. Note: This evaluation is not intended to be an impact evaluation. As explained in details in the present SOW, the Evaluator is expected to use a primarily qualitative approach to addressing the research questions of interest. Nonetheless, the Evaluator is highly encouraged to consider and suggest the collection and use of quantitative data, whenever possible and relevant, to help strengthen and nuance the qualitative findings. Whenever both qualitative and quantitative data are used, the Evaluator should clearly describe how it would effectively weigh and integrate the two types of data to address the research questions. C.2 SUMMARY INFORMATION OF ACTIVITIES FOR RFS PROJECT ACTIVITY NAME IMPLEMENTER AGREEMENT/ CONTRACT # TOTAL ESTIMATED COST TIME FRAME (START/ END DATE) GEOGRAPHIC REGIONS Reading for Success - SMALL SCALE EXPERIMENTATI ON (RFS-SSE) Chemonics, International AID-608-TO-15- 00002 $5,820,752 Sept 22, 2015 to March 22, 2018 4 regions: Rabat￾Kenitra; Oriental; Souss-Massa; Fes￾Meknes Reading for Success - NATIONAL PROGRAM FOR READING (RFS￾NPR) Creative Associates International, Inc. (CAII) AID-608-C-17- 00001 $19,272,436 May 19, 2017 to May 19, 2022 National DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 33 ACTIVITY NAME IMPLEMENTER AGREEMENT/ CONTRACT # TOTAL ESTIMATED COST TIME FRAME (START/ END DATE) GEOGRAPHIC REGIONS Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT) Institute for Disabilities Research and Training (IDRT) AID-608-A-15- 000002 $1,563,935 Oct 9,2015 to Oct 8, 2018 8 provinces: Sale, Rabat, Tangier, Marrakech, Agadir, Fes, Meknes, Casablanca Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD) Kaizen Institute AID-608-TO-16- 0001 $2,013,404 Jun 16, 2016 to June 30, 2017 National The SOWs and the Monitoring, Evaluation and Learning Plans will be shared with the bidders as references, and these will include information on beneficiaries for each of the activities; activity-level data collected, the timing of that data, and the sample size for that data, and all reports completed to date for each activity. C.3 BACKGROUND Description of the Problem Problem Statement: In the past decade, Morocco has made significant gains in making primary schooling nearly universal with net enrollment rates in grade one surpassing 97%. However, in terms of learning achievement, Morocco lags behind other lower middle income countries, and has nearly the lowest overall literacy rates in the MENA region. The youth literacy rate is 87% for males and 72% for females32. These numbers hide significant disparities in literacy between rural and urban areas in Morocco, with a staggering illiteracy rate of over 51% among rural girls33. Based on enrollment and literacy data, the United Nations Development Program (UNDP) considers Morocco to be 15 years behind Tunisia, 10 years behind Algeria, and 5 years behind Egypt in terms of overall educational outcomes34. Ultimately, performance indicators for each of the RFS activities are focused on closing this gap. On international assessments, Moroccan students rank among the lowest achievers. In the 2011 Progress in International Reading Literacy Study (PIRLS) Moroccan fourth graders ranked last among the 48 participating countries, with an average scale score of 310 (PIRLS scale centerpoint is 500). In 2007 and 2011, Morocco ranked eighth out of the nine participating Arab countries, in the Trends in International Mathematics and Science Study (TIMSS)35. 32 UNESCO 2011, including the Education for all 2000 Assessment. 33 Direction de la Lutte Contre l’Analphabetisme (DLCA), Ministry of Education, 2011. 34 World Bank Report No: 52749-MA, 20 April 2010. 35 PIRLS and TIMSS 2011 Reports. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 34 A National Analysis of Student Performance (PNEA) conducted in 2008 by the Higher Council of Education (a research and watchdog organization reporting directly to the King) showed that only 24% of fourth graders and 36% of sixth graders possess the basic reading skills mandated in the national curriculum. The study showed that less than 32% of students could read at grade level. At all four grade levels examined in the study, girls outperform boys in reading but not in the other subject areas. It is hypothesized that girls seemed to better master the various reading sub-skills than their male peers. This national study associates the causes of poor levels of literacy to a multitude of factors including a weak reference framework for reading skills (standards and benchmarks), a rigid and overloaded curriculum that does not respond to children’s needs, inadequate reading assessment tools, a school environment not conducive to learning, and the persistence of inequality in access to school and education opportunities36. In 2011, USAID funded an Early Grade Reading Assessment (EGRA) in the Doukkala Abda region. The EGRA mirrored previous findings, and showed that only 34% of students in grades two and three read well enough to fully comprehend a grade-two level text. Boys and girls equally lack basic letter naming skills and are not able to sound out simple words, and 33% of second graders and 17% of third graders (21% for girls and 29% for boys both grades combined) could not read a single word of text in Classical Arabic, the language of instruction in the formal education system37. Only 2.5% of students tested responded correctly to five of the six reading comprehension questions on the EGRA. The assessment showed that factors inhibiting student performance are multifaceted and complex: • Inadequate levels of teacher preparation: Both the Snapshot of School Management and Effectiveness (SSME) component of the study and a recent review of the pre-service teacher education curriculum showed that Moroccan teachers receive little to no specialized training in reading instruction and assessing students’ reading abilities, leaving them ill- equipped to help children excel in the early grades. The absence of supplemental reading material for all students at school severely constrains the overall learning environment. • The absence of reading materials in the home environment: Many children do not experience a print-rich environment at home. It is well-documented that children whose family members read to them and who grow up in homes where reading is valued do better in school. The SSME showed that few grade two and three students read outside the school environment. Only 9.5% of students reported using their textbooks at home; an equal percentage reported reading at home every day. Over 52% of students reported never reading aloud at home, and 32% reported sometimes reading aloud at home. • The lack of parental involvement in student and school performance: While parents reported meeting with teachers at least once a year, teachers largely reported parental involvement to be insufficient. • The language context in Morocco adds further complexity to the learning environment. Classical Arabic is the official language of instruction. However, children’s first exposure to Classical Arabic happens at school. Moroccan students’ mother tongue is either Arabic Dialect (Darija) of one of the Amazigh Dialects. • The EGRA report showed that there are only minimal achievement differences between boys and girls in the early grades. However, gender disparities in learning outcomes seem to 36 PNEA Report, May 2009, Higher Council for Education. 37 EGRA/EGMA/SSME Report, RTI, 2011. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 35 progress as students get older and as cultural beliefs about gender roles particularly in rural areas are reinforced. The primary school dropout rate is higher for girls (2.7% compared to 1.2% for boys in 2012-2013). Girls begin to drop out in the fourth grade (1.8% for girls and 1.3% for boys), and their rate of dropout increases (9.9% for girls and 2.1% for boys in grade 6)38.The repetition rate in primary school was 10.5% with 12.6% for boys and 8.2% for girls in 2012-201339. The data show that gender disparities exist, with more boys repeating grades in primary school and more girls permanently leaving school at a young age. Although several factors explain why students drop out, numerous studies40 have found that the most effective way to reduce the number of children who will ultimately drop out is to provide the best possible classroom instruction from the beginning of their school experience through the primary grades. The road to academic success starts early in the education system. Early grade reading proficiency has been cited as indicators of future academic success. • Reading is a foundational skill and the basis of most other academic learning. Thus, it is crucial to ensure that students acquire good reading and comprehension skills in the first three grades so that they have a more equal opportunity to master other skills and knowledge41. The gap between poor and good readers continues to widen as they advance through primary and into middle school. Children who do not read well at the primary level are often on a lifetime trajectory of limited educational progress and therefore limited economic and developmental opportunity. Having all children reading well and completing primary school is foundational for a more equitable society and lower disparities in human and economic development across Morocco. Struggling readers will experience difficulty in later years and, because catching up is nearly impossible without a strong remedial support system, may opt to drop out instead. At the primary level, 90% of students who start grade one complete the primary cycle (89% for girls)42. However, primary school dropout begins, in earnest, starting at grade four, and proceeds to the point where only 18% of a cohort of first graders will reach Grade 12. Despite dropout prevention efforts, over 300,000 children and youth under the age of 15 leave the formal school system every year. Approximately 10% of primary school enrollment in 2012-13 were students repeating at least one grade. Two-thirds of primary school repeaters were boys. While the MoE data does not indicate the numbers of students repeating a grade for the second time, officials in the School Dropout Prevention program in the MoE state that students who have repeated two grades or more are twice as likely as one-time repeaters to drop out of school completely. 38 Ibid. 39 Ibid. 40 Ekstrom et al., 1986 : Who drops out of high school and why?; Goldschmidt & Wang, 1999: When can schools affect dropout behavior? ; Rumberger, 1995: Dropping out of middle school: A multilevel analysis of students and schools. American Educational Research Journal, 32, 583-625; Swanson & Schneider, 1999: Students on the move: Residential and educational mobility in America's schools. Sociology of Education, 72, 54-67; Wehlage & Rutter, 1986: Dropping out: How much do schools contribute to the problem? Teachers College Record, 87,374-392 41 Moats, 1999, Teaching Reading is Rocket Science: What Expert Teachers of Reading Should Know and Be Able to Do, American Federation of Teachers. 42 Persistence to last grade of primary is the percentage of children enrolled in the first grade of primary school who eventually reach the last grade of primary education. The estimate is based on the reconstructed cohort method. Source: United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 36 Development Hypothesis(es) There is growing evidence that the quality of service delivered is an important factor in enrolling and keeping children in school. A study43 on grade repetition in primary schools shows that in order to effectively reduce grade repetition, reforms need to tackle the reasons behind poor attendance, the quality and relevance of teaching, and provide sufficient support for teachers. The Reading for Success project supports the GOM Strategy for improving the quality of service delivery to citizens – by improving equity, access and quality of education service delivery by 2016. The Project will enhance educational attainment for children at the primary level in targeted areas in three ways: by supporting the MoE to increase the reading abilities of learners enrolled in the early years of primary education; by increasing community support for reading; and by improving the policy environment for reading. The development hypothesis is based on the premise that if USAID supports the improvement of reading skills for students in formal and non-formal primary education, parental and community support for reading, and strengthens the MoE’s capacity and policies to support reading, then student learning outcomes will improve, repetition rates will decrease, and retention rates for students through the primary cycle will increase. These interventions will result in improved overall learning outcome for students in target areas. Helping students in the early grades to read at or above a proficient level will prepare them for future success. Through the provision of technical and material assistance, the project will develop a strong evidence base on successful interventions to improve reading outcomes in Moroccan classrooms. It will support the use of evidence-based reading materials, improved reading instruction, and improved school-based reading enrichment programs. This will be reinforced by increased family engagement in supporting reading, and increased civil society organization (CSO) capacity to implement reading activities. The evidence base developed through these activities will inform policy for standards and benchmarks for early grade reading (EGR), define EGR instructional reading time, and establish national assessment protocols for EGR. Description of the Strategy/Project/Activity/Intervention to be Evaluated The Reading For Success (RFS) Project The Reading for Success project is designed to enhance educational attainment for Moroccan children at the primary level. Specifically, the project supports the Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MoE) in its curriculum reform initiative by developing curricula and instructional reading materials in Arabic for grades 1-4, including students with disabilities; by providing training and coaching to teachers; and by implementing a community mobilization approach tailored to support student reading. In addition to developing and implementing a new reading instruction approach, a human and institutional capacity assessment is also necessary to help identify the Ministry's needs to carry out their reform effectively. The project works with Government of Morocco (GOM) counterparts to develop an evidence base for the design of effective project interventions aimed at improving reading skills, both in targeted primary schools and in non- formal education settings. The project supports a principal objective of USAID/Morocco’s Country Development Cooperation Strategy for FY 2014-2018, “Enhanced educational attainment for children at the primary level,” as well as the Agency-wide Education 43 Jonathan Penson, “Grade repetition in primary schools in Sub-Saharan Africa: An evidence base for change,” February 2008. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 37 Strategy. The project comes at a time when the Ministry of Education (MoE) is developing a significant education reform, including a National Strategy for Reading. The ministry has specifically asked USAID to be its lead international partner to help find effective approaches at the community, school, and classroom level to improve the reading skills of Moroccan children. The project will be implemented over a five-year period (FY 2014-2018) at a total estimated cost of approximately $25 million. The RFS project is composed of four activities (described in more details below). These four activities are designed around three principal streams as outline below. • Improved Reading Skills in Grades 1-4 • Increased Community Support for Reading • Improved Policy Environment to Support Reading See Development Objective (DO 3) Logical Framework in Folder 05, REF # 2 1- Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE), implemented by Chemonics: The activity was awarded as a 30 month Task Order (September 23, 2015 to March 22, 2018) with a total estimated cost of $5,820,752. The activity concluded March 2018. The objective was to support the Ministry in its curriculum reform initiative by developing reading instructional materials in Arabic for grades 1-2; providing training and coaching to teachers; and, implementing a community mobilization approach tailored to support student reading. In its first year, RFS-SSE focused on Result 1 – Developing Arabic Reading Materials for Grades 1 and 2. Based on international best practices and Moroccan expertise, RFS-SSE introduced the phonemic method of teaching, which uses individual syllables and sounds as building blocks for basic reading skills. RFS￾SSE engaged with CSOs through small grants to design and implement summer reading activities. The CSOs engaged students from the target provinces and schools in the reading activities. Chemonics tested via EGRA the impact of these summer programs on reducing students’ reading loss over the summer. 2- Reading for Success - NATIONAL PROGRAM FOR READING (RFS-NPR), implemented by Creative Associates International, Inc. (CAII). This is a $19.2 million five-year activity (awarded May 19, 2017 through May 18, 2022) which aims to build on the data, evidence and lessons learned from the experimental reading activity (RFS-SSE) to further improve reading instruction and learning materials; expand the implementation of reading instruction practices to grades 3-4; and support the Ministry in developing a new Arabic language curriculum (grades 1-4) that the Ministry will scale up nationwide. The National Program for Reading will: a. Improve reading and writing materials used in primary schools: The materials will be aligned to an updated and improved Early Grade Reading and Writing (EGR/W) curriculum with clear standards, benchmarks and appropriately sequenced content. Textbooks will be aligned with the new EGR/W curriculum and will meet expectations for usability, complexity/readability, content and quality. In addition, the program will support the MoE to develop Arabic Language Curriculum and Pedagogical Guide in the Early Grades and establish Arabic language benchmarks with an objective method for monitoring progress of children throughout the year and governments with a method to communicate performance standards. b. Improve reading and writing instruction in primary schools: NPR is working to develop and deliver an improved training program encompassing extensive preservice and in-service training. c. Expand Reading and writing enrichment programs: In addition to improved reading and writing materials and effective instruction, students need access to reading and writing enrichment opportunities outside of formal school hours and in the home. The core package will offer DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 38 enrichment reading clubs that meet multiple times a week after school, tutoring, one-on-one support, a buddy system and more. d. Improve learning and assessment systems: to support continuous learning and improvements to the curricula, teaching methods, teacher training, reading and writing materials, and enrichment programs requires a cohesive learning and assessment system. The system will provide Creative, USAID, MoE, teachers, administrators, and others to identify blockages, assess effectiveness, and generate evidenced to design, refine and improve EGRA/W package components. The system will include a research agenda, implementation of a series of evaluations, assessments, analytical tools and participatory workshops. This will allow MoE to institute an adaptive management approach to inform program revision, support scale-up, and create a support policy environment and accountability system. In addition to these goals, and as a cross cutting task, RFS-NPR is building the capacity of the Ministry of Education for sustained implementation. Local capacity building at every level is critical for program implementation, and to sustain the ongoing use of evidence-based approaches after the activity ends. Program implementation is designed to support the role of the Ministry of Education and build its capacity at the central and provincial levels, employing innovative and practical modalities, strategies and activities that will reinforce teacher knowledge and practices that ensure the success of the program beyond the life of activity. Capacity building, which is one of the indispensable elements for ensuring country ownership and responsibility, can be only reached if the activity is carried out hand-in-hand with MoE key officials at all levels. In this sense, RFS-NPR capacity building approach goes beyond simple organization of and participation in capacity building workshops and trainings. RFS-NPR is seeking ways of providing the MoE with capacity building opportunities in real settings and in a practical and comprehensive manner. RFS-NPR will develop a milestone plan with the MoE and use this, alongside regular feedback loops, to ensure the effective delivery of this component and sustainability of results. In addition to hands on learning, RFS-NPR may include partnerships with external entities, such as national and international universities, to provide MoE officials and technical staff with focused and intensive training (including training with certification) in key areas such as early grade reading and child literacy, curriculum development, teacher training, coaching, assessment, and other. This evaluation is expected to coincide with the last quarter of the second year of NPR implementation (Jan-Feb 2019). By then, most of the instructional materials would have been designed, training of trainers would have taken place, and Grade 1-4 curricula would have been tested and implemented in the experimental schools. The detailed workplan of NPR will be provided for reference. Quarterly and Annual reports will be shared with the winning contractor as they become available. 3- Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): This activity was awarded to Chemonics in October 2015, with a total estimated cost of $1,563,935 and a revised end date of October 2018. The objective of this activity is to improve the availability and quality of reading materials and instruction available for deaf children in Grades 1-3, and ultimately to improve their literacy through: the development of software that helps teachers and parents easily create and publish Moroccan Sign Language￾supported educational materials; training of teachers; and assessment (Early Grade Reading Sign Language Assessment, EGR-SLA). A cost extension of the Cooperative Agreement was signed in April 2017 extending the project until October 2018 and increasing the budget to accommodate the added activities. These activities primarily cover another year of EGRA-SLA testing, more teacher training, and the expansion of MSL Clip and Create to include more functionality and database expansion, including Chemonics’ first and second grade national curriculum words, and regional variations. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 39 4- Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): This $2,112,724 award to The Kaizen Company aimed to ensure that USAID and the Ministry have a full understanding of the human and institutional challenges that may hinder the implementation of the Ministry’s reform efforts, and that the Ministry would receive a package of actionable solutions to face identified challenges. The purpose of the HICD Analysis of Morocco’s MoE was to support the implementation of the 15-year reform entitled “Vision 2015- 2030”, especially the components that will be supported by both USAID/Morocco and the Millennium Challenge Corporation (MCC). The activity was procured in June 2016 and concluded June 30, 2017. Activity Monitoring, Evaluation, and Learning (MEL) Plans Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE) (Reference: FOLDER 01 RFS-SSE, SUB-FOLDER 01.2, REF# 1): The purpose of this MEL Plan is to collect data, and assess accurately the progress of RFS-SSE towards its goals. The RFS-SSE team uses the collected data to measure the success of project efforts to identify first and second graders’ reading competencies via EGRA and SSME tools in order to provide data that will assist in identifying interventions to improve reading skills as well policy environment to support reading. The MEL Plan describes the RFS-SSE project methodology for activity monitoring and evaluation and is used to manage the collection of performance data. The MEL Plan includes a performance indicator summary table (pages 16-22) that includes performance indicators, data sources and collection methods, timing of data collection, disaggregation, and units of measure. The results are used to regularly update and engage relevant stakeholders, including USAID, to analyze progress and quality. In addition to the MEL Plan, both DevResults (Chemonics) and TraiNet (USAID) are used to store the collected data. The DevResults Monitoring and Evaluation (M&E) software are used for the project’ M&E data collection, storage, management, and reporting. Its web-based interface can be used from any internet-enabled device (tablet, mobile phone, desktop). Where internet access is unavailable or inconsistent, data can be captured via formatted Excel spreadsheets and other offline tools and later uploaded. Features include Geographic Information System (GIS) functions, data visualizations, results dashboards, and document and photo storage capabilities. All documents will be provided upon award by the Education Office and Chemonics. Reading for Success - NATIONAL PROGRAM FOR READING (RFS-NPR): RFS-NPR employs several highly consultative and participatory research and evaluation methods to test activities and support scale-up. The “stepped” approach (1) tests the effectiveness and viability of the EGR/W instructional and enrichment interventions in schools; (2) provides the opportunity to identify and alleviate policy, procedural, and resource blockages in the delivery and support/ supervision systems at local, provincial, and regional levels; and (3) supports the MoE roll-out of its EGR/W program nationally. With the Assessment Quality Assurance Working Group (AQA-WG), led by CNEE, Creative is developing a research agenda, reviewing SSE results, detailing key research questions and identifying the types of studies required. The agenda includes methods, sampling, and periodicity. The research agenda should be updated annually to reflect emerging areas of inquiry. Some of the research questions that RFS-NPR may address include: • What are the major policy factors that significantly contribute to or inhibit the project effectiveness in the Morocco context? • What should successful teacher profile for teaching reading look like and how should teachers be supported in terms of training, in-school support and community support? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 40 • Which one of the four enrichment models that significantly contribute to student reading outcomes and to what extent? • What types of correlations exist between teachers’ classroom practices and student reading outcomes and to what extent? The performance indicator summary table is available in (Reference: FOLDER 02, SUB-FOLDER 02.2, REF # 1) and outlines performance indicators, data sources and collection methods, timing of data collection, disaggregation, and units of measure. Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): The proposal for Improving Deaf Children’s Reading Through Technology (IDCRT) was written using a logic model structure. It contains hypotheses, goals, objectives, tasks, inputs, outputs and outcomes. These are incorporated both into the body of the proposal and into the Life of Activity Performance Plan. The purpose of this structure is to clearly set the course of the project’s implementation, explicitly define those measures and products that provide evidence of completion and quality, and to monitor progress on an ongoing basis so as to provide for timely corrective actions and sustaining efforts that bolster positive outcomes. Monitoring of project implementation is specifically addressed within Objective 12 of IDRT’s workplan, is highly structured, and has a number of measures incorporated to facilitate month-by￾month analysis and assertion of compliance. Each of the 48 tasks is tethered to the timeline, so that at any given moment it can be ascertained whether the project is staying on task and on time. (Reference: FOLDER 03, SUB-FOLDER 03.2 REF # 1 and 2) Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): The Monitoring Evaluation & Learning System (MELS) provided the overall framework for Monitoring and Evaluation (M&E) and Learning in the Morocco HICD Assessment Activity. It served as the basis for generating, analyzing and disseminating a basic set of information useful to set broad activity targets and to chart the progress of Morocco HICD in the attainment of these. (Reference: FOLDER 04, SUB-FOLDER 04.2, REF # 1) C.4 EVALUATION QUESTIONS The evaluation questions for the RFS project evaluation below may be further developed and finalized in collaboration with USAID prior to the finalization of the evaluation design. The Evaluator will have access to student reading performance data as well as performance monitoring data for all activities collected by the Implementing Partner (IP) (as described in above). For the ongoing RFS-NPR, EGRA baseline data was collected in May 2018 and raw data as well as analysis should be readily available to the Evaluator. Midline and endline data will not be available as midline data is expected to be collected in May 2020, and endline data in May 2021. The Evaluator will be expected to use the IP’s monitoring data (student reading performance); and will not be expected to collect primary reading assessment data. Additional sets of quantitative data may be available through the host country Ministry of Education website, and could be useful to analyze by the evaluation team: https://www.men.gov.ma/en/Pages/Statistics.aspx. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 41 The Evaluator is expected to disaggregate results by gender, as relevant and whenever possible. As relevant and necessary, the Evaluator should consider and suggest other variables to disaggregate the data by, such as by milieu (rural vs. urban) and/or by activity. This evaluation will focus on outcomes in classical Arabic only, and will not look at any aspects related to the use of mother tongue or other languages for instruction. RFS Project Draft Evaluation Questions Note: Evaluation questions are separated into two categories; (1) project implementation and management and (2) lessons learned and sustainability, which are described in more detail below. Offerors should not consider these categories or the order of the questions as a prioritization and give each category and questions equal weighting. Project Implementation and Management: Currently, the implementing partners (IPs) for the activities under this project are addressing related questions using the EGRA data and are looking at possible correlation of observed factors that may influence performance of students. Using robust, representative qualitative data, the evaluation questions under this dimension aim to enrich and deepen the IPs’ analysis and results by helping to shed light on what and how factors affect achievement of educational outcomes outlined in the project. 1. To what extent have the activities under the RFS project contributed to the overall goal of the RFS project of enhancing educational attainment for Moroccan children at the primary level? a. How and why did students under the RFS-SSE, RFS-IDCRT, and RFS-NPR activity realize the observed learning results? (In addressing this question, discuss the strengths and the gaps of the project activities in contributing to the objectives of RFS.) b. How did RFS activities intersect with, provide learning about, and result in progress towards each of the 5Ts of reading instruction? c. How have the activities performed against the higher level indicators at the project level? d. To what extent have the activities effectively used (or are currently using) activity-level M&E data to make necessary and relevant adjustments to program design, implementation and management? 2. How have the structure and management of the project and its activities facilitated or hampered the realization of the project goal? 3. Which populations of children currently in (or not in) the educational system are not benefiting from the current RFS project activities and how can they be better served in future projects/activities? Lessons Learned and Sustainability: Sustainability is a key component of success for the RFS project as many of the activities will be managed and implemented by the GOM moving forward. The following questions seek to better understand whether the project has contributed to sustainable outcomes to date and inform any future programming changes. 1. To what extent (and how) has the ministry taken over responsibilities of project activities and ownership of the national system for reading and literacy? 2. How did the MoE’s lead role in the reading and literacy project activities influence the implementation, management, and results of the activities? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 42 3. What key strengths or constraints affected the project activities’ ability to contribute to establishing a sustainable national system for reading and literacy? 4. What key lessons and recommendations in reading and literacy can be drawn from the successes and limitations of the RFS project in order to more effectively achieve the Education Development Objective (Enhanced Educational Attainment for Children at the Primary Level) as well as help inform Morocco's new CDCS? C.5 EVALUATION APPROACH In the subsections below, general guidance is provided on methodology, selection of respondents/sampling, data sources and collection, data analysis, and methodological limitations. In their responses, offerors should propose a specific evaluation strategy for each evaluation that specifically addresses the needs and requirements of each evaluation. Evaluation Design: The Evaluator is expected to employ a primarily qualitative evaluation approach to address the research questions outlined above. Whenever feasible and relevant, the Evaluator should consider and suggest the use of quantitative data to help strength and enrich the qualitative findings. The overarching analytical strategy should be clearly summarized in an evaluation matrix that includes the following information: evaluation questions, sub-evaluation questions, indicators or assessment criteria, sampling/selection criteria, data sources and collection methods, and data analysis methods. The evaluation team, in collaboration with USAID, will finalize the overall evaluation methodology for the evaluation before fieldwork begins. Evaluation Design Workshop: An evaluation design workshop is a key component of the evaluation and must be conducted as part of the evaluation design component. It is considered a best practice in the field of evaluation and is highly recommended. The purpose of such a workshop is to help further develop/ finalize the evaluation design plan and ground the evaluation to the context and realities on the ground. During the workshop, questions will be refined and finalized, evaluation approach will be further refined/finalized based on the realities of the implementation of the activities, potential issues related to sampling, data sources, timeline are discussed and addressed. It must be led in country by the Evaluation team, and should bring together relevant Mission staff, Implementing partners, and as appropriate, host-country/ministry stakeholders. The Evidence Team in the E3/ED office of USAID/Washington will provide technical support and will help facilitate the workshop. Selection of Respondents/Sampling Strategy: The Evaluator should carry out relevant data collection with a representative sample of purposively selected USAID project beneficiaries, partners, and host government counterparts at appropriate levels; and other key program stakeholders as appropriate. The Evaluator should propose a robust strategy for selecting respondents that will enable adequate geographical, language representation; cross-site comparison; and the capture a diversity of perspectives, among other key considerations. Final criteria for sample selection should be developed by the Evaluator in consultation with USAID. Data sources and collection methods: The Evaluator must collect both primary data and secondary data. Primary data may include, but is not limited to key informant interviews, focus group discussions, observations, and questionnaires. For secondary data, and whenever relevant, the Evaluator must consider complementary sources of secondary data, such as the IP’s performance monitoring data, activity documents, MoE national statistics44, etc. Prior to any field work, the 44 https://www.men.gov.ma/en/Pages/Statistics.aspx. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 43 Evaluator is expected to carry out a targeted review and analysis of recent pertinent published literature on educational program evaluations in Morocco and MENA as well as relevant project documents and performance data provided by USAID to: 1) gain a better and more in-depth understanding of the implementation of the RFS project and, 2) assess the quality and relevance of the data to help answer the evaluation questions. Primary data should only be collected if the data does not already exist via the IP’s monitoring data, school administrative data, census data, etc. The Evaluator is expected to collect at least one round of data collection and should outline a clear plan and specify which data exactly would be collected and why. The Evaluator should use, whenever possible, pre-existing data collection instruments with documented strong measurement validity and internal reliability and with a suitable plan for cultural adaptation. For new instruments, the Evaluator should propose a plan for the development, cognitive testing (to test reliability and validity of instruments), and piloting of the new data collection instruments. It should be noted that: • Prior to beginning field work (including field testing), the evaluation team is required to share data collection instruments with USAID for review and feedback before they are used in the field. • Issues of data confidentiality must be addressed as part of the IRB process. • The use of electronic data collection for quantitative data is required, whenever relevant. • Data collection methods must take into account contextual factors to maximize data reliability and minimize unintended consequences. Data Analysis: The Evaluator must analyze both primary and secondary data. All secondary data must be assessed for quality and relevance in answering the research questions. The Evaluator must provide a data analysis plan for the qualitative data, and for the quantitative data (if quantitative data is used) that directly addresses each evaluation question with specific methods for analyzing the data. As part of the plan, the Evaluator should specifically discuss how the qualitative and quantitative data will be weighted and integrated (whenever relevant) to address the evaluation questions. Data should be triangulated across multiple sources and data triangulation should be evident in the evaluation findings. Methodological limitations: As part of USAID’s evaluation policy, the Evaluator is required to discuss and document any issues potentially affecting the quality of evaluation data (including the data validity, integrity, timeliness, precision, and reliability) in the evaluation planning stage, and to be assessed on an ongoing basis during implementation. The Evaluation team and USAID should discuss all limitations and measures to address or overcome limitations in the implementation phase and these should be detailed in the reports. The evaluation reports must be clear and transparent about any notable limitations and if and how they may affect the evaluation findings, conclusions, and recommendations. [END OF SECTION C] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 44 SECTION F - DELIVERIES OR PERFORMANCE F.1 REPORTS/DELIVERABLES All reports and deliverables under this Contract must be submitted to USAID at the times indicated in the table below. The Contractor must, however, promptly notify the Contracting Officer’s Representative (COR) of any problems, delays or adverse conditions which materially impair the Contractor’s ability to meet the requirements of the contract. The Contractor must follow the language requirements listed in the table below, as needed. Required reports and deliverables are described as follows: A. Schedule of reports and deliverables for RFS project performance evaluation 1. Draft Work Plan: Contractor must submit a draft workplan within 2 weeks of award that describes in detail how the team intends to implement all tasks specified in the contract and provide a detailed timeline for the completion of all tasks. The workplan must include the anticipated schedule and logistical arrangements and delineate roles and responsibilities of members of the evaluation team. 2. Consultations & desk review report: Within 3 weeks after award, the contractor must discuss the draft work plan and present a summary of the desk review report. The discussion can be held by conference call. 3. Evaluation Design Report: The draft Evaluation Design Report (EDR) must be submitted within 5 weeks of award. The EDR must describe the overarching analytical strategy. The report must incorporate the final work plan and desk review report (deliverables 1 and 2 above) and include at minimum the following information: evaluation questions, sub-evaluation questions, indicators or assessment criteria, sampling/selection criteria, data sources and data collection instruments as well as methods, and data analysis and triangulation methods. A separate summary of the report must be provided in French (to be shared with the Ministry of Education). 4. Evaluation Design Validation Workshop45: An evaluation design workshop is a key component of the evaluation and must be conducted by the evaluation team in Morocco within 7 weeks of award. Feedback provided during the workshop is to be incorporated into the final EDR and must be approved by USAID before the evaluation team begins actual field work (i.e. data collection). The purpose of such a workshop is to help further develop/finalize the evaluation design plan and ground the evaluation to the context and realities on the ground. During the workshop, questions will be refined and finalized, evaluation approach will be further refined/finalized based on the realities of the implementation of the activities, potential issues related to sampling, data sources, timeline are discussed and addressed. It must be led in-country by the Evaluation team, and should bring together relevant Mission staff, Implementing Partners, and as appropriate host￾country/ministry stakeholders. 5. Interim-Briefing: Within 14 weeks of award (and following approximately 8 weeks of field work), the contractor must have completed the data collection process of the RFS project performance evaluation. The evaluation team will hold an interim￾briefing meeting, in Rabat, with USAID to present, in PowerPoint format, a brief 45 The Evidence Team in the E3/ED office of USAID/Washington will provide technical support and will help facilitate the workshop. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 45 overview of the preliminary findings and confirm completion of data collection. This briefing may include implementing partners or other stakeholders, as determined by USAID. 6. Draft evaluation report: Within 20 weeks of award (approximately 6 weeks after completion of data collection), Contractor must submit a draft evaluation report for the RFS project performance evaluation. The draft must be in English and must be submitted in electronic copy to the COR for review. 7. Presentations of findings & Infographic: Within 24 weeks of award, the evaluation team must make a presentation in Morocco of the evaluation findings to USAID and provide a 2-page (i.e. front and back of one piece of paper) infographic. After discussion of results and recommendations, any changes suggested by USAID must be included in the final report and presentation and infographic. An Arabic version of the presentation as well as a French and Arabic version of the infographic must be prepared and approved by USAID. The presentation will be presented to the Ministry of Education officials at a date and time agreed on with USAID and MoE. 8. Final report in English, with French and Arabic Executive Summaries: The contractor must submit the evaluation final report within 27 weeks of award. The report must include an executive summary; introduction; background of the local context and the projects being evaluated; the evaluation questions; the methodology; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable). The executive summary must be less than 5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned. Submission of draft French and Arabic executive summaries will take place only after the COR has approved the final English version of the executive summary. Deliverables Table KEY DELIVERABLES DUE DATE FOR SUBMISSION 1. Draft Work Plan Within week 2 of award 2. Consultations & desk review report Within week 3 of award 3. Evaluation Design Report (EDR) Within week 5 of award 4. Evaluation Design Validation Workshop Within week 7 of award 5. Interim-Briefing Within week 14 of award 6. Draft evaluation report Within week 20 of award 7. Presentations of findings & Infographic Within week 24 of award 8. Final report in English, with French and Arabic Executive Summaries Within week 27 of award DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 46 Deliverables Timeline F.2 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and ADS 201, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 1. The evaluation report must represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. 2. Evaluation reports shall address all evaluation questions included in the scope of work. 3. The evaluation report must include the scope of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline must be approved by the Contracting Officer and agreed upon beforehand by the COR. 4. The evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an Annex in the final report. 5. Evaluation findings must assess outcomes and impact on males and females. 6. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 7. Evaluation findings must be presented as analyzed facts, evidence, and data, and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence. 8. Sources of information need to be properly identified and listed in an annex. 9. Recommendations need to be supported by a specific set of findings. 10. Recommendations must be action-oriented, practical, and specific, with defined responsibility for the action. 11. Recommendations must be detailed separate from findings and conclusions. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 47 For further guidance on USAID quality evaluation policies, please visit http://www.usaid.gov/evaluation. F.3 OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, nonproprietary formats as required by USAID’s Open Data policy (see ADS 579). The data must be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions must be updated in the SOW that is included as an annex to the Evaluation Report. F.4 EVALUATION TEAM COMPOSITION The Evaluator must propose a staffing plan that fully supports the requirements of this SOW. At minimum, the staffing plan must: (1) list the proposed team members and clearly articulate each team member’s responsibilities within the team and their relevant qualifications to meet these responsibilities (2) describe how the team members will work together, and their anticipated LOE for the duration of the evaluation activities. As part of this response, the evaluation team will need to complete the table below and provide CVs and references for all team members proposed. EVALUATION TEAM EXPERIENCE Written disclosures of conflicts of interest are required from all evaluation team members. Key Personnel a) The key personnel that the Contractor must furnish for the performance of this contract is as follows: Position Title Name 1. Evaluation Team Leader TBD 2. Education Evaluation Specialist TBD b) The key personnel specified in this Contract are considered to be essential to the work being performed. Prior to replacing the above specified individuals, the Contractor must notify both the CO and the COR reasonably in advance, and must submit written justification (including proposed substitutions) in sufficient detail to permit evaluation of the impact on the program. No replacement will be made by the Contractor without the written consent of the CO. c) Minimum qualifications for the key personnel are as follows: Evaluation Team Leader: Team Member Name Title Team Member Responsibilities Team Member Relevant Experience DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 48 Role: The Evaluation Team Leader will be responsible for overall and day-to-day management of contract activities, involving multiple tasks across multiple locations. S/he is the principal interlocutor between the Contractor team members and USAID, and is responsible for the overall compilation of all deliverables under this contract. The Team Leader is expected to be the technical leader for the evaluation and therefore is expected to be involved in the dissemination of findings. S/he is responsible for staffing, project planning, project financials, and staff direction and oversight, and maintains and manages the relationship with USAID. S/he collaborates closely with technical evaluation staff to ensure quality of design, methodologies, tools, data collection processes and data analysis. Minimum requirements: • Graduate degree or higher in international development or related field. • Minimum 7 years of experience conducting qualitative research. • Education sector expertise required, with experience in Morocco and/or MENA region preferred. • Has led previous education evaluations. • Strong background and training in qualitative research methodologies • Experience and knowledge of data analysis and data management techniques • Experience managing and coordinating complex evaluations, including evaluations of international projects. • Strong organizational, oral and written communication skills. • Proven ability in writing evaluation reports, and ability to use research findings to evaluate program implementation and operations. • Evaluation Team Leader must be fluent in English, spoken and written. • Candidates speaking French and/or Arabic in addition to English preferred Education Evaluation Expert: Role: The Senior Evaluation Expert will be responsible for supervising the quality of the evaluation design and execution, including qualitative data collection and analysis, identifying and disseminating of findings, lessons learned, and evidence from evaluations. Minimum requirements: • Graduate degree in sociology, anthropology, education or related field with relevant experience in ethnographic research preferred. • Minimum 7 years of experience managing and conducting qualitative and mixed methods evaluations. • Strong background in qualitative research methodologies. • Experience managing or conducting evaluation of education and early grade reading programs (in the Moroccan context preferred) • Experience conducting site visits, key informant interviews, and focus group discussions • Experience in successful dissemination, such as writing evaluation reports, delivering presentations and providing consultation to a diverse range of stakeholders, and effectively tailoring messages to the audience. • Candidates speaking French and/or Arabic in addition to English preferred d) Overall the Evaluator will ensure that: DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 49 • All evaluation team members provide signed statements attesting to a lack of conflict of interest or describing any existing conflict of interest. • The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 201. F.5 TECHNICAL APPROVAL REQUIREMENTS The Contractor is required to seek the following approvals/Technical concurrence from the COR: • Approval of reports described in Section F.4 and F.5. • Technical concurrence on subcontracts, key personnel and other approvals submitted to the CO • Approval of international travel not explicitly identified in the approved annual work-plan • Events, workshops, stakeholder meetings not explicitly identified in the approved annual work-plan [END OF SECTION F] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 50 ANNEX II: DATA COLLECTION INSTRUMENTS AND IRB MATERIALS Focus Group Discussion Guide: Parents Qualitative Instruments: USAID RFS Performance Evaluation Background A. Respondent Type: _____________________________________________________ B. School Name: ________________________________________________________ C. Province: __________________________________________________________ D. Region: ____________________________________________________________ E. Facilitator name: ____________________________________________________ F. Note taker name: ____________________________________________________ G. Date: ______________________________________________________________ H. Start Time: _________________________________________________________ I. Introduce team members and obtain consent: ____________________________ A note about this tool: Purpose: This focus group discussion (FGD) guide will enable you to gather information from parents to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of their perception of RFS activities, project outcomes, relevance and effectiveness of specific activities, inclusion, and lessons learned. Recommended sources: Information should be collected through FGDs with parents at eight RFS pilot school, two RFS scale-up schools, two RFS-IDCRT schools. For the RFS pilot schools, participants should be parents of primary school students in grades 2 to 4, including a cross-section of parents of high-performing and low-performing students. Up to 10 parents should participate in each FGD. FGDs will aim for gender-balanced participation where possible; for example, half of the parents of students in early primary grades participating in FGDs will be fathers/male guardians and half will be mothers/female guardians. Only one parent or guardian is expected per family. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 51 Introduction and Consent [NOTE TO INTERVIEWER: please review and complete the FGD consent form with the participants individually. Once agreement to participate voluntarily is obtained from all participants, please continue. If a participant does NOT agree to take part in the study, thank them for their time, and make a note they did not want to participant. While reviewing and completing the consent form, make sure to also ask for permission to record the interview on our audio-recorder; if all participants say YES please continue; if any participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Only begin audio-recording AFTER consent to participate and be audio-recorded has been received from ALL participants. If consent is not obtained from all participants, do not audio-record the discussion. Indicate in notes if discussion was not audio-recorded. A. Perceptions of the RFS projects 1. Have you heard of the USAID Reading for Success [SSE/NPR/IDCRT] project? If so, what can you tell me about it? [PROMPT: Allow participants to give open-ended responses first, but if they do not know about the program, give a short orientation on the relevant project’s main activities and approach to improving children’s education and reading in Morocco, by noting the following... USAID/Morocco is working with the MOE to support children’s primary education in Morocco. This work to improve children's reading skills takes place in schools, in the community, and with government policy. USAID support is provided through the reading for Success project. This project has different activities, including the National Program for Reading (NPR) activity as well as Small Scale Experimentation (SSE) and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT).] 2. What information, if any, have you received about the new reading instruction approach at your children's school? 3. This discussion will ask about your children who might have benefited from activities provided by the USAID Reading for Success project. These are children who were in grades 1-4 between 2015 and now. Did you have children in grades 1-4 between 2015 and now? What grades were they in? 4. Are you aware of any other education projects in your communities? a. If so, what can you tell me about them? 5. Have you been involved in the USAID Reading for Success [SSE/NPR/IDCRT] project? a. If yes, how so? b. Are you engaged in your PTA? c. Do you read more with your children? Why or why not? 6. Have you seen any change in how well your children are learning to read? Please describe 7. Have you seen any change in your children's attitude toward reading? Please describe DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 52 8. Have you seen any change in your children's attitude toward school? Please describe 9. What are the greatest challenges your child or other children face in learning to read? a. For SSE/NPR schools do you have any specific concerns related to the quality of… 1. Teaching methods and instruction (including training and materials) 2. Student reading materials like textbooks 3. Enrichment programs such as school and community-based extracurricular reading programs for students and family engagement in support of reading b. For IDCRT schools, do you have any specific concerns related to the quality of… 1. Clip and Create software, which helps create and publish Moroccan Sing Language educational materials 2. Training for teachers, schools, directors, parents, members of Deaf associations, and others on operating the software and instructional strategies for early grade reading for students who are deaf or hard of hearing. B. Project Outcomes 10. Do you think the RFS [SSE/NPR/IDCRT] project had any effects on your school's approach to or supports for reading instruction? a. Alternatively: For those of you who have had children in Grades 1 -4 both before and after 2015, has your school's ability to teach children to read changed? b. Why or why not? c. Can you provide specific examples? 11. What has the school, teacher, family members, or community done that has been most effective for helping your child/children learn to read? 12. Have these worked equally well for all children? Why or why not? a. For boys and girls? b. For children whose mother language is not Arabic? c. For children with disabilities? 13. What, if any, changes have you seen in your child's level of interest in reading? Please describe DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 53 C. Relevance/Effectiveness 14. We would like to ask about some specific activities in primary schools. Which of these do you think are/were most important for your child in learning how to read? [If possible, write the list on flipchart paper for easy reference, potentially with images/visual cues for reference.] a. For SSE/NPR schools, these activities include: 1. Improving teaching methods and instruction (including training and materials) 2. Providing student reading materials like textbooks 3. Providing enrichment programs such as school and community-based extracurricular reading programs for students and family engagement in support of reading b. For IDCRT schools, these activities include: 1. Using technology to develop software (called Clip and Create) that helps teachers and parents easily create and publish Moroccan Sing Language￾supported educational materials; 2. Training for teachers, schools, directors, parents, members of Deaf associations, and others on operating the software and instructional strategies for early grade reading for students who are deaf or hard of hearing. D. Inclusion 15. Are there any children in grades 1-4 in your household who you think have not benefited from reading instruction? If yes, please provide additional details and why. a. How about in your Community? 1. Are there certain types of children who continue to struggle to learn to read? 2. Are there certain activities that some students are not eligible for or do not take part? For example, the summer schools programs; If yes, why. [Probe respondents if they do not provide examples: Are there children who are frequently absent from school? Are there children who might not focus on school due to responsibilities at home or working?] 16. What are the barriers to reaching those children who did not benefit/are not benefitting? 17. How could these children be better served in future education and reading activities? E. Lessons Learned 18. Do you have any other recommendations for USAID, the MOE, or schools for helping improving children's reading in the future? [PROMPT: Note that this builds on the items previously discussed. Link this question to build on what participants said previously] 19. How about for improving reading for all children? a. What about... DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 54 1. Children with disabilities; 2. Children whose mother language is not Arabic; 3. Children in rural areas or other geographically marginalized areas; 4. Girls; 5. Any other marginalized or vulnerable children? 20. Were there any unexpected outcomes or results (positive or negative)? F. Closing 21. Those are all of my questions. a. Are there any additional comments that you would like to share with us? b. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? H. Summary and Conclusion Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes POST-FOCUS GROUP NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 55 Focus Group Discussion Guide: Teachers Qualitative Instruments: USAID RFS Performance Evaluation Background A. Respondent Type:_____________________________________________________ B. School Name:________________________________________________________ C. Province: __________________________________________________________ D. Region: ____________________________________________________________ E. Facilitator name: ____________________________________________________ F. Note taker name: ____________________________________________________ G. Date: ______________________________________________________________ H. Start Time: _________________________________________________________ I. Introduce team members and obtain consent: ____________________________ A note about this tool: Purpose: This focus group discussion (FGD) guide will enable you to gather information from teachers to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of RFS project outcomes, project implementation, inclusion, and lessons learned. Recommended sources: Information should be collected through FGDs with teachers at eight RFS pilot school, two RFS scale-up schools, and two RFS-IDCRT schools. For the RFS pilot schools, participants should be Grade 2 to 4 teachers. Up to 10 teachers should participate in each FGD. FGDs will aim for gender-balanced participation where possible; for example, half of the teachers participating in FGDs will be male and half will be female. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 56 Introduction and Consent [NOTE TO INTERVIEWER: please review and complete the FGD consent form with the participants. Once agreement to participate voluntarily is obtained from all participants, please continue. If a participant does NOT agree to take part in the study, thank them for their time, and make a note they did not want to participant. While reviewing and completing the consent form, make sure to also ask for permission to record the interview on our audio-recorder; if all participants say YES please continue; if any participants say NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Only begin audio-recording AFTER consent to participate and be audio-recorded has been received from ALL participants. If consent is not obtained from all participants, do not audio-record the discussion. Indicate in notes if discussion was not audio-recorded. A. Project Outcomes 1. This discussion will ask about your experience with USAID's Reading for Success project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. What do you know about the project? [PROMPT: Allow participants to give open-ended responses first, but if they do not know about the program, give a short orientation on the relevant project’s main activities and approach to improving children’s education and reading in Morocco, by noting the following... USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment to support reading. This work is currently taking place through the National Program for Reading (NPR) activity as well as Small Scale Experimentation (SSE) and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT).] 2. Have you seen differences with your students' ability to read in Arabic during RFS? Can you give specific examples? a. Have you seen different rates of progress for… 1. Decoding (The ability to map letters to sounds) 2. Reading aloud with fluency (The ability to read orally with speed, accuracy, and proper expression) 3. Reading with comprehension (The ability to actively engage with, and derive meaning from, the texts that are read)?? b. Have you seen different rates of progress for… 1. Children with disabilities 2. Girls vs. boys 3. Disadvantaged or marginalized students 4. Less successful students 5. Students who do not speak Arabic at home? 3. Which strategies/interventions have you found most effective for helping your students learn to read? [Probe for specific strategies used in the intervention (phonemic awareness, increased reading time, new reading materials). Then ask if they have any other strategies they want to name.] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 57 4. What are some the challenges you, and teachers like you, face in teaching/improving children's reading skills? [Probe to find examples that might be helpful to supporting them improving their teaching.] 5. What are some the challenges facing your students as they try to learn to read? 6. Are you aware of any of your students participating in the summer activities being lead by NGOs? a. If yes, have you noticed any difference between their performance and those who did not participate in the summer program? B. Project Implementation 7. What type of support have you and your school received from the MOE since 2015 to improve students' reading? [Probe and differentiate for provincial, regional, national levels of MOE.] [Probe for attendance/participation in trainings, should use training list handout for relevant region as reference] 8. If you attended the RFS trainings, what difference did they make to your teaching (if any)? a. Can you give examples of what you can now do, perhaps, that you weren’t able to do previously? 9. Have you received coaching on Arabic reading instruction? If so, what kind? Please describe it. a. Who provides this support and how often? b. Can you comment on the effectiveness (strengths) and challenges (weaknesses) of this approach? 10. Have you utilized or participated in any new Arabic reading assessments? If so, can you describe it? a. Who introduced these assessments? b. Can you comment on the effectiveness (strengths) and challenges (weaknesses) of this approach? 11. Have you received new materials for Arabic reading instruction? If so, what kind? Please describe it. a. Who provided these materials? b. Can you comment on the effectiveness (strengths) and challenges (weaknesses) of this approach? 12. Have your students received new materials for Arabic reading instruction? If so, what kind? Please describe it. a. Who provided these materials? b. Can you comment on the effectiveness (strengths) and challenges (weaknesses) of this approach? 13. [NPR and SSE] RFS introduced a new Arabic language instruction approach and materials that have recently been introduced to grades 1-4. This approach emphasizes phonics, or helping readers understand how letters correspond to sounds and identify common spelling patterns. What are your thoughts and experiences about using this approach in your teaching, versus an approach that prioritizes recitation and comprehension over decoding? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 58 a. Can you describe any successes related to implementing or using the new curriculum and/or materials? (Things that have been particularly relevant or useful) b. Can you describe any challenges you encountered with implementing the new curriculum or using the materials? (Prove, for example: things that were not helpful or relevant; issues such as a lack of or inadequate training; extra demand on teachers/increased workload; or pacing of the lesson plans) 14. For those of you who have been at this school for at least three years, what changes, if any, have you seen in any of the following areas in your school since 2015? a. Amount of time devoted to teaching reading b. Techniques used for teaching reading c. Number of texts in the hands of children d. Teaching children in their mother language (a language they speak and understand) e. Testing children's reading progress [NOTE: Question should be directed at teachers who have been at the school for at least 3 years; If teachers have not been part of the project since 2015, probe about any observed changes since they have engaged in the project, including noting the relevant timeframe in your field notes] [If changes are observed] f. Do you think RFS helped bring about these changes? How? [Probe for and capture successes and challenges if they are raised] C. Inclusion 15. Are there any children in your school who have not benefitted from the USAID Reading for Success [SSE/NPR/IDCRT] project? If yes, please provide additional details and why. a. Are there certain types of children who continue to struggle to learn to read? Please describe b. Are there certain aspects or activities within the project that some students are not eligible for or do not take part? Please describe [Probe respondents if they do not provide examples: Are there children who are frequently absent from school who might not benefit from the project? Are there children who might not focus on school due to responsibilities at home or working? Are there certain students who do not participate in the summer program?] 16. What are the barriers to reaching those children who did not benefit/are not benefitting? [PROMPT: this question should be asked separately after each group that is brought up in previous questions] 17. How could these children be better served in future education and reading activities? [PROMPT: this question should be asked separately after each group that is brought up in previous questions] D. Lessons Learned 18. Do you have any other recommendations for USAID, the MOE, or schools for helping improve children's reading in the future? [PROMPT: Note that this builds on the items previously discussed. Link this question to build on what participants said previously] 19. How about for improving reading for all children? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 59 a. What about... 1. Children with disabilities; 2. Children whose mother language is not Arabic; 3. Children in rural areas or other geographically marginalized areas; 4. Girls; 5. Any other marginalized or vulnerable children? 20. Were there any unexpected outcomes or results (positive or negative)? E. Closing 21. Are there any additional comments that you would like to share with us? 22. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? F. Summary and Conclusion Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 60 POST-FOCUS GROUP NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ ____________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 61 _____________________________________________________ _______________________________________ _______________________________________ _______________________________________ _______________________________________ _____________________________________________________ _____________________________________________________ ______________________________________________________________ __________________________________________________________ ________________________________ _____________________________________________ Key Informant Interview Guide: Implementers Qualitative Instruments: USAID RFS Performance Evaluation A note about this tool: Purpose: This interview guide will enable you to gather information from implementing partner key informants to get an overview of their involvement in the RFS project. The objective is to gain insights and understanding of implementing partners’ work and their engagement with the Ministry of Education, the new Arabic language curricula introduced as part of RFS, inclusion, sustainability, and lessons learned. Recommended sources: Information should be collected through at least three interviews with implementing partners from each RFS activity. At a minimum, evaluators will conduct one KII with the Chief of Party (COP) and Deputy Chief of Party (DCOP), one with technical staff including monitoring and evaluation specialists, and for projects with subcontractors or similar partners, at least one with a subcontractor identified by evaluators together with the Mission and implementing partners. Specific subcontractors include subject matter experts engaged by Management Systems International, Creative Associates International, and Al Akhawayn University under RFS-HICD; two of the six publishers of RFS￾NPR materials; technical staff working on RFS-NPR from University of Oregon, Management Systems International, and Al Akhawayn University; and two of the 10 civil society organizations (CSOs) that have implemented summer enrichment reading activities under RFS-SSE. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. Background A. Respondent type: B. Respondent role/position: C. Respondent Organization: D. Sex: E. Age: F. Interviewer name: G. Note taker name: H. Date: I. Start Time: J. Introduce team members and obtain consent: K. State purpose of evaluation: Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 62 YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. IMPLEMENTATION/ENGAGEMENT WITH MOE 1. Can you please tell me about your role in your organization? [NOTE TO INTERVIEWER: Probe for the following: - Title? - Which department/ division/etc.? - How long have you been in this role for?] a. Were you previously involved in… i. SSE? ii. IDCRT? iii. HICD? b. If yes, how so? 2. What can you please tell me about your organization's role with the USAID/Morocco RFS activity? 3. How would you describe your activity's (i.e., your implementing partner's) relationship with the MOE? 1. Which departments/centers/individuals does your activity engage with? On which activities? How often? 4. [NOTE TO INTERVIEWER: For COPs/Project Leadership] How was the MOE involved in leading the implementation and management of your RFS activity? Can you provide specific examples? 1. What about at the Regional level? Provincial level? School level? 2. Do you feel that MOE's involvement has not been enough, limited, about right, or too much? Can you provide specific examples? 5. Do you feel that your RFS activity is/was effective in helping improve primary school children’s reading skills? Can you provide specific examples about why or why not? [NOTE TO INTERVIEWER: Probe respondent to please describe why and provide examples.] 6. [NOTE TO INTERVIEWER: For SSE, NPR, IDCRT ONLY] Do you feel that your RFS activity is/was effective in increasing community support for reading? In what ways was it effective, and can you provide specific examples about why or why not? 1. What about engaging families to support reading? In what ways was it effective, and can you provide specific examples about why or why not? 2. What about building CSO's capacity to implement reading initiatives? In what ways was it effective, and can you provide specific examples about why or why not? 7. [NOTE TO INTERVIEWER: For SSE, NPR, IDCRT prime implementers ONLY, not applicable for NGOs] Do you feel that your RFS activity is/was effective in improving the MOE's policy environment around reading? Can you provide specific examples about why or why not? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 63 1. Can you provide examples of any laws, policies, regulations or guidelines that have been developed and/or successfully implemented to improved reading at the primary level? 8. What kinds of concerns or obstacles, if any, have you [or your staff] encountered during the implementation of your RFS activity? 1. Do you have any suggestions on how to address these concerns and obstacles? 9. How has your project used M&E data to adjust your activities? 1. Can you provide examples? What about EGRA/EGSLA specifically? 2. What, if any, adjustments to your project implementation came from M&E data? 3. Adjustment to the management of the project? 10. What do you see as the most important and valuable contributions of your RFS activity? 1. What about to improving children's reading in Morocco? 11. Have there been some aspects or components of your RFS activity that fell short of the project objectives? Please describe, with specific examples if possible. 1. What might have helped these aspects or components reach their targets? 12. Are there components of your RFS activity you would change to increase their effectiveness? Please describe, with specific examples if possible. 13. What aspects of your RFS activity do you feel will have the least lasting effects? Please describe, with specific examples if possible. [NOTE TO INTERVIEWER: Probe by key outcomes, including: a. on children's reading skills; b. on teacher's reading instruction; c. on CSO/community/family support and engagement in children's reading; d. and on MOE capacity to implement a reading instruction approach within the Arabic language curriculum] B. CURRICULA [FOR SSE AND NPR PRIME IMPLEMENTERS ONLY] Now I’d like to ask you about the new Arabic language curriculum developed as part of RFS. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 14. As part of the RFS project, a new Arabic language curriculum has recently been introduced to grades 1-4. Can you tell us about your role with these activities and materials? 1. Can you comment on the MOE's level of engagement and leadership on the design, implementation and roll-out of the new reading curriculum and materials? 15. Can you comment on the quality and relevance of the new reading curriculum and training materials? For example, are they appropriate for the learning context of Morocco? 16. Can you describe any successes related to the implementation and roll-out of the new curriculum and materials? Things that have been particularly relevant or useful? 17. Can you describe any challenges that have emerged in implementing the new curriculum or using the materials? Things that were not helpful, relevant, or DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 64 made it difficult to implement and roll-out? 18. Do you think the MOE will be likely to continue using the early grade Arabic curriculum/materials in the future (after the project ends)? 1. If so, in what way? 2. If no, why not? 19. How has/does the MOE support teachers and administrators in implementing the new curricula and materials? 1. Do you feel your RFS activity has built the capacity of the teachers and administers to improve reading instruction at the primary level? Can you provide specific examples about why or why not? 2. Can you comment of the quality and effectiveness of the support being provided to teachers on reading instruction? 3. Can you describe any challenges or factors that have affected the quality of the training or coaching? 20. Has your activity had any successes with getting … [NOTE TO INTERVIEWER: Probe for and capture challenges if they are raised] 1. More time devoted to teaching reading? 2. Better techniques for teaching reading? 3. More texts in the hands of children? 4. Teaching children in their mother language (a language they speak and understand)? 5. Testing children’s reading progress? C. INCLUSION Now I’d like to ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 21. As noted, USAID/Morocco is working with the Ministry of Education to enhance educational development for primary-school aged children (i.e. age 6-12). Based on your experience, which children in Morocco continue to face challenges in accessing and thriving in their education at the primary level? Please describe. [NOTE TO INTERVIEWER: Allow respondent to answer unprompted first, then ask about the following if not already discussed: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls; e. Any other marginalised or vulnerable children?] 22. What are the barriers to reaching those children and supporting their education? 1. What about for children with disabilities specifically? [NOTE TO INTERVIEWER: Probe for deaf and hard of hearing children in particular] 23. How could these children be better supported in their education in the future? D. SUSTAINABILITY Now I’d like to ask you about sustainability. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 65 24. Are the benefits of the RFS activities likely to continue after the NPR activity ends in 2021? 1. If yes, which ones? 2. If no, why not? 25. Does the MOE know yet if it would like to sustain any RFS accomplishments? 1. Which aspects and why? 2. Which individuals or subunits of the MOE are most influential in sustaining RFS accomplishments? 26. What challenges does the MOE face in creating a sustainable reading instruction approach within the Arabic language curriculum? 27. What strengths/capacity exist that will enable/support continuation of NPR activities after NPR closes? Can you provide specific examples? 1. Within the MOE? 2. Within the CSOs? 3. Within the communities? 28. What support is needed to continue to build and sustain the current reading instruction approach within the Arabic language curriculum? E. LESSONS LEARNED The last set of questions I’d like to discuss with you are about lessons learned. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 29. What specific activities do you think USAID can undertake to best improve reading education for children in Morocco in the future? [NOTE TO INTERVIEWER: For each response ask, “Why do you think this activity would be the most effective way for USAID to improve reading education?” 30. How about for improving reading for all children? 1. What about... i. Children with disabilities; ii. Children whose mother language is not Arabic; iii. Children in rural areas or other geographically marginalized areas; iv. Girls; v. Any other marginalized or vulnerable children? 31. Were there any unexpected outcomes or results (positive or negative)? F. CLOSING 32. Those are all of my questions. Are there any additional comments that you would like to share with us? 33. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? G. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 66 to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 67 _____________________________________________________ _______________________________________ _______________________________________ _______________________________________ _______________________________________ ________________________________________________________ ________________________________________________________ _________________________________________________________________ _____________________________________________________________ ____________________________________ _________________________________________________ Key Informant Interview Guide: Ministry of Education Official Qualitative Instruments: USAID RFS Performance Evaluation A note about this tool: Purpose: This interview guide will enable you to gather information from Ministry of Education key informants to get an overview about their involvement in the RFS project. The objective is to gain insights and understanding of their engagement with the RFS activities and implementers, the new Arabic language curricula introduced as part of RFS, inclusion, and sustainability. Recommended sources: Information should be collected through with a host of MoE officials at the provincial, regional, and national levels. The provincial KIIs will be conducted in 10 provinces in six regions, including all eight provinces where RFS-SSE and RFS-NPR experimental schools are located and two provinces where RFS-NPR has scaled up its activities In each of the 10 provinces, two MoE officials will be interviewed—the provincial director and the inspector/provincial coordinator—as well as the Arabic teacher trainer at the province’s teacher trainer center. At the regional level, interviewees will include the MoE regional director in each of the four regions where RFS-SSE and RFS-NPR schools are located as well as two regional directors of teacher training centers—one in Fes-Meknes and one in Rabat-Salé-Kenitra. National-level interviews will be conducted with the directors of five divisions—the Directorate of Strategy, Statistics, and Planning; the Curriculum Directorate; the Directorate of Evaluation; the National Center for Pedagogical Innovations and Experimentation (CNIPE); and the National Center for Evaluation and Examinations (CNEE). If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. Background A. Respondent type: B. Respondent role/position: C. Respondent Ministry or Organization: D. Sex: E. Age: F. Interviewer name: G. Note taker name: H. Date: I. Start Time: J. Introduce team members and obtain consent: K. State purpose of evaluation: Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 68 take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. ROLE IN MOE 1. Can you please tell me about your role in the MOE? a. Title? b. Which directorate/division/etc.? c. How long have you been in this position? B. IMPLEMENTATION/ENGAGEMENT WITH RFS 2. [NOTE TO INTERVIEWER: For MOE regional and province-level officials and coordinators] For MOE regional and province-level officials and coordinators] This discussion will ask about your experience with USAID's Reading for Success project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment to support reading. This currently being done by working with the MOE on the NPR project as well as SSE and HICD in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through a project called IDCRT. What involvement, if any, have you had with any of these four USAID education activities? a. Are you currently involved in the NPR activity? If yes, what do you do? 3. [NOTE TO INTERVIEWER: For MOE central office officials] This discussion will ask about your experience with USAID's Reading for Success project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. Can you share a little about your role during the following phases of the project? a. Preparation, evidence building, and design? b. Experimentation? c. Implementation? d. Monitoring and evaluation? 4. How would you describe your [and your center/directorate's] relationship with USAID/Morocco's education activities and their implementers? You might know them as IDCRT, SSE, NPR, and HICD; or Chemonics (RFS-SSE prime), Creative (RFS-NPR prime), Kaizen Company (HICD prime), and Institute for Disabilities Research and Training (IDCRT prime). [NOTE TO INTERVIEWER: Follow-up on what they said they worked on from the previous question.] a. Were you involved in the design of RFS or any of its associated activities? If yes, can you share a little about your role during the preparation, evidence building, design, DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 69 experimentation, deployment, and monitoring phases of the project, as applicable? 5. Do you feel your [or your Center/directorate] involvement in RFS activities was not enough, limited, about right, or too much? Can you provide specific examples? [NOTE TO INTERVIEWER: If needed, probe for involvement in design, implementation and management of RFS.] a. What about the involvement of others from the Ministry at the National level? Regional level? Provincial level? School level? b. Did USAID/Morocco and the RFS activities seek your [and your staff] input and feedback on their activities and concerns around implementation? If yes, do you think it improved implementation? Why? 6. Do you feel that the RFS activities are/were effective in helping improve primary school children’s readings skills? Can you provide specific examples about why or why not? Please describe which ones, why and provide examples. [NOTE TO INTERVIEWER: Prompt with examples of project inputs.] 7. Do you feel that the RFS activities are/were effective in increasing community support for reading? Can you provide specific examples about why or why not? a. What about engaging families to support reading? b. What about building CSO's capacity to implement reading initiatives? 8. Do you feel that the RFS activities are/were effective in improving the policy environment around reading? a. Can you provide examples of any laws, policies, regulations or guidelines that have been developed and/or successfully implemented to improved reading at the primary level? b. Are you familiar with the RFS-HICD recommendations? If yes, do you feel these recommendations are useful? i. Has the MOE taken any action in response to these recommendations? Can you provide specific examples? 9. What would you say are the biggest successes, if any, you [or your staff] have seen with any of the RFS [or NPR/SSE/IDCRT/HICD] activities? Please describe and provide examples. a. How have these successes affected the MOE? 10. What would you say are the biggest challenges, if any, you [or your staff] have seen with any of the RFS [or NPR/SSE/IDCRT/HICD] activities? Please describe and provide examples. a. How have these challenges affected the MOE? 11. What kinds of concerns or obstacles, if any, have you [or your staff] encountered during the implementation of these RFS [or NPR/SSE/IDCRT/HICD] activities? Please describe and provide examples. a. How have these concerns affected the MOE? b. Do you have any suggestions on how to address these concerns and obstacles in the future? 12. What have been the most important and valuable contributions of the RFS activities to your office/region/department? a. What about to improving children's reading in Morocco? 13. Have there been some RFS activities that have not met your expectations? Please describe and provide examples. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 70 a. How would you suggest these shortcomings be addressed in the future? 14. What would you change about RFS to increase its effectiveness? Please describe. C. CURRICULA [For Curricula Directorate and TTI] Now I’d like to ask you about the new Arabic language curriculum. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 15. As part of the RFS project, a new Arabic language curriculum has recently been introduced to grades 1-4. Could you tell me about the current Arabic reading curriculum being implemented in Morocco? a. What has been your experience with the new reading curriculum teaching and learning materials? b. What role did you [or your department/institute] play in the design, implementation and roll-out of the new reading curriculum and materials? 16. Can you comment on the quality of the new reading curriculum and training materials? Please give examples. For example, are they appropriate for the learning context of Morocco? 17. Can you describe any successes related to the implementation and roll-out of the new curriculum and materials? Things that have been particularly relevant or useful? 18. Can you describe any challenges that have emerged in implementing the new curriculum or using the materials? Things that were not helpful, relevant, or made it difficult to implement and roll-out? 19. Do you think the MOE will be likely to continue using the early grade Arabic curriculum/materials in the future (after the project ends)? a. If so, in what way? b. If not, why not? 20. How has/does the MOE support teachers and administrators in implementing the new curricula and materials? a. Do you feel that the RFS activities has successfully built the capacity of the teachers and administrators to improve reading instruction at the primary level? Can you give some examples? b. Can you comment on the quality and effectiveness of the support being provided to teachers on reading instruction? c. Can you describe any challenges or factors that have impacted the quality of the training or coaching? 21. Has the RFS project had any successes with getting... [NOTE TO INTERVIEWER: Probe for and capture challenges if they are raised] a. More time devoted to teaching reading? b. Better techniques for teaching reading? c. More texts in the hands of children? d. Teaching children in their mother language (a language they speak and understand)? e. Testing children’s reading progress? D. INCLUSION DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 71 Now I’d like to ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 22. As we've mentioned, USAID/Morocco is working with the Ministry of Education to enhance educational development for primary-school aged children (i.e. age 6-12). Based on your experience, what factors help children thrive in their education at the primary level? Can you provide specific examples? [NOTE TO INTERVIEWER: [Examples of factors might include materials, quality teaching, school environment, encouragement and support from home and community, etc.] 23. Based on your experience, which students in Morocco face challenges in thriving in their education at the primary level? Please describe. [NOTE TO INTERVIEWER: allow respondent to answer unprompted first, then ask about the following if not already discussed: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls; and e. Any other marginalised or vulnerable children] 24. What are the barriers to ensuring those children get a quality primary education? a. What about for children with disabilities specifically? 25. How could these children be better supported in their education in the future? E. SUSTAINABILITY Now I’d like to ask you about sustainability. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 26. To what extent, if any, has the project had an impact on education in Morocco, especially in Arabic reading? Please describe and provide examples. a. Are there any aspects of the RFS project you feel will have a lasting impact? If yes, which aspects and why? b. Are there any specific aspects of the RFS project you feel will not have a lasting impact? If yes, which aspects and why? [NOTE TO INTERVIEWER: Probe by key areas, including: i. on children's reading skills; ii. on teacher's reading instruction; iii. on CSO/community/family support and engagement in children's reading; and iv. on MOE capacity to implement a reading instruction approach within the Arabic language curriculum] 27. What strengths exist that will enable continuation of NPR activities after NPR closes? Can you provide specific examples? a. Within the MOE at national, provincial, regional, and local levels? b. Within the CSOs? c. Within the communities? 28. What is your understanding of the current state of progress toward creating a reading instruction approach within the Arabic language curriculum in Morocco? 29. What challenges do you [or does your department/MOE] face in creating a sustainable reading instruction approach within the Arabic language curriculum? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 72 30. What support is needed to continue to build and sustain the current a reading instruction approach within the Arabic language curriculum? 31. What aspects of your RFS activity do you feel will have the most lasting effects? Please describe, with specific examples if possible. 32. What aspects of your RFS activity do you feel will have the least lasting effects? Please describe, with specific examples if possible. [NOTE TO INTERVIEWER: Probe by key outcomes, including: a. on children's reading skills; b. on teacher's reading instruction; c. on CSO/community/family support and engagement in children's reading; and d. on MOE capacity to implement a reading instruction approach within the Arabic language curriculum] 33. Do you have any other recommendations for USAID, the MOE, or schools for helping improve children's reading in the future? [NOTE TO INTERVIEWER: Note that this builds in the item above. Link this question to build on what participants said previously.] 34. How about for improving reading for all children? a. What about children with disabilities? b. What about children whose mother language is not Arabic? c. What about children in rural areas or other geographically marginalized areas? d. What about girls? e. What about any other marginalized or vulnerable children? 35. Were there any unexpected outcomes or results (positive or negative)? F. CLOSING 36. Those are all of my questions. Are there any additional comments that you would like to share with us? 37. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? G. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 73 interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 74 _____________________________________________________ _______________________________________ ___________________________________ _______________________________________ _______________________________________ _______________________________________________ _______________________________________________ ________________________________________________________ ____________________________________________________ ___________________________ ________________________________________ Key Informant Interview Guide: Non-Ministry of Education Official Qualitative Instruments: USAID RFS Performance Evaluation A note about this tool: Purpose: This interview guide will enable you to gather information from non-Ministry of Education key informants to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of their engagement with RFS, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with various non￾Ministry of Education officials, including one representative from the Ministry of Solidarity and one representative from the Ministry of Health, two representatives from disabled persons organizations, and one education specialist from Parliament. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. Background A. Respondent type: B. Respondent role/position: C. Respondent Ministry or Organization: D. Sex: E. Age: F. Interviewer name: G. Note taker name: H. Date: I. Start Time: J. Introduce team members and obtain consent: K. State purpose of evaluation: [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 75 A. ROLE IN ORGANIZATION AND RELATIONSHIP WITH MOE 1. Can you please tell me about your role in this ministry [or organization] ? a. Title? b. Which directorate or division? c. How long have you been in this role for? B. ENGAGEMENT WITH RFS 2. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading and improving the policy environment to support reading. This is currently being done by working with the MOE on the NPR activity as well as SSE and HICD in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called IDCRT. Have you had any involvement with any of these four USAID education activities? If yes, how so? 3. How would you describe your and your ministry's [or organization’s] relationship with USAID education activities and their implementers. You might know them as IDCRT, SSE, NPR, and HICD; or Chemonics (RFS-SSE prime), Creative (RFS-NPR prime), Kaizen Company (HICD prime) ,and Institute for Disabilities Research and Training (IDCRT prime). C. INCLUSION Now I’d like to ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 4. As we've mentioned, USAID/Morocco is working with the Ministry of Education to enhance educational development for primary-school aged children (i.e. age 6-12). Based on your experience, which students in Morocco face challenges in thriving in their education at the primary level? [NOTE TO INTERVIEWER: allow respondent to answer unprompted first, but if they have difficulty answering you may prompt them to ask about the following: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls e. Children who are orphans; f. Children in poor households; g. Children who are migrants (e.g., nomads, refugees); h. Children engaged in child labor; i. Children in areas experiencing insecurity] 5. What are the barriers to reaching those children and supporting their education? D. LESSONS LEARNED Now I’d like to ask you about lessons learned. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 76 6. What type of support currently exists for these children in their education, such as policies, strategies, organizations or activities? [NOTE TO INTERVIEWER: Probe for all groups of children mentioned by respondent above.] a. Which of these have been particularly effective? Why? b. Which of these have been ineffective? Why haven’t they been successful? 7. How could these children be better supported in their education in the future? 8. Do you have any other recommendations for USAID, the MOE, or schools for helping improve children's reading in the future? 9. How about for improving reading for all children? a. What about children with disabilities? b. What about children whose mother language is not Arabic? c. What about children in rural areas or other geographically marginalized areas? d. What about girls? e. What about any other marginalised or vulnerable children? 10. Were there any unexpected outcomes or results (positive or negative)? E. CLOSING 11. Those are all of my questions. Are there any additional comments that you would like to share with us? 12. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? F. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 77 POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 78 _____________________________________________________ _______________________________________ _______________________________________ _______________________________________ ________________________________________________________ ____________________________________________________ _____________________________________________________________ _________________________________________________________ __________________________________ Key Informant Interview Guide: School Director Qualitative Instruments: USAID RFS Performance Evaluation A note about this tool: Purpose: This interview guide will enable you to gather information from school director key informants to get an overview about their involvement with the RFS project. The objective is to gain insights and understanding of general perceptions of quality of education, social outcomes, RFS project implementation, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with school directors who are actively working on early grade reading and/or inclusive education. Twelve school directors will be interviewed overall, including one apiece in each of the eight provinces where RFS-NPR pilot schools are located, two from RFS scale-up schools, and two with former directors of RFS-IDCRT schools. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. Introduction A. Respondent type: B. School Name: C. Province: D. Region: E. Facilitator name: F. Note taker name: G. Date: H. Start Time: I. Introduction of team member and consent: Introduction and Consent [NOTE TO INTERVIEWER: Please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. GENERAL PERCEPTIONS OF QUALITY OF EDUCATION DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 79 1. This discussion will ask about your experience with USAID's Reading for Success project. This project focuses on improving children's reading who were in grades 1-4 between 2015 and now. As you may know, USAID/Morocco is working with the MOE to support children’s education at the primary level in Morocco. This includes improving children's reading skills, community support for reading, and improving the policy environment to support reading. . This work is currently taking place through the National Program for Reading (NPR) activity as well as Small Scale Experimentation (SSE) and Human and Institutional Capacity Development (HICD) in the past. In addition, USAID also previously supported work with children who are deaf and hard of hearing through an activity called Improving Deaf Children's Reading through Technology (IDCRT).] What involvement, if any, have you had with any of these four USAID education activities? Please describe 2. Do you see any difference in students' reading in your school today compared with before 2014-2015? [PROMPT: ask for explanations for answers including concrete examples] a. What was RFS doing during this time that may have contributed to this difference? 3. Do you see any difference in families and community members' support for their children's reading today compared to before 2014-2015? [PROMPT: ask for explanations for answers including concrete examples] a. What was RFS doing during this time that may have contributed to this difference? 4. Are you aware of any students from your school participating in the summer activities being lead by NGOs? a. If yes, have you noticed any difference between their performance and those who did not participate in the summer program? B. SOCIAL OUTCOMES 5. Is there a difference in the quality of teachers' reading instruction between when the program started and now? a. If yes, how? b. What was RFS doing during this time that may have contributed to this difference in teachers' reading instruction? c. What could the project have done to better support teachers' reading instruction? 6. Which strategies/interventions have you found most effective for helping teachers improve their reading instruction? [Probe for specific strategies used in the intervention (phonemic awareness, increased reading time, new reading materials). Then ask if they have any other strategies they want to name.] a. Why? Please describe. 7. What are some of the challenges teachers in your school face in teaching reading? [Probe to find examples that might be helpful in supporting them improve their teaching.] a. Did RFS provide support to address any of these challenges? Explain. C. PROJECT IMPLEMENTATION DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 80 8. Since, 2015, what type of support have you and your school received from the MOE to improve students' reading? [Probe and differentiate between provincial, regional, and national levels of MOE.] [Probe on types of support: teacher training, materials, curriculum advising, policy.] 9. Have you, or any of the teachers in your school, participated in any of the RFS/early grade reading trainings? a. If yes, please describe which training, when you participated and the role you played (e.g., received training, served as trainer). [Use training list handout for relevant region as reference] [Probe for details and specific examples: Did you provide instructional leadership, observation, or coaching to teachers as part of your involvement in these activities?] b. If yes, could you comment on what difference they made in your school's reading instruction, if any? c. Can you give examples of what teachers can now do, perhaps, that they weren’t able to do previously? 10. What type of coaching or materials have you or teachers in your school received to support effective reading instruction, if any? [Probe to be sure director considers training, coaching, materials to support teaching, materials for students, and assessment activities.] a. If yes, who provides this support and how often? b. Can you comment on the effectiveness (strengths) and challenges (weaknesses) of these approaches? 11. [NPR and SSE] RFS introduced a new Arabic language instruction approach and materials that have recently been introduced to grades 1-4. What are your thoughts and experiences about teachers using this approach in your school, versus the whole language method? a. Can you describe any successes related to the new curriculum and/or materials? (Things that have been particularly relevant or useful) b. Can you describe any challenges encountered with implementing the new curriculum or using the materials? (Things that were not helpful or relevant) 12. What changes, if any, have you seen in any of the following areas in your school since 2015? [NOTE: Question should be directed at school director if they have been at the school for at least 3 years] a. Amount of time devoted to teaching reading b. Techniques used for teaching reading c. Number of texts in the hands of children d. Teaching children in their mother language (a language they speak and understand e. Testing children's reading progress [If directors have not been part of the project since 2015, probe about any observed changes since they have engaged in the project, including noting the relevant timeframe in your field notes] [Probe for and capture successes and challenges if they are raised] D. INCLUSION 13. Are there any children in your school who have not benefitted from the USAID Reading for Success [SSE/NPR/IDCRT] project? If yes, please provide additional details and why. a. Are there certain types of children who continue to struggle to learn to read? Please describe DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 81 b. Are there certain aspects or activities within the project that some students are not eligible for or do not take part? Please describe [Probe respondents if they do not provide examples: Are there children who are frequently absent from school who might not benefit from the project? Are there children who might not focus on school due to responsibilities at home or working? Are there certain students who do not participate in the summer program?] 14. What are the barriers to reaching those children who did not benefit/are not benefitting? [PROMPT: this question should be asked separately after each group that is brought up in previous questions] 15. How could these children be better served in future education and reading activities? [PROMPT: this question should be asked separately after each group that is brought up in previous questions] E. LESSONS LEARNED 16. Do you have any other recommendations for USAID, the MOE, or schools for helping improve children's reading in the future? [PROMPT: Note that this builds on the items previously discussed. Link this question to build on what participants said previously] a. What additional training, if any, should be provided to school directors? 17. How about for improving education for all children? a. What about... 1. Children with disabilities; 2. Children whose mother language is not Arabic; 3. Children in rural areas or other geographically marginalized areas; 4. Girls; 5. Any other marginalized or vulnerable children? 18. Were there any unexpected outcomes or results (positive or negative)? F. CLOSING 19. Are there any additional comments that you would like to share with us? 20. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? G. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes POST-KII NOTES: Please comment on DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 82 • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 83 ___________________________________________________ _______________________________________ _______________________________________ _______________________________________ __________________________________________________ _________________________________________________ __________________________________________________________ _______________________________________________________ ______________________________ Key Informant Interview Guide: USAID Qualitative Instruments: USAID RFS Performance Evaluation A note about this tool: Purpose: This interview guide will enable you to gather information from USAID key informants to get an overview of USAID’s involvement in the RFS project. The objective is to gain insights and understanding of USAID’s role with RFS, project implementation and outcomes, engagement with the Ministry of Education and project sustainability, inclusion, and lessons learned. Recommended sources: Information should be collected through interviews with members of the USAID/Morocco Education Office—the Director, the Contracting Officer’s Representative for the activities, and the Education Specialist. If more than one official is interested and available to participate at the designated date, time and location, the KII may be administered as a group interview but the potential impact of power dynamics (such as age, gender, and level of authority in the organization or ministry) within the interview must be taken into consideration and mitigated. Background A. Respondent type: B. Respondent role/position: C. Sex: D. Age: E. Interviewer name: F. Note taker name: G. Date: H. Start Time: I. Introduce team members and obtain consent: Introduction and Consent [NOTE TO INTERVIEWER: please review and complete the KII Consent form with the participant. If agreement to participate voluntarily is obtained, please continue. If participant does NOT agree to take part in the study, thank them for their time, make a note they did not want to participant. Make sure to also ask for permission to record the interview on our audio-recorder; if participant says YES please continue; if participant says NO, acknowledge that you will not record the conversation and proceed without turning on the audio-recording.] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 84 **IMPORTANT NOTE**: Begin audio-recording AFTER consent to participate and be audio-recorded has been received. A. ROLE WITH RFS 1. Can you please tell me about your role working with RFS? [NOTE TO INTERVIEWER: Probe the following: a. Are you currently involved in NPR? If yes, how so? b. Were you previously involved in SSE? c. Were you previously involved in HICD? d. Were you previously involved in IDCRT, which worked with children who are deaf and hard of hearing?] B. OUTCOMES 2. Do you feel that the RFS activities are/were effective in helping improve primary school children’s readings skills? In what ways? Please describe why and provide examples. 3. Do you feel that the RFS activities are/were effective in increasing community support for reading? In what ways? For all responses, please describe why and provide examples. a. What about engaging families to support reading? b. What about building CSO's capacity to implement reading initiatives? 4. Do you feel that the RFS activities are/were effective in improving the policy environment around reading? 1. Can you provide specific examples about why or why not? 2. Can you provide examples of any laws, policies, regulations or guidelines that have been developed and/or successfully implemented to improved reading at the primary level? 5. Do you feel the HICD recommendations were useful for the MOE? 1. Has the MOE taken any action in response to these recommendations? Can you provide specific examples? C. IMPLEMENTATION Now I’d like to ask you about RFS implementation. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 6. What have been the most important and valuable contributions of the RFS activities to improving children's reading in Morocco? 1. Why do you consider these the most important or valuable? 7. What factors helped facilitate achievement of these accomplishments? 1. How did they help? 2. Were there any other factors that helped achieve the RFS project objectives? 8. Have there been some aspects of the RFS activities that have not met your expectations? Please describe and provide examples. 1. How would you suggest these shortcoming be addressed? DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 85 9. What kinds of concerns or obstacles, if any, have impeded implementation of the RFS [or NPR/SSE/IDCRT/HICD] activities? 1. Why did these obstacles impede implementation? 10. Thinking specifically about the project's objectives, what kinds of obstacles limited achievement of the objectives? 1. Why? 2. Do you have any suggestions on how to address these obstacles? D. ENGAGEMENT WITH MINISTRY/SUSTAINABILITY I’d also like to ask you about USAID’s engagement with the Ministry of Education and sustainability of RFS. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 11. How did USAID/Morocco and the RFS implementers seek engagement from the Ministry? a. How effective was this engagement? b. What do you think made engagement effective (or ineffective)? 12. Do you feel that the Ministry's involvement has been not enough, limited, about right, or too much? Can you provide specific examples? [NOTE TO INTERVIEWER: Probe for differences between Ministry bodies at the national, provincial, and regional levels] 13. Can you comment on the MOE's level of engagement and leadership on the design, implementation and roll-out of the new reading curriculum and materials under RFS-NPR? 14. Are the outcomes of the RFS activities likely to continue after the NPR activity ends in 2022? 1. If yes, which ones? Why? 2. If no, why not? 15. What aspects of the RFS project do you feel will have the most lasting effects? Please describe, with specific examples if possible. [NOTE TO INTERVIEWER: Probe by key areas, including: a. on children's reading skills b. on teacher's reading instruction c. on CSO/community/family support and engagement in children's reading d. on MOE capacity to implement national reading] [NOTE TO INTERVIEWER: If not already addressed by respondent, ask specifically: Do you think the MOE will be likely to continue using the early grade Arabic curriculum/materials in the future (after the project ends)? Why or why not?] 16. What strengths or capacity exist that will support continuation of RFS activities after RFS closes? Can you provide specific examples? 1. Within the MOE? 2. Within the CSOs? 3. Within the communities? 17. What challenges do you think the MOE faces in creating a sustainable national reading system? [NOTE TO INTERVIEWER: Allow respondent to reply unprompted first. If not discussed, probe for: DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 86 a. Speed of RFS-NPR scale up and national rollout b. Teacher coaching c. Time available for teacher training d. Breadth of support for RFS-NPR across multiple MOE subunits e. MOE support to education for students with disabilities f. Anything else that has arisen earlier in the interview] 18. What additional support could USAID provide that would best help the MOE or other stakeholders continue to build and sustain the current national reading system? E. INCLUSION I’d also like you ask you about inclusion. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 19. Based on your experience, what factors help children thrive in their education at the primary level? Can you provide specific examples? [NOTE TO INTERVIEWER: Examples of factors might include materials, quality teaching, school environment, encouragement and support from home and community, etc.] 20. Which students in Morocco face challenges in meeting expectations in their education at the primary level? [NOTE TO INTERVIEWER: Allow respondent to answer unprompted first, but if they have difficulty answering you may prompt them to ask about the following: a. Children with disabilities; b. Children whose mother language is not Arabic; c. Children in rural areas or other geographically marginalized areas; d. Girls; e. Children who are orphans; f. Children in poor households; g. Children who are migrants (e.g., nomads, refugees); h. Children engaged in child labor; i. Children in areas experiencing insecurity] 21. What are the barriers to reaching those children and supporting their education? 1. What about for children with disabilities specifically? 22. How could these children be better supported in their education in the future? F. LESSONS LEARNED The last set of questions I’d like to discuss with you are about lessons learned. [NOTE TO INTERVIEWER: As needed, define the topic and/or population being discussed.] 23. Is there anything you would have done differently with RFS to increase its effectiveness? Please describe, with specific examples if possible. 24. What specific activities do you think USAID can undertake to best improve reading education for children in Morocco in the future? [NOTE TO INTERVIEWER: For each response, ask: "Why do you think this activity would be the most effective way for USAID to improve reading education?"] DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 87 25. How about for improving reading for all children? 1. What about... vi. Children with disabilities; vii. Children whose mother language is not Arabic; viii. Children in rural areas or other geographically marginalized areas; ix. Girls; x. Any other marginalised or vulnerable children? 26. Were there any unexpected outcomes or results (positive or negative)? G. CLOSING 27. Are there any additional comments that you would like to share with us? 28. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? H. SUMMARY AND CONCLUSION Thank you for your time. Your help in this research is very important. As I mentioned, the results of the report will be used by USAID Morocco and the MOE to understand how the RFS project activities are going and are being implemented. The final results of our research project will be published in a report in the coming months. These results will be communicated back to the USAID and the ministry. End Time: ___________________________ Total length: ________Hours _________Minutes DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 88 POST-KII NOTES: Please comment on • Any factors that may have affected the truthfulness of the responses given and the willingness of the interview subject to participate, • If more than one respondent participated, the different perspectives that emerged through disagreements in the interviews, • Any additional insights or comments that should be noted. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 89 ANNEX III: ESTIMATED LOE AND STAFFING PLAN NORC provides operational management, technical oversight, and analytical support to the evaluation, with STS leading the evaluation design, field data collection, and analysis and reporting. The evaluation team includes experts from both NORC and STS. To ensure maximum efficiency and fluidity of communication and information exchange, NORC and STS include French-speaking staff on the evaluation team and will recruit several local research staff with fluency in Moroccan Arabic and French. The evaluation team relies on a Principal Investigator/Education Evaluation Expert from STS, supported by two senior evaluators (Evaluation Experts) from NORC and STS, to lead the evaluation design, including sampling, data collection instrument development, data analysis, and report-writing. An Evaluation Team Lead from STS leads the overall management of the study by coordinating between all actors involved and ensuring all evaluation activities are progressing according to the timeline. The evaluation team also includes a Qualitative Evaluation Expert, who advises during the evaluation design and data analysis as well as support report writing on qualitative research and gender issues. Finally, a Research Analyst and six researcher coordinators support the Team Lead and Education Evaluation Experts in conducting fieldwork, coding and cleaning qualitative data, quantitative and qualitative data analysis, and report-writing. A Project Coordinator from STS, with support from the local research coordinators, oversees all logistics related to the project. Field work will be conducted by three teams of three researchers, ensuring that each team consists of French and Arabic speakers. Estimate level of effort is provided in the following table: R&A CONTRACT LABOR CATEGORIES NAME LOE (HOURS) Project Manager Project Manager 80 Sr. Evaluation Specialist/PI Senior Technical Advisor 64 Field Coordinator Evaluation Expert 284 Research Assistant/Data Research Analyst 720 Research Assistant/Writing & Analysis Desktop Publisher 16 Research Assistant/Writing & Analysis Budgeting Assistant 16 Research Assistant/Data Financial Analyst 8 Sr. Evaluation Specialist/PI Evaluation Team Leader 496 Sr. Evaluation Specialist/PI Education Evaluation Expert 544 Research Assistant/Writing & Analysis Evaluator 288 Research Assistant/Writing & Analysis Qualitative Evaluation Expert 744 Research Assistant/Data Program Coordinator 816 SHORT-TERM LOCAL STAFF NAME LOE (HOURS) Sr. Research Coordinator (1) TBD 464 Research Coordinator (2) TBD 240 Research Coordinator (3) TBD 240 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 90 ANNEX IV: ROLES AND RESPONSIBILITIES MATRIX EVALUATION PHASE SPECIFIC TASKS RESPONSIBLE NOTES NORC STS USAID Preparation Communications with USAID team X NORC will be the primary POC for USAID and will manage communications. STS may engage in direct conversations with USAID for technical planning purposes. Site Selection (Sampling) X X STS will work with USAID to identify regions, provinces, and schools to be included in the sample based on agreed upon selection criteria to ensure variability. USAID will facilitate identification, recruitment and introduction of participants from the MoE to be invited to participate in the FGDs or KIIs (for both field test and operational data collection). Organization of logistics/ procurements for workshops and data collection X STS will organize all local logistics for data collector trainings and data collections. Development of data collection tools X STS will lead the development of data collection tools in collaboration with NORC, who will provide review and feedback. Hiring of local staff and data collectors X STS will hire data collectors based in Morocco. NORC may provide guidance on data collector qualifications as needed. Preparation of data entry and management templates X STS will provide templates for all data collectors to use for data entry and management – including Interviewer/facilitator Guide Notes, FGD and KII Note￾Taker Forms, Expand Field Notes form, Focus Group/KII Debriefing Form, Qual Data Tracking Form, and Archival Information Sheets (where relevant) Training Training of Data collectors X STS will lead training for qualitative data collectors. Data Collection Creation/implementation of field plan for data collection – including teaming X STS will create a field plan for data collection – including teaming assignment – and distribute to NORC for review and feedback DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 91 EVALUATION PHASE SPECIFIC TASKS RESPONSIBLE NOTES NORC STS USAID Organization of field work-related logistics for data collection teams X STS will plan and organize all logistics for operational data collection. Submission (via upload) of typed Expanded Field Noted and Debriefing Notes in English word documents at the end of each day of data collection X STS-hired data collectors will type up and upload the relevant notes each evening on shared, password protected server. They will also record audio from each interview and focus group, which will serve as reference for producing expanded field notes. Data monitoring and management during field work X STS will be responsible for tracking and collecting qualitative data46 from data collectors and documenting and/or resolving any issues. STS will provide preliminary review of expanded field notes and debrief notes during data collection. Review and final submission of qualitative data X STS will review all qualitative data, obtain any clarifications or additional information needed, as well as make updates or revisions where relevant in order to finalize the qualitative data Data Cleaning/ Analysis Compile raw data files X STS will compile all qualitative data files and prepare documents for importing into NVivo Conduct all data analysis X STS will conduct all qualitative data analysis. Reporting Draft report X STS will draft the evaluation report using the USAID￾provided template. Submit final report X STS will revise the baseline report based on feedback from NORC and USAID. Validation and dissemination Validation workshop X STS will facilitate an in-country validation workshop after generating initial findings. Final results presentation X STS will lead a remote presentation of final results following completion of the evaluation report. 46 This may include audio-recordings, interview/FGD facilitator guide notes, debriefing note, handwritten field notes, expanded field notes (typed in English). DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 92 ANNEX V: TIMELINE WITH DELIVERABLES AND MILESTONES TASK / ACTIVITY LEAD START END DELIVERABLE EVALUATION DESIGN Draft workshop agenda submitted to USAID NORC 4/22/2019 4/22/2019 X Draft evaluation design report outline submitted to USAID NORC 4/22/2019 4/22/2019 X Stakeholder mapping exercise STS 4/22/2019 4/22/2019 X Agenda, desk review, and outline revised/approved by USAID USAID 4/23/2019 4/30/2019 Design workshop planning and coordination STS 4/23/2019 5/3/2019 In-briefing/meeting with USAID USAID/STS 5/6/2019 5/6/2019 Evaluation design workshop STS 5/7/2019 5/9/2019 Out-briefing/meeting with USAID USAID/STS 5/10/2019 5/10/2019 Evaluation design report and work plan drafted and submitted to USAID NORC 5/13/2019 5/31/2019 X Evaluation design report and work plan revised/approved by USAID USAID 6/3/2019 6/21/2019 Data collection tools in English drafted and submitted to USAID for review NORC/STS 6/14/2019 6/26/2019 X Draft tools revised/approved by USAID in preparation for pre-testing USAID 6/27/2019 7/12/2019 Tools and final design submitted to IRB for review and approval NORC 7/15/2019 8/5/2019 Tools translated to Arabic and submitted to USAID for review STS 7/15/2019 7/23/2019 X USAID review and approval of Arabic tools USAID 7/24/2019 8/2/2019 Pre-test (non-school-based) by local research coordinator of data collection tools in Rabat and environs STS 8/5/2019 8/9/2019 Pre-test memo and revised tools drafted and submitted to USAID NORC/STS 8/12/2019 8/23/2019 X USAID review/approval of pre-test memo and tool revisions USAID 8/26/2019 8/30/2019 DATA COLLECTION AND FIELD WORK Field work planning and coordination STS/NORC 6/28/2019 9/6/2019 Engage with MoE to obtain MoE approval of school-based and government stakeholder data collection USAID 6/28/2019 8/12/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 93 TASK / ACTIVITY LEAD START END DELIVERABLE In-briefing with USAID STS/NORC 9/11/2019 9/11/2019 In-person training and data collection practice, including pre-test of school-based data collection STS/NORC 9/12/2019 9/13/2019 Conduct qualitative field work STS/NORC 9/16/2019 10/3/2019 School-based data collection STS/NORC 9/16/2019 10/3/2019 Out-briefing with USAID STS/NORC 10/4/2019 10/4/2019 Data Analysis and Reporting Data cleaning and transcription STS/NORC 9/23/2019 10/11/2019 Qualitative data coding and analysis STS/NORC 10/11/2019 11/1/2019 Quantitative data analysis STS/NORC 10/11/2019 11/1/2019 Evaluation report drafted and submitted to USAID STS/NORC 10/18/2019 11/8/2019 X USAID provides written feedback on evaluation report STS/NORC 11/11/2019 11/22/2019 Prepare for in-country results presentation STS/NORC 10/28/2019 12/3/2019 In-briefing with USAID STS/NORC 12/4/2019 12/4/2019 In-country results validation presentation and infographic STS/NORC 12/5/2019 12/5/2019 X Out-briefing with USAID STS/NORC 12/6/2019 12/6/2019 Evaluation report revised following validation workshop STS/NORC 12/9/2019 12/20/2019 Evaluation report revised/approved by USAID USAID 12/23/2019 1/8/2020 Executive summary translated into French and Arabic STS 1/8/2020 1/15/2020 Translated executive summaries revised/approved by USAID USAID 1/15/2020 1/22/2020 RESULTS DISSEMINATION Remote dissemination presentation of results STS 1/28/2020 1/28/2020 TASK ORDER COMPLETION Final call with USAID NORC 1/27/2020 1/31/2020 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 94 ANNEX VI: DETAILED DRAFT FIELDWORK SCHEDULE TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 1 XXXXXXX SSE Remote KII English By 9/16/2019 1 Technical staff SSE Remote KII English By 9/16/2019 1 XXXXXX IDCRT Remote KII English By 9/16/2019 1 XXXXXX HICD Remote KII English By 9/16/2019 1 HICD subcontract staff Management Systems International Remote KII English By 9/16/2019 1 HICD subcontract staff Creative Associates International Remote KII English By 9/16/2019 1 XXXXXX MoE Curriculum Directorate Rabat KII French 9/16/2019 1 XXXXXX MoE Directorate of Evaluation Rabat KII French 9/16/2019 1 XXXXXX MoE Directorate of Strategy, Statistics, and Planning Rabat KII French 9/17/2019 1 XXXXX CNIPE Rabat KII French 9/17/2019 1 XXXXX Directorate of Cooperation and Teaching in Private Schools Rabat KII French 9/17/2019 1 XXXXXX CNEE Rabat KII French 9/18/2019 1 XXXXX Communication Directorate Rabat KII French 9/18/2019 1 XXXXX Ministry of Health Rabat KII French 9/19/2019 1 XXXXXX Ministry of Solidarity, Women, Family and Social Development: Division of social affairs Rabat KII French 9/19/2019 1 XXXXXX CMREF Rabat KII French 9/20/2019 1 XXXXXXXX NPR Rabat KII English 9/20/2019 1 TBD Deaf Association Steering Committee Sale KII Arabic 9/23/2019 1 TBD Other disabled persons organization Rabat KII Arabic 9/23/2019 1 Technical staff NPR Rabat KII English 9/24/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 95 TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 1 Publisher 1 Publishers Rabat KII French 9/24/2019 1 Publisher 2 Publishers Rabat KII French 9/25/2019 1 Technical staff SSE Rabat KII French 9/25/2019 1 XXXXXXXXXX Civil society organization Rabat KII French 9/26/2019 1 XXXXXXXXXX IDCRT Rabat KII French 9/26/2019 1 XXXXXXXXXX IDCRT Casablanca KII French 9/27/2019 1 TBD Al Akhawayn University Al Akhawayn University in Ifrane KII English 9/30/2019 1 XXXXXXXXXX MoE - Provincial El Hajeb, Fes￾Meknes KII Arabic 9/30/2019 1 XXXXXXXXXXX XXXXX MoE - Provincial El Hajeb, Fes￾Meknes KII Arabic 9/30/2019 1 Arabic teacher trainers (provincial) Curriculum department El Hajeb, Fes￾Meknes KII Arabic 9/30/2019 1 XXXXXX HICD Rabat KII French 10/1/2019 1 Subject matter expert 1 HICD Rabat KII French 10/1/2019 1 Subject matter expert 2 HICD Rabat KII English 10/1/2019 1 Education specialist USAID Rabat KII English 10/2/2019 1 XXXXXXXXXXX XXXXX USAID Rabat KII English 10/2/2019 1 XXXX USAID Rabat KII English 10/2/2019 2 XXXXXXXXX RFS Experimental School 1 Temara-skihart, Rabat KII Arabic 9/16/2019 2 Teachers RFS Experimental School 1 Temara-skihart, Rabat FGD Arabic 9/16/2019 2 Parents RFS Experimental School 1 Temara-skihart, Rabat FGD Arabic 9/16/2019 2 XXXXXXXX Fondation Lalla Asma pour Enfants Sourds Rabat KII Arabic, with sign language interpretation 9/17/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 96 TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 2 Teachers Fondation Lalla Asma pour Enfants Sourds Rabat FGD Arabic, with sign language interpretation 9/17/2019 2 Parents Fondation Lalla Asma pour Enfants Sourds Rabat FGD Arabic, with sign language interpretation 9/17/2019 2 Travel day 9/18/2019 2 XXXXXXXX Al Fath Association School for the Deaf Meknes KII Arabic, with sign language interpretation 9/19/2019 2 Teachers Al Fath Association School for the Deaf Meknes FGD Arabic, with sign language interpretation 9/19/2019 2 Parents Al Fath Association School for the Deaf Meknes FGD Arabic, with sign language interpretation 9/19/2019 2 XXXXXXX RFS Experimental School 2 El Hajeb, Fes￾Meknes KII Arabic 9/20/2019 2 Teachers RFS Experimental School 2 El Hajeb, Fes￾Meknes FGD Arabic 9/20/2019 2 Parents RFS Experimental School 2 El Hajeb, Fes￾Meknes FGD Arabic 9/20/2019 2 Travel weekend 9/21-22/2019 2 XXXXXX Regional AREF Souss Massa KII Arabic 9/23/2019 2 XXXXXXXXX MoE - Provincial Inezgane-Aït￾Melloul, Souss Massa KII Arabic 9/23/2019 2 XXXXXXXXXXX XXXXXXX MoE - Provincial Inezgane-Aït￾Melloul, Souss Massa KII Arabic 9/23/2019 2 Arabic teacher trainers (provincial) Curriculum department Inezgane-Aït￾Melloul, Souss Massa KII Arabic 9/23/2019 2 XXXXXXXX RFS Experimental School 3 Inezgane-Aït￾Melloul, Souss Massa KII Arabic 9/24/2019 2 Teachers RFS Experimental School 3 Inezgane-Aït￾Melloul, Souss Massa FGD Arabic 9/24/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 97 TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 2 Parents RFS Experimental School 3 Inezgane-Aït￾Melloul, Souss Massa FGD Arabic 9/24/2019 2 XXXXXXXXX MoE - Provincial Tiznit, Souss Massa KII Arabic 9/25/2019 2 XXXXXXXXXXX XXXXXX MoE - Provincial Tiznit, Souss Massa KII Arabic 9/25/2019 2 Arabic teacher trainers (provincial) Curriculum department Tiznit, Souss Massa KII Arabic 9/25/2019 2 XXXXXXXXX RFS Experimental School 4 Tiznit, Souss Massa KII Arabic 9/26/2019 2 Teachers RFS Experimental School 4 Tiznit, Souss Massa FGD Arabic 9/26/2019 2 Parents RFS Experimental School 4 Tiznit, Souss Massa FGD Arabic 9/26/2019 2 Travel weekend 9/28-29/2019 2 XXXXXX Regional AREF Marrakesh-Safi KII Arabic 9/30/2019 2 XXXXXXXXX MoE - Provincial Marrakesh-Safi KII Arabic 10/1/2019 2 XXXXXXXXXXX XXXXXX MoE - Provincial Marrakesh-Safi KII Arabic 10/1/2019 2 Arabic teacher trainers (provincial) Curriculum department Marrakesh-Safi KII Arabic 10/1/2019 2 XXXXXXXX RFS Scale-up School 1 Marrakesh-Safi KII Arabic 10/2/2019 2 Teachers RFS Scale-up School 1 Marrakesh-Safi FGD Arabic 10/2/2019 3 XXXXXXXX RFS Experimental School 5 School TBD, Kenitra, Rabat KII Arabic 9/16/2019 3 Teachers RFS Experimental School 5 School TBD, Kenitra, Rabat FGD Arabic 9/16/2019 3 Parents RFS Experimental School 5 School TBD, Kenitra, Rabat FGD Arabic 9/16/2019 3 XXXXX Regional AREF Tangier, Tanger￾Tetouan-Al Hoceima KII Arabic 9/17/2019 3 XXXXXXXXXX MoE - Provincial Tangier, Tanger￾Tetouan-Al Hoceima KII Arabic 9/17/2019 3 XXXXXXXXXXX XXXXXX MoE - Provincial Tangier, Tanger￾Tetouan-Al Hoceima KII Arabic 9/17/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 98 TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 3 Arabic teacher trainers (provincial) Curriculum department Tangier, Tanger￾Tetouan-Al Hoceima KII Arabic 9/17/2019 3 Travel day 9/18/2019 3 XXXXXXXXX RFS Experimental School 6 Taounate, Fes￾Meknes KII Arabic 9/19/2019 3 Teachers RFS Experimental School 6 Taounate, Fes￾Meknes FGD Arabic 9/19/2019 3 Parents RFS Experimental School 6 Taounate, Fes￾Meknes FGD Arabic 9/19/2019 3 XXXXXXXXXXX XXXXXXX MoE - Provincial Taounate, Fes￾Meknes KII Arabic 9/20/2019 3 XXXXXXXXXXX XXXXXXXX MoE - Provincial Taounate, Fes￾Meknes KII Arabic 9/20/2019 3 Arabic teacher trainers (provincial) Curriculum department Taounate, Fes￾Meknes KII Arabic 9/20/2019 3 XXXXX Regional AREF Fes-Meknes KII Arabic 9/20/2019 3 XXXXX CMREF Fes-Meknes KII Arabic 9/20/2019 3 Travel weekend 9/21-22/2019 3 XXXXXX Regional AREF Oriental KII Arabic 9/23/2019 3 XXXXXXXXXXX XXXXXXXXX MoE - Provincial Oujda-Angad, Oriental KII Arabic 9/23/2019 3 XXXXXXXXXXX XXXXXXX MoE - Provincial Oujda-Angad, Oriental KII Arabic 9/23/2019 3 Arabic teacher trainers (provincial) Curriculum department Oujda-Angad, Oriental KII Arabic 9/23/2019 3 XXXXXXXXX RFS Experimental School 7 Oujda-Angad, Oriental KII Arabic 9/24/2019 3 Teachers RFS Experimental School 7 Oujda-Angad, Oriental FGD Arabic 9/24/2019 3 Parents RFS Experimental School 7 Oujda-Angad, Oriental FGD Arabic 9/24/2019 3 XXXXXXXXXXX XXXXXXXXX MoE - Provincial Figuig, Oriental KII Arabic 9/25/2019 3 XXXXXXXXX MoE - Provincial Figuig, Oriental KII Arabic 9/25/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 99 TEAM RESPONDENT ORGANIZATION LOCATION TYPE LANGUAGE DATE 3 Arabic teacher trainers (provincial) Curriculum department Figuig, Oriental KII Arabic 9/25/2019 3 XXXXXXXX RFS Experimental School 8 Figuig, Oriental KII Arabic 9/26/2019 3 Teachers RFS Experimental School 8 Figuig, Oriental FGD Arabic 9/26/2019 3 Parents RFS Experimental School 8 Figuig, Oriental FGD Arabic 9/26/2019 3 Travel weekend 9/28-29/2019 3 XXXXX XXXXXXXX Drâa-Tafilalet KII Arabic 9/30/2019 3 XXXXXXXXX XXXXXXXX Drâa-Tafilalet KII Arabic 9/30/2019 3 XXXXXXXXXXX XXXX XXXXXXXXX Drâa-Tafilalet KII Arabic 9/30/2019 3 Arabic teacher trainers (provincial) Curriculum department Drâa-Tafilalet KII Arabic 9/30/2019 3 XXXXXX XXXXXXXXXX Drâa-Tafilalet KII Arabic 10/2/2019 3 Teachers RFS Scale-up School 2 Drâa-Tafilalet FGD Arabic 10/2/2019 3 XXXX XXXXXXXX Béni Mellal￾Khénifra KII Arabic 10/3/2019 3 XXXXXXXXX XXXXXXXXX Béni Mellal￾Khénifra KII Arabic 10/3/2019 3 XXXXXXXXXXX XXXXXXX XXXXXXXXX Béni Mellal￾Khénifra KII Arabic 10/3/2019 3 Arabic teacher trainers (provincial) Curriculum department Béni Mellal￾Khénifra KII Arabic 10/3/2019 DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 100 ANNEX VII: DISCLOSURE OF REAL OR POTENTIAL CONFLICT OF INTEREST FOR USAID EVALUATIONS Memorandum Regarding Organizational Conflict of Interest for School-to-School International (STS) NORC and School-to-School International (STS) do not foresee an Organizational Conflict of Interest (OIC) resulting from STS’s role as a subcontractor on Reading for Success – Small Scale Experimentation (RFS-SSE) and Reading for Success – Improving Deaf Children’s Reading Through Technology (RFS-IDCRT) as outlined by the terms in Contract Information Bulletin 99-17 (CIB). According to the CIB, high potential for OIC occurs in the following scenarios:47 1. Where a firm that designs a USAID activity under contract with USAID wants to be eligible for the competition to implement the activity; 2. Where a firm that evaluates an activity or contractor under contract with USAID wants to provide services that are requested as a result of the evaluation; and 3. Where a firm that audits USAID contractors under contract with USAID seeks to do consulting work under contract with USAID (sometimes in competition with the firms audited). None of the above-mentioned scenarios outlined in the CIB apply to STS’s subcontractor role on RFS￾SSE or RFS-IDCRT. In both instances, STS acted strictly in an evaluation role having no role in project implementation. STS’s role on RFS-SSE included conducting assessments of student learning using the Early Grade Reading Assessment (EGRA) tool and measures changes in teacher and school administrator behaviors using the Snapshot of School Management Effectiveness (SSME) methodology. As part of this work, STS designed data collection tools, facilitated enumerator training, conducted student assessments and teacher surveys, and completed data analysis. On RFS-IDCRT, STS provided technical assistance to grantee IDRT on program design and theory of change, monitoring, evaluation, and learning and fidelity of implementation approaches, and research design. STS also worked collaboratively with Institute for Disabilities Research and Training, Inc. (IDRT) and local stakeholders on the development, piloting, administration, results analysis and interpretation of one of the first ever Early Grade Reading Sign Language Assessments in Moroccan Sign Language, used to measure the MSA and MSL skills of G1 and G2 students who are deaf or hard of hearing. Though NORC and STS do not foresee any potential for impaired objectivity stemming from STS’s evaluation role on past projects, staffing will be responsive to this concern with NORC staff being included to ensure objectivity in data collection and analysis. 47 https://www.usaid.gov/sites/default/files/documents/1868/cib9917.pdf DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 101 ANNEX VIII: EVALUATION DESIGN WORKSHOP AGENDA READING FOR SUCCESS PERFORMANCE EVALUATION DESIGN WORKSHOP Tuesday, May 7, 2019 Ministry of Education Conference Center Takaddoum, Rabat TIME TOPICS LEAD 09h30–10h15 Welcome and introductions USAID 10h15–11h30 Overview of the RFS performance evaluation, including objectives and desk review findings STS 11H30–11H45 BREAK 11h45–12h30 Present draft evaluation questions STS 12h30–13h30 Stakeholder mapping, including identifying appropriate data sources and methods for answering the evaluation questions STS 13H30–14H00 BREAK 14h00–14h30 Identify data sources and methods (continued) STS 14h30–15h30 Review evaluation workplan and timeline for data collection visits STS 15h30 End of workshop STS DRG-LER II – CONTRACT No. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV RFS PERFORMANCE EVALUATION DESIGN REPORT | 102 ANNEX IX: EVALUATION DESIGN WORKSHOP PARTICIPANTS # NAME TITLE ORGANIZATION 1 Mariam Britel-Swift Senior Education Specialist USAID 2 Amina El Abdellaoui Education Specialist USAID 3 Fouad Chafiqi Director Directorate of Curricula 4 Anouar Boukili Principal Administrator responsible for education of students with disabilities Directorate of Curricula 5 Ahmed Chaibi Director National Center for Evaluation and Exam (CNEE) 6 Abdallah El Shariat Director Regional Teacher Training Center (CRMEF) in Rabat 7 Abdallah Taïe Director Regional Teacher Training Center (CRMEF) in Fes-Meknes 8 Lamia Faris Trainer Regional Teacher Training Center (CRMEF) in Souss Massa 9 Zahya Filay Trainer Regional Teacher Training Center (CRMEF) in Oriental 10 Rachid El Amrani Head of the Languages and Arts Teaching Division Directorate of Curricula 11 Kristina Solum Evaluation Team Leader STS 12 Matthew Murray Evaluator STS 13 Selene Rangel Program Coordinator STS 14 Zainab Amaiktan Interpreter STS (consultant) 15 Badr Laadam Interpreter STS (consultant)