VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY 2 Final Report OCTOBER 2019 This publication was produced for review by the United States Agency for International Development. It was prepared by Eurosis as part of a subcontract with Management Systems International (MSI), a Tetra Tech Company, for the Mozambique Monitoring and Evaluation Mechanism and Services. VAMOS LER! DLAS 2 Final Report Contracted under Contract No. AID-656-C-17-00002 Mozambique Monitoring and Evaluation Mechanism and Services LIABILITY DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States government. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | i CONTENTS ACRONYMS ........................................................................................................... III EXECUTIVE SUMMARY ........................................................................................ 1 Introduction ............................................................................................................................................................... 1 Study Purpose and Questions ............................................................................................................................... 1 Methodology and Sample Area.............................................................................................................................. 1 Conclusions ................................................................................................................................................................ 2 Recommendations .................................................................................................................................................... 4 INTRODUCTION .................................................................................................... 7 VL! Program Summary............................................................................................................................................. 7 Study Purpose, Audience, and Questions ........................................................................................................... 8 1. METHODOLOGY ........................................................................................... 10 Data Collection ....................................................................................................................................................... 10 Study and Sample Area .......................................................................................................................................... 10 Data Collection and Analysis Methods .............................................................................................................. 13 Research Limitations .............................................................................................................................................. 14 2. FINDINGS ........................................................................................................ 15 Learning Question 1............................................................................................................................................... 15 Learning Question 2............................................................................................................................................... 16 Learning Question 3............................................................................................................................................... 19 Learning Question 4............................................................................................................................................... 24 3. CONCLUSIONS .............................................................................................. 27 Learning Question 1............................................................................................................................................... 27 Learning Question 2............................................................................................................................................... 27 Learning Question 3............................................................................................................................................... 29 Learning Question 4............................................................................................................................................... 30 4. RECOMMENDATIONS .................................................................................. 31 Desired Training Outcomes................................................................................................................................. 31 Gender-Sensitive Practices and Inclusion ......................................................................................................... 31 Training Model Strengths and Weaknesses...................................................................................................... 32 Differences Between Provinces ........................................................................................................................... 32 ANNEX I: TERMS OF REFERENCE ................................................................... 33 Statement of Work ................................................................................................................................................ 33 ANNEX II: ANALYZED DOCUMENTS ............................................................. 44 ANNEX III: PERSONS CONTACTED ............................................................... 46 ANNEX IV: ANALYTICAL FRAMEWORK DATA, SOURCES, AND METHODS .............................................................................................................. 52 ANNEX V: CRITERIA FOR DISTRICT, ZIP, AND SCHOOL SELECTION . 54 Criteria for District Selection .............................................................................................................................. 54 Criteria for ZIP Selection ..................................................................................................................................... 54 Criteria for School Selection ............................................................................................................................... 54 ii | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX VI: WORK PLAN AND DETAILED SCHEDULE .............................. 55 ANNEX VII: DETAILED FIELDWORK PLAN .................................................. 56 ANNEX VIII: SCHOOL CHARACTERISTICS .................................................. 57 ANNEX IX: FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ... 61 Question 1: …………………………………………………………………………………………. 61 Question 2: ............................................................................................................................................................. 63 Question 3: ............................................................................................................................................................. 71 Question 4: ............................................................................................................................................................. 75 ANNEX X: INTERVIEW SCRIPTS AND FOCUS GROUP DISCUSSIONS . 77 National/Central Level .......................................................................................................................................... 77 Provincial Level ........................................................................................................................................................ 80 District Level............................................................................................................................................................ 84 ANNEX XI: OBSERVATION FORMS ................................................................ 89 Observation Form for Training of Trainers ..................................................................................................... 89 Observation Form for Training of Teachers .................................................................................................... 93 Observation Form for L1 Reading/Writing Class ........................................................................................... 99 Table of Tables Table 1. Schools, cohorts, and grades observed by province, district, and ZIP ............................................... 11 Table 2. Number of people contacted by sector, province, and gender ........................................................... 12 Table 3. Research techniques and primary information sources ......................................................................... 13 Table of Figures Figure 1. Map with sample districts ............................................................................................................................. 11 Figure 2. Examples of school conditions .................................................................................................................... 26 USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | iii ACRONYMS CAC Creative Associates Consortium CATI Computer-Assisted Telephone Interview DSP Deputy School Principal SP School Principal FGD Focus Group Discussion DLAS Deep Learning and Adaptive Study DNEP National Directorate of Primary Education DNFP National Directorate of Teacher Training DPEDH Provincial Directorate of Education and Human Development BE Bilingual Education DO District Officer EGRA Early Grade Reading Assessment FCR Findings, Conclusions, and Recommendations ToT Training of Trainers TTDL Teacher Training at District Level GRM Government of the Republic of Mozambique TTI Teacher Training Institute EGR Early Grade Reading LI First Language (mother tongue) L2 Second Language LPB Lesson Plan Book MINEDH Ministry of Education and Human Development MMEMS Mozambique Monitoring and Evaluation Mechanism and Services MSI Management Systems International PW Pedagogical Workshop RAP Administration and Planning Office REG General Education Office IR Intermediate Result SDEJT District Education, Youth, and Technology Service ToR Terms of Reference USAID United States Agency for International Development VL! Vamos Ler! ZIP Zone of Pedagogical Influence USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 1 EXECUTIVE SUMMARY Introduction Vamos Ler! (VL!), the five-year (2016–2020) flagship program of the United States Agency for International Development (USAID) Education Office in Mozambique, implemented by Creative Associates Consortium (CAC) in partnership with the Ministry of Education and Human Development (MINEDH), aims to improve primary (grades 1–3) students’ reading fluency and comprehension in the local languages in Nampula and Zambézia Provinces. VL! also aims to help MINEDH lay the foundations for strengthening the bilingual education (BE) system.1 Study Purpose and Questions VL! is monitored and evaluated by USAID Mozambique Monitoring and Evaluation Mechanism and Services (MMEMS), which is implemented by Management Systems International (MSI) in Mozambique. As part of this monitoring and evaluation, MMEMS conducted a second deep learning and adaptive study (DLAS 2) on progress toward VL! intermediate result (IR) 1—improvement in early grade reading (EGR) classroom instruction—focusing on the effectiveness of VL! teacher training for grade 1 and grade 2 teachers. This report presents DLAS 2 results and provides recommendations to improve VL! teacher training performance. To determine the teacher training model’s effectiveness, this study addressed the following questions: 1. What were the desired outcomes of VL! capacity-building activities2 (training of trainers [ToT], teacher training at district level [TTDL], and pedagogical workshops [PWs]) for acquisition of EGR pedagogical skills, knowledge, and practices? 2. Are teachers implementing gender-responsive instruction practices that support BE EGR, in accordance with the desired outcomes established in the training model? 3. What are the cascade training model’s strengths and weaknesses as it is being applied? 4. How do training outcomes vary by province and what factors can explain differences in capacity￾building experience in Nampula and Zambézia? Methodology and Sample Area DLAS 2 was conducted between June and September 2019 by a team of five senior researchers and three assistants, hired by Eurosis, with three fieldwork cycles (June, July, and August). The study used qualitative methods to collect and analyze data and information on defined learning issues. In addition to document review, the team observed BE literacy classes, teacher training sessions, teacher training in 1 In this report, “bilingual teaching” and “bilingual education” are used interchangeably. 2 In this report, the term “capacity building” includes not only the training (course) but also other activities that increase capacity, such as exchange of experience, clarification of doubts, and technical meetings. The term “training” designates provincial- and district-level courses. 2 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV provincial capitals and districts, and PWs. The team also conducted focus group discussions and semistructured interviews with education sector managers, implementers, and authorities. Data were collected in seven districts of the two provinces where VL! is being implemented (Nampula and Zambézia). In these districts, eight schools were selected, corresponding to eight pedagogical influence zones (ZIPs). MMEMS and USAID chose the districts and ZIPs directly, and MMEMS approved the research team’s proposed school selections, which aimed to capture diverse socioeconomic, cultural, and linguistic realities. In each school, the team observed native language (L1) literacy classes given by cohort 1 grades 1 and 2 teachers and cohort II grade 1 teachers, for a total of 34 class observations and 51 class hours. The team also observed four PW sessions, as well as two ToT courses and two TTDL courses. The team contacted or consulted 101 people at different levels (37 in Nampula, 58 in Zambézia, and 6 in Maputo), of which 23 were women. The vast majority of respondents and observers were men, reflecting the fact that in the countryside most teachers, school principals, and education officials are men. Conclusions The study findings show that the VL! cascade teacher training model is enabling teachers to teach L1 literacy. By 2019, more than 5,000 teachers in 21 districts of Nampula and Zambézia were teaching grade 1 and grade 2 BE classes as a result of training activities carried out by MINEDH with VL! support. This level of expansion was made possible by the cascade training model, which has some limitations, however. Desired Training Outcomes Specific aspects of BE philosophy and pedagogy are at the center of the cascade teacher training model, but VL! does not neglect broader pedagogical skills in which teachers may have insufficient capacity. VL! also provides training in participatory teaching methods, assessment methods and processes, gender equality and inclusion, and other practices. Gender-Sensitive Practices and Inclusion The observed practices correspond to those established in the training model, although quality varies from teacher to teacher. However, demonstration of technical teaching skills does not ensure student learning. First, it does not guarantee that what was observed during the study was everyday practice: the teachers and the school boards were aware that they were being observed and this awareness may have influenced teaching dynamics. Second, teacher training alone is not sufficient to ensure the desired learning outcomes, as many other factors influence the learning process—namely, school management, class conditions, student attendance and delays, and physical school conditions. The planned BE program weekly learning content is generally followed. However, weekly plan activities are not always fully or correctly implemented. The teaching content and dynamics are ambitious for class conditions (space, infrastructure, student number and punctuality, etc.). These conditions constrain program implementation despite efforts by teachers and school boards to overcome some challenges. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 3 Teachers have some knowledge of gender and inclusion issues and have been given general guidelines to promote them. However, these guidelines focus on promoting equal class participation of boys and girls and distributing students appropriately throughout the classroom. Other aspects of gender equality are not addressed. The teachers have limited ability to put gender equality and inclusion processes into practice. The low proportion of female teachers also counteracts efforts toward gender equality. BE training and monitoring still pay little attention to operationalizing the gender approach (what to do and how to do it). Almost all teachers showed attention to integrating children with more difficulties into class dynamics. In the observed classes, no students were marginalized based on their characteristics. Class time and size constrain teachers’ efforts to support students individually, however, and teachers do not use interstudent help enough to address some learning difficulties. All respondents observed that classroom interaction was much better in BE classes than in monolingual3 classes. The teachers successfully and systematically used teaching methods that promote student participation and achieve a favorable learning environment. Classroom interaction is centered more on interaction with teachers than on interaction among students. Most of the observed teachers have enough BE skills, knowledge, and practices and know how to do their work. There are, however, important limitations, notably in lesson planning, phonetics, vocabulary, and numeracy teaching approaches in L1. Proficiency in L1 varies among teachers; the fact that writing in L1 is new to most teachers influences the teaching process. Teachers demonstrated good mastery and use of some methods, but other methods (peer reading, silent reading, tale and retelling, group work, role play) were less widely used. It is not possible to determine whether this is due to lack of mastery or to other reasons. No association was found between level of mastery and belonging to cohort 1 or II, although cohort II teachers were more comfortable in managing learning processes. The student assessment process—the daily and weekly assessment—does not appear to be standardized and it is unclear how it should be done, possibly due to changes introduced during program implementation. VL! has feedback mechanisms to adjust teacher training in accordance with what is being learned in BE implementation. Some respondents reported improvements in the 2019 training process, particularly in trainer selection. Implementation of feedback processes was below expectations, however, particularly for PWs. Training Model Strengths and Weaknesses The cascade teacher training model is familiar within the Mozambican education system. Its implementation has gradually and cost-effectively trained more than 5,000 teachers in a short time and with sufficient quality for BE implementation. VL! gave access to expertise that would not otherwise be readily available because of limited local experience with large-scale BE implementation. 3 Although no lessons have been observed in the monolingual modality, all teachers and other consulted sources categorically expressed this view. 4 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV The model, however, has some weaknesses that hamper its effectiveness. First, the model assumes that teachers have a reasonable command of L1, which is not the case for L1 writing and reading. Also, the PWs, teacher supervision, and local follow-up—fundamental for outcome quality—are not yet having the desired effects. There is also a widespread perception that the current teacher training time is not sufficient for BE requirements, especially in the May and August sessions. The limited time of training sessions has led to insufficient practice in some areas, such as L1 writing, phonetics, and gender equality. The teachers and various respondents are aware of teachers’ language and pedagogical shortcomings, however, to which the current model gives a limited response. TTIs have participated in provincial- and district-level training and in some oversight, which enhances the program’s technical sustainability. TTI participation in PW supervision and support is, however, below expectations, due reportedly to lack of funds. Training quality generally meets stakeholder expectations. Except for one district, there have been no significant criticisms of provincial- and district-level training quality, and there is a general awareness of time, human, and financial limitations. Subsidies for ToT and TTDL trainees are insufficient. Differences Between Provinces There were no significant differences in the way the cascade training model was implemented in the two provinces, either in the activities carried out or in the inputs for program implementation. There were also no noticeable differences in school or teacher characteristics. There were more teachers with pedagogical deficiencies in Nampula than in Zambézia, but it is not possible to generalize these observations due to the small sample size. The sample size did not enable identification of factors associated with the few differences in training results found between the two provinces. Recommendations Desired Training outcomes  R.1.1. Make the desired outcomes for PW and in-school follow-up activities more explicit and measurable.  R.1.2. Formally include the PW and in-school follow-up activities as components of the cascade teacher training model gender-sensitive instructional practices  R.2.1. Ensure that the lesson plans (activity volume and type) consider class conditions, allowing teachers to apply them with the expected quality.  R.2.2. Guide teachers on what to do in situations when the weekly plan cannot be fully met to maintain the integrity of the learning process (letter, syllable, word, etc.).  R.2.3. Make the quarterly plan manuals available before the beginning of each quarter so that teachers can plan their classes more accurately.  R.2.4. Strengthen teacher training in elaborating/adapting detailed lesson plans. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 5  R.2.5. Strengthen the program’s emphasis on gender issues by addressing them systematically in all capacities down to school level and introducing adequate teacher materials and tools.  R.2.6. Place greater emphasis in teacher training on having the strongest students help the weaker students to promote solidarity and enable teachers to focus on the students with the most difficulty.  R.2.7. Ensure that the program implementation monitoring system assesses gender-sensitive practices and inclusion.  R.2.8. Continue to use participatory methods to promote student interest and achieve better learning outcomes.  R.2.9. Reinforce the use of program methods that promote student cooperation.  R.2.10. Strengthen teachers’ capacity building on identified weak elements (lesson planning; phonetics; writing; numeracy; vocabulary; and methodologies, including pedagogical remediation methodologies).  R.2.11. Increase teacher training time by revisiting the approach of excluding teachers from the previous year’s cohort from the training package for a given year.  R.2.12. Strengthen teacher training in L2 teaching methodologies.  R.2.13. Clarify the student daily and weekly assessment system and process to standardize its implementation.  R.2.14. Reinforce the training model’s feedback processes to ensure that BE implementation informs the design and content of subsequent teacher training. Training Model Strengths and Weaknesses  R.3.1. Reflect on the implementation of capacity-building activities in the first two years, produce a document explaining the cascade training model, and clarify the model for all stakeholders.  R.3.2. Strengthen PWs’ role in teacher training, supervision, and follow-up and training model feedback.  R.3.3. Improve PW documentation to better capture difficulties in pedagogical areas and LI.  R.3.4. Revisit logistical support at various levels to better respond to training and trainee needs and sustainability of results.  R.3.5. Maintain external experts’ involvement in providing training technical support throughout the program to improve the quality of local human resources and create sustainable skills.  R.3.6. Revisit TTIs’ role in supervising and monitoring BE to improve the quality of follow-up.  R.3.7. Maintain the cascade training model but devote more time to teacher training and reinforce PWs (alignment between their agenda and the process of monitoring teachers’ difficulties and detailed lesson planning), ZIP pedagogical support to schools, and follow-up within schools.  R.3.8. Revisit logistical support to ensure PW teacher participation. 6 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Differences between Provinces  R.4.1. Compare this study’s findings with the EGRA learning outcomes and further analyze any factors associated with significant differences. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 7 INTRODUCTION Vamos Ler! (VL!), the five-year (2016–2020) flagship program of the United States Agency for International Development (USAID) Education Office in Mozambique, implemented by Creative Associates Consortium (CAC) in partnership with the Ministry of Education and Human Development (MINEDH), aims to improve primary (grades 1–3) students’ reading fluency and comprehension in the local languages in Nampula and Zambézia Provinces. VL! also aims to help MINEDH lay the foundations for strengthening the bilingual education (BE) system.4 VL! is monitored and evaluated by USAID Mozambique Monitoring and Evaluation Mechanism and Services (MMEMS), which is implemented by Management Systems International (MSI) in Mozambique. As part of this monitoring and evaluation, MMEMS conducted a first deep learning and adaptive study (DLAS 1) on progress toward VL! intermediate result (IR) 2—improvement in national early grade reading (EGR) policies and delivery systems. DLAS 2 assesses progress toward IR 1—improvement in EGR classroom instruction—focusing on the effectiveness of VL! teacher training for grade 1 and grade 2 teachers. The study was conducted between June and September 2019 by a team of researchers from Eurosis Consultoria e Formação em Gestão and a researcher hired directly by MMEMS. Three fieldwork cycles were carried out in the two provinces where VL! is implemented and in Maputo. This report presents DLAS 2 results and provides recommendations to improve VL! teacher training performance. VL! Program Summary VL!’s overall objective is to support MINEDH’s efforts to improve early primary grade reading and writing skills using the BE approach. VL! aims to achieve its objectives through three main IRs:  IR 1: EGR classroom instruction improved.  IR 2: National EGR policy framework and delivery systems improved.  IR 3: Parental and community engagement in EGR increased. IR 1 is made up of four sub-IRs in the VL! 2018–2019 work plan: 1. Evidence-based, gender-responsive EGR materials in Mozambican languages used. Activities: a. Development of materials for grade 3. b. Printing of materials (reading, management, and mobilization) for teacher training cohorts 1, II, and III. c. Distribution of the above materials. d. Elaboration and development of appropriate grade 2 and grade 3 stories in low-cost local adult language literature. 2. Teacher use of evidence-based, gender-responsive EGR instructional practices increased. Activities: a. Training of trainers (ToT) for BE. 4 In this report, “bilingual teaching” and “bilingual education” are used interchangeably. 8 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV b. District-level training of zone of pedagogical influence (ZIP) teachers, coaches, and facilitators. c. Implementation and monitoring of teacher training at ZIP level (through pedagogical workshops [PWs]). 3. School management and governance in support of EGR strengthened. Activities: a. Training of school principals. b. Review of training materials for school councils. c. Strengthening of school councils. d. Capacity building for data management and reporting. 4. Coaching and supervision of EGR instruction improved. Activities: a. Class follow-up by school-based companions. b. Follow-up visits by ZIP coordinators. c. District monitoring and support. VL! aims to reach 2,800 public primary schools in Nampula and Zambézia Provinces by 2020 in a process that began in 2018 with 914 schools and added 1,036 schools in 2019. These schools—spread across 8 Zambézia districts and 13 Nampula districts—are managed by the government within the framework of public administration, with staff and mode of operation in accordance with the rules of the Government of the Republic of Mozambique (GRM). Expansion proceeds by ZIP, a form of school organization established by MINEDH several decades ago: VL! covers all schools within a ZIP and then adds more ZIPs from the same district each year. In 2019, about 5,400 teachers, 27 percent of which were women, were involved in VL! BE. VL! works with three Mozambican languages—Echuwabo and Elómwe (Zambézia) and Emakhuwa (Nampula)—and with Portuguese (in both provinces) and focuses on literacy, with some attention to numeracy. Study Purpose, Audience, and Questions DLAS 2 aims to determine the effectiveness of the VL! cascade training model for teacher performance in the classroom.5 This model comprises activities at four levels: (1) provincial-level ToT; (2) district￾level ToT (TTDL) for teachers, school-level coaches, and ZIP facilitators/counselors; (3) literacy PWs; and (4) follow-up of classes at school.6 ToT and TTDL activities take place three times a year in January, May, and August, taking advantage of the school holidays. PWs and school-level coaching are conducted monthly. This study’s audience and users are USAID/Mozambique (program financier), CAC (contracted program implementer), the GRM education sector at national and provincial levels (MINEDH and the Provincial Directorate of Education and Human Development [DPEDH]), and EGR working group members. 5 The study terms of reference are presented in Annex 1. 6 In this report, levels 3 and 4 (ZIP and schools), which appear together in VL! documents, are separated to distinguish the issues that arise in each. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 9 To determine the teacher training model’s effectiveness, the study undertook the following:  Determine the desired teacher training outcomes, particularly at the training sessions (I, II, III) held in January, May, and August 2019, related to teachers´ acquisition of new skills, knowledge, and practices that support EGR in grades 1 and 2.  Document the teacher training approach used in session III at provincial and district levels (August 2019) and at PWs 5 and 6 (June and July 2019), looking at the merits and demerits of standardized implementation to generate evidence and lessons for EGR programming.  Compare and contrast teacher training results from cohorts 1 and II in Nampula and Zambézia.  Describe the process of teacher coaching, mentoring, and supervision at the school level.  Document teachers’ classroom practices against the desired teacher training outcomes.  Document classroom arrangements and learning environment, including student activities and student interaction with teachers. DLAS 2 answers the following learning questions:  Learning Question I: What were the desired outcomes of VL! capacity-building activities (ToT, TTDL, and PWs) for acquisition of EGR pedagogical skills, knowledge, and practices?  Learning Question 2: Are teachers implementing gender-responsive instructional practices that support BE EGR, in accordance with the desired outcomes established in the training model?  Learning Question 3: What are the cascade training model’s strengths and weaknesses as it is being applied?  Learning Question 4: How do training outcomes vary by province and what factors can explain differences in capacity-building experience in Nampula and Zambézia? 10 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 1. METHODOLOGY DLAS 2 is a qualitative study designed to quickly collect and analyze specific program elements to improve them in the short term. Data Collection DLAS 2 used qualitative methodologies to collect and analyze data and information on defined learning issues. These methodologies are appropriate to the study objectives—to assess the effectiveness of the VL! cascade teacher training model used in 2018 and 2019 for cohorts 1 and II and to make viable recommendations to improve training. The study began with a desk review to understand the context and activities undertaken and to inform the design of data collection and analysis tools. Data were collected between June and August 2019 from teachers; school principals (SPs) and deputy school principals (DSPs); directors, department heads, and technicians linked to BE training, implementation, and monitoring in Nampula and Zambézia and in selected districts; and MINEDH central divisions. VL! technicians and specialists were also consulted at all levels of program implementation. (See Annex VI for a detailed work plan and activity timetable.) Study and Sample Area Data were collected in seven districts—four from Zambézia and three from Nampula (Figure 1). In Zambézia, two districts (Mocuba and Namacurra) used the Echuwabo language and two (Alto Molócue and Gilé) used Elomwe; in Nampula, the three districts (Eráti, Mogovolas, and Moma) used Emakhuwa. The district, ZIP, and school selection criteria were set jointly by the research team, MMEMS, and USAID (see criteria in Annex V). MMEMS and USAID chose the districts and ZIPs, and MMEMS approved the research team’s proposed school selections. The selections sought to capture a diverse socioeconomic, cultural, and linguistic reality (rural/urban; interior/coastal, district size) as well as a diverse range of cohort 1 and cohort II schools. Eight ZIPs were chosen in these districts. In each ZIP, a school was selected for observation of classroom teacher performance. The team observed native language (L1) literacy classes given by cohort 1 grades 1 and 2 teachers and cohort II grade 1 teachers. Teachers were chosen mutually by the research team and the school to limit disturbance of school functioning.7 In one case, it was necessary to change the sample school because the BE teacher was absent during fieldwork.8 A replacement school was chosen nearby (5 km away). 7 Most schools had only two grade 1 or 2 teachers doing BE. 8 Veruca school, whose teacher was absent due to family illness. This school was replaced by the school of Namacorrone. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 11 Figure 1. Map with Sample Districts Table 1 shows the schools, classes, and cohorts observed by ZIP, district, and province, as well as the distance from school to district headquarters and observation start dates. Table 1. Schools, Cohorts, and Grades Observed by Province, District, and ZIP Province District ZIP School Cohort Class Language Distance (km) Date Zambézi a Mocuba Derruba EP1 de Murrué I Grade 1 Echuwabo 6 July 8 II Grade 2 Echuwabo Namacur ra Vuruca EPC Namacorron e I Grade 1 Echuwabo 39 July 22 Alto Molócue Namicop o EPC Michulane II Grade 1 Elomwe 30 July 11 I Grade 2 Elomwe Gilé Khayane EPC Khayane II Grade 1 Elomwe 73 July 16 Nampula Mogovol as Moquito EPC de Nameturi I Grade 1 Emakhuwa 18 July 8 II Grade 2 Emakhuwa Moma Pilivili EP1 Eduardo Mondlane II Grade 1 Emakhuwa 32 July 11 Moma Milige EPC de Camaculo II Grade 1 Emakhuwa 36 July 17 Eráti 25 de Junho EP1 de Thumar I Grade 1 Emakhuwa 40 July 22 I Grade 2 Emakhuwa 12 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV In July, the research team made 34 class observations (3 observations per teacher, except in Mocuba, where only two classes could be observed for each of two teachers).9 Each observation lasted 90 minutes, covering two 45-minute classes. The research team observed a total of 51 hours. The first two levels of the cascade model were observed in August, with the researchers participating in the ToT sessions held in the provincial capitals and the TTDL sessions in two of the districts where class observations took place, Eráti and Namacurra. Also, four PWs were observed in four districts in June and July. See Annex VII for a detailed fieldwork plan. The research team contacted or consulted 101 people at various levels (37 in Nampula, 58 in Zambézia, and 6 in Maputo), 23 percent of whom were women. (See Annex III for list of persons contacted). Most respondents and teachers observed were men, reflecting the fact that in rural areas most teachers, SPs, and education staff are men. Table 2 shows the number of people surveyed by sector, gender, and province. Table 2. Number of People Contacted by Sector, Province, and Gender Sector Women Total Nampula (37, 9 women) DPEDH 1 3 SDEJT 2 (Mogovolas) 9 TTI 1 1 ZIP 0 4 DE/DAE 0 5 Teachers 5 (Mogovolas) 11 VL! 0 4 Zambézia (58, 13 women) DPEDH 0 2 SDEJT 6 27 TTI10 2 3 ZIP 0 1 DE/DAE 1 5 Teachers 4 16 VL! 0 4 National Level, Maputo (6, 1 woman) MINEDH 0 2 VL! 1 4 Total 23 101 9 In Mocuba, grades 1 and 2 overlapped on the first day of the visit, and a teacher was ill on the second. 10 The Teacher Training Centers, managed by ADPP, were included in this category. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 13 Data Collection and Analysis Methods The study used primary and secondary sources. For primary sources, the data collection techniques used were direct observation, semistructured interviews (individual and group), and focus group discussions (FGDs). The interview and FGD scripts, as well as class and training observation guidelines, were prepared and submitted to MMEMS for consideration (see Annex X and XI guidelines). Some guidelines, such as the PW observation and ToT and TTDL guidelines, were then enhanced based on observations from the first PW and school lessons. Direct observation was fundamental to collecting teacher performance data in the classroom and on approaches, content, and learning from teacher education activities. The direct observations also allowed the research team to deepen some analyses and illustrate the interview and document review findings. The interviews provided information about VL!’s implementation and context and served to problematize the observation and document review data. The FGD and group interviews deepened the understanding of the training process and results and enabled comparison of teachers’ experiences. The secondary sources were VL! reports and documents, particularly those related to teacher training. The team reviewed training programs and reports, PW reports, PW summary reports, teaching materials, and teacher training and lesson planning manuals (see Annex II for documentation reviewed). VL! annual plans and quarterly reports were also studied. Document analysis was used for initial information about the program and then for comparison of some data, in particular for learning questions 1 and 3. The team also consulted telephone survey data compiled in April 2019 to identify factors that could be associated with training results.11 Table 3 shows the intersection of the various techniques with the primary information sources. Table 3. Research Techniques and Primary Information Sources Research Technique Information Sources Individual interviews Directors and managers of the MINEDH, DPEDH, SDEJT, CFP/TTI; SPs; and ZIP coordinators FGDs and group interviews BE teachers, educational technicians at district level, ZIP facilitators Note Selected teachers; ToT, TTDL, and PW training sessions The analytical framework presented in Annex IV shows how these techniques and sources of information were used to collect data for each learning question. The consultant team consisted of five senior researchers and three assistants. All team members had experience in qualitative methodologies, and one is a reputed BE expert who has published on the subject. Four of the five senior researchers conducted fieldwork, accompanied by local research assistants with deep knowledge of the three native languages used in the program, as well as teaching and training experience. The assistants, two of whom had served as VL! consultants before, paired up with the senior researchers for classroom observations, focusing their attention on lesson content, 11 Computer-assisted telephonic interviews (CATIs), conducted regularly by MMEMS to collect monitoring data. 14 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV student learning, and teacher skills. The research team leader had also led DLAS 1. All team members except the team leader were men. Data from primary and secondary sources were triangulated to ensure reliability. Stakeholders such as teachers, trainers, government, and the VL! team supplied information on teacher training. Abundant written information, particularly on PWs, allowed the observation data to be compared with reported data. Comparison of information from various sources and with various methods ensured the validity of the findings and conclusions. Research Limitations DLAS 2 covered a limited number of schools and teachers. Consequently, classroom observation results cannot be generalized to all teachers or even to groups of teachers. However, similarities among results in various locations may indicate trends in a large number of BE schools supported by VL!. The study’s limited scope prevented further analysis of some findings, such as the concentration of teachers with the most difficulties in Nampula. A second important limitation is that the research activity calendar did not follow the teacher training model activity chain. Classroom observations were therefore conducted before observation of the ToT and TTDL sessions, whereas the reverse would have enabled a closer comparison between what was taught in the sessions and what was applied in the classrooms. The team sought to overcome this limitation by collecting more detailed information on the May 2019 training session content and approaches, as the observed practices would be influenced by this training. Finally, the study results should be read in conjunction with the results of the early grade reading assessment (EGRA) that was being prepared at the time of DLAS 2. In the EGRA, teacher impact is measured by student learning. Although teacher training makes an important contribution to learning, training itself is only one of several impact factors, including school management (attendance, punctuality, and teacher presence in classrooms), teaching conditions (student attendance, infrastructure quality and adequacy, existence of basic teaching materials), and teacher motivation.12 12 The Report of the Second National Assessment of Third Grade 2016 of the Instituto Nacional do Desenvolvimento da Educação (INDE) analyzed the factors associated with student performance. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 15 2. FINDINGS The study findings address VL! capacity-building activities, curriculum compliance, training model strengths and weaknesses, and differences in training outcomes between Nampula and Zambézia. See Annex IX for a complete table of findings, conclusions, and recommendations. Learning Question 1. What were the desired results of VL! capacity-building activities (ToT, TTDL, and PWs) for acquisition of EGR pedagogical skills, knowledge, and practices? The cascade training model defines activities at three levels: provincial-level ToT; TTDL for teachers, ZIP facilitators, and coaches; and ZIP-level PWs. The team also considered sub-IR 1.4 on in-school coaching part of the teacher training model. Held in the provincial capitals, the ToT is intended to train those who will then guide the TTDL. The ToT desired results are that trainees:  Understand how the BE model works and the roles and responsibilities of the various actors in the BE training process.  Acquire technical and methodological skills to train BE teachers in reading and writing teaching methods and routines.  Be able to use current BE teaching materials and produce others.  Learn how to work across themes within the classroom.  Acquire knowledge and skills to assess students based on defined standards, identify students with learning disabilities, and plan and implement remediation activities.  Gain skills to train teachers and coaches and train ZIP facilitators on PW organization and implementation and BE monitoring. The TTDL desired results for teachers are that trainees:  Apply the BE model as taught.  Teach lessons based on curriculum maps and lesson plans.  Use appropriate methodologies and techniques for teaching L1 reading and writing.  Use approved materials and produce complementary materials.  Address crosscutting themes in classes.  Perform continuous and periodic student assessments.  Manage student attendance and punctuality.  Manage student participation by ensuring gender inclusion and equality.  Identify students with learning disabilities and plan and implement remediation activities. 16 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV The TTDL desired results for ZIP facilitators and coaches are that trainees:  Organize and implement PWs to complement teacher training.  Supervise and monitor program implementation, identifying teachers’ pedagogical needs and addressing them.  Monitor school implementation of BE methodologies and techniques. The PWs are expected to simulate teacher-suggested routines and help teachers:  Identify and respond to teaching difficulties, supplementing district-level training.  Plan classes.  Produce needed class materials.  Exchange experience among peers. VL! thus addresses specific BE training areas (approaches, methodologies, and content) but also empowers teachers in other areas of more general teacher training, such as classroom dynamics, gender equality, inclusion, and participation. Teachers are expected to have acquired some of these general skills in initial teacher training courses. Learning Question 2. Are teachers implementing gender-responsive instructional practices that support BE EGR, in accordance with the desired outcomes established in the training model? These findings address four instructional dimensions: plan compliance; application of gender-sensitive and inclusive practices; classroom interaction; and pedagogical skills, knowledge, and practices. Plan Compliance The teachers observed by the research team followed the defined curriculum map in their lesson plans.13 The quality of the lesson plans drawn up by teachers varied, however. In most cases, teachers used the first-quarter lesson plan book (LPB) to draw up the current lesson plans, as they had not received the second-quarter LPB. In one case, a teacher used a 45-minute lesson plan for a 90-minute class for the three observed days. Many teachers mentioned having enough skills to make detailed lesson plans. They considered it necessary to elaborate lesson plans to adapt the lessons to different situations, even when they have LPBs. Teachers’ lesson plans did not always take into account the amount of time required for participatory teaching techniques. Additionally, most of the observed classes had many students, many of whom were late, compromising the effective class start time. Under these conditions, teachers could not fully comply with the standard lesson plan for most of the observed classes. Attention to crosscutting themes was one of the curriculum areas most affected by time constraints. Most observed schools and classrooms had rudimentary physical conditions (see Annex VIII) that do not favor the implementation of some defined teaching approaches (such as group work and talking room) 13 The curricular map defines the letter to be studied, the text from the student textbook to address, the text for reading aloud, and the crosscutting theme of the week. The weekly map generally defines the content for each week and class day and the methodologies to use. From the weekly map more specific lesson plans are drawn up. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 17 and make classroom management difficult for the teacher. Several schools had no desks or doors, and room covers were precarious (in rainy weather it rains inside the classroom). Although teachers were always on time in class (only 34 observations were delayed) and were present throughout the class, student tardiness hampered lesson plan implementation. Delays of 30–40 minutes were frequent. In addition, between 10 percent and 46 percent of enrolled students did not attend classes on observation days. These absences will require remediation activities later, with consequences for lesson plan compliance. Gender-Sensitive Practices and Inclusion Observation of classes, training sessions, and PWs, as well as interviews and FGDs, has shown that the teaching materials used by teachers give visibility to girls and women and issues of exclusion and inclusion. However, teachers made no efforts to use these materials to their full potential to stimulate further discussions on gender equality and inclusion. Attention to gender equality was almost always limited to the balanced participation of students in the classroom and the attendance of students within the classroom. Although training provided guidance on promoting gender equality, it did not provide enough analysis, methodologies, and/or tools. Also, the time available to discuss this issue at the training sessions observed was very limited. One factor to consider is the limited number of female teachers involved in VL!. Although MINEDH statistics indicate that about 51 percent of primary school teachers are women, this proportion is smaller in the central and northern provinces and in rural areas (where VL! is implemented). According to VL! sources, women constitute 32 percent or less of the trained teachers.14 In the sample of observed teachers, women constituted only 25 percent of the total.15 In most schools observed, there were no female teachers implementing BE. Interviews and FGDs indicated that teachers had difficulty identifying and addressing gender issues, and the few analyses and discussions were stimulated either by outsiders (research team members) or by female teachers with knowledge of the subject. Because a substantial proportion of the teachers come from the areas where they teach, they are part of a social organization with cultural norms that complicate the promotion of gender equality. The importance of culture in shaping values and attitudes toward gender and inclusion should not be minimized: one female teacher said that in her class a grade 1 student (6–7 years old) told a classmate with some difficulties not to worry because education wouldn’t do her any good as an adult and what she needed was to learn to cook. In this case, the teacher presented her own life as an example of how a girl can study to become a teacher. Male teachers may not be able to respond to this situation, themselves belonging to the culture that maintains this value. 14 Data communicated by VL! in Maputo on September 24, 2019. 15 Communication of VL! on September 24, 2019. 18 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV No evidence was found that BE monitoring pays adequate attention to gender equality and inclusion issues within the classroom. Although a gender monitoring form (gender assessment and inclusion in school education practice) was introduced in the August ToT, it was not used in teacher training in the districts observed the following week. No children with physical characteristics or limitations normally associated with social exclusion were noted, except in a classroom where there was a well-integrated albino child. In the observed classes, teachers made efforts to support children with learning disabilities, with good results in some cases (e.g., Alto Molócue, Mocuba, and Mogovolas). In other cases, it was not clear whether teachers had enough knowledge to accurately identify the learning problem and to effectively respond to students’ difficulties. For example, in one room there were three or four children who could not write the letter “S”; two improved with the teacher’s support, but one did not make any progress. A more accurate diagnosis of the condition was needed to help the student. In the observed training (ToT and TTDL), this problem was addressed in a generic way rather than defining various types of difficulties and concrete measures for each. Classroom Interaction In all the classes observed, there was a very comfortable environment among students and between students and the teacher. All respondents attributed the difference between this comfortable environment and the environment prevailing in monolingual classrooms to the use of the mother tongue as a vehicle for teaching and learning. Within the classroom and on class subjects,16 there was more communication between students and teachers than among students themselves. Some BE approaches favor communication between teacher and student, such as the gradual progression model (“I do, we do, you do”). Activities that encourage students to collaborate and interact with each other, such as peer reading and help provided by more advanced students to struggling ones, were less commonly used, but others, such as writing on a colleague’s back, were often used. Pedagogical skills, knowledge, and practices In the observed classes, teachers demonstrated good mastery of interconnecting language skills (listening, speaking, reading, and writing) and some competence in the gradual progression routine (“I do, we do, you do”). However, this routine was not always used even when it was needed and sometimes was not used correctly and completely (possibly due to time limitations). For example, in some exercises only the first two steps were completed and in others the correct sequence was not implemented. In another case where it was important to apply the routine, it was not applied. The recommended reading techniques were correctly applied in about 70 percent of the observations (25/34). In particular, phonological awareness activities were applied well. However, there were some difficulties dealing with phonetic issues such as different sounds for the same letter and phonetic variations between dialects of the same L1. Teachers generally used the recommended techniques for teaching writing. 16 In addition to communication on class subjects, the students communicated widely and seemingly as a result of fellowship and neighborhood. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 19 In most classes observed, all students had received the reading book and exercise book, but three classrooms (25 percent) experienced gaps: in a grade 1 room in Mogovolas, the books received were not enough for all enrolled students; in Alto Molócue there was a room where no grade 2 student had the student’s book; and in Gilé no grade 1 students observed had the exercise book (the books were at ZIP headquarters). Most of the observed classrooms had the complementary textbooks although, for example, in Alto Molócue not all classrooms had Big Books.17 Local production materials were limited, and in 8 of 12 classrooms there was no material affixed to the walls. In one school, the teacher placed the materials daily on the classroom wall and removed them at the end of the class because of poor classroom conditions that did not guarantee their conservation. In some schools there was no safe storage place for books. Interview respondents often reported delays in the arrival of teaching materials to schools and consequent disturbance in the teaching and learning process. The teachers made systematic, continuous student assessments although they did not record them in files. Many student notebooks were corrected daily, but sometimes the teacher marked incorrect answers as correct. The team conducted four observations on Fridays, the planned day for weekly evaluations, but in two schools no weekly evaluations were performed. In another school, the evaluation was well done, with writing recovery exercises in the classroom and groups without difficulties who were outside the classroom to perform reading exercises and then retell their stories. In the fourth school, a teacher tried to do the weekly evaluation but had some difficulty because the excessive number of students (90 of 98 enrolled students) prevented him from doing the remediation activities well and controlling the group outside the classroom. The learning assessment process has undergone some modifications during VL!’s implementation to make it more appropriate to classroom conditions. Interviewees mentioned that the original plan to fill out student daily assessment sheets proved inappropriate for the time needed. At the time of fieldwork, the guideline was to make summary assessments by recording them in student notebooks, but not all observed teachers seemed to be aware of this new approach. The observations from teacher training sessions at district and ZIP levels showed that teachers still have many doubts about BE pedagogy and the (written) mastery of the Mozambican languages used in teaching. Students varied in their progress—in the observed classes, some children could already read after five months of classes (as a result of effective learning rather than memorization), but some still had great difficulty even in identifying the letters. Learning Question 3. What are the cascade training model’s strengths and weaknesses as it is being applied? The cascade training model as it is being applied has strengths and weaknesses. Stakeholders presented various suggestions for improvement. 17 It was difficult to determine how many classrooms lacked the complementary materials planned because some teachers kept the materials at home and brought them to school only when needed (because of poor school conditions). 20 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Cascade Training Model The cascade training model has three components: the ToT; the TTDL for teachers, ZIP facilitators, and coaches; and the annual PWs. The model also includes teacher support and supervision by trained companions—namely the SPs and/or those responsible for the first cycle of primary education. ToT: The ToT is held in provincial capitals in three phases: January, May, and August. The January training has two stages—two days for training a group of Teacher Training Institute (TTI) teachers and four days for the TTI teachers to train the teacher trainers from the districts. In January 2019, 50 TTI teachers were trained in the first two days and 384 teachers from the selected districts in the remaining four days.18 This group then went on to train teachers in the 21 VL! districts. The May and August ToTs each lasted three days. The training sessions in the provincial capitals were led by experts from Maputo, provincial technicians, and some TTI trainers. TTDL: The TTDL was always held in the week following the ToT at the district headquarters. In January 2019, the six-day training covered new grade 1 teachers (1,524) and grade 2 teachers (1,554) from cohort 1 and grade 1 teachers from cohort II (3,658).19 In May the three-day training covered only grade 1 teachers from both cohorts. In August the three-day training covered grade 1 teachers from cohort II. The final days of these sessions were used to train ZIP coaches and facilitators, who also participate in the early days of teacher training.20 These trainings are led by provincial-trained trainers (TTI teachers and trained teachers) with the support of national-level experts (Provincial Directorate of Education and Human Development [DPEDH] and TTI). In August 2019, national-level experts did not attend district training sessions. PW: The PW takes place at the ZIP level. By 2019 seven PWs had been held, lower than anticipated at program onset (12). The half-day sessions, held on Saturdays, are facilitated by trained district-level teacher facilitators and are attended by BE teachers, SPs (coaches), and the ZIP coordinator of the implementation areas. According to VL!, the PWs have a predetermined, standardized structure consisting of reflective analysis of the previous week’s classes, planning for next week’s classes, production of materials, simulation of teacher-suggested routines, and analysis and recommendations. Some topics are openly addressed to answer the questions raised by teachers. In 2019 there was a large drop in PW teacher participation.21 The model also provides for teacher supervision and follow-up by ZIP coordinators and coaches once a month at each school and by the SP once a month for each teacher (although this is not formally included in the model design). These follow-ups are complemented by quarterly District Education, Youth, and Technology Service (SDEJT) supervisory visits, including by the VL! district officer (DO). Implementation and Feedback Model The training model’s implementation has changed, particularly the duration of training, VL! and MINEDH support for district-level training, and grant payment for PW participation. The number of training days 18 VL! Quarterly Report 2019, 19. 19 This statistic refers to teachers who participated in at least 75 percent of these trainings (VL! second quarterly report of 2019, 20–23). It does not include teachers who were missing 25 percent of the time or more. 20 In January, there were two days for this training, whereas in May and August, there was one day. 21 Only 36 percent of the expected participants were present in the PW, according to the second quarterly VL! report of 2019. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 21 per teacher has been reduced (e.g., BE teachers who entered in 2018 completed only January training in 2019 and did not attend the May and August trainings as originally planned). From mid-2019, the participation of VL! specialists and MINEDH in the district training was eliminated, limiting participation to ToT in the provincial capital. Since early 2019, transport and food subsidies for PW participation have also been suspended. There is a widespread perception that the time spent on teacher training is not sufficient for teachers to acquire the appropriate level of competence for good BE quality. Many teachers, local leaders, and other coaches interviewed indicated that teachers need more time to master L1 and improve their pedagogical skills, as many techniques (such as participatory techniques) are new to these teachers. The quality and type of questions posed by teachers at district training sessions observed by the research team support this perception. Respondents expressed widespread appreciation of trainer capacity at both provincial and district levels. Many respondents reported that they did not see significant differences in training quality between district and provincial levels. Although it might not be easy for respondents to question their trainers, observations revealed little difference in essential teaching issues between provincial- and district-level training. One district director noted that cascade training models always present a risk of quality reduction from one level of implementation to another—in this case, from the provincial to the district level. This view is widespread among those working with these models. The observation of the August trainings showed that the experts from Maputo (VL! and MINEDH) provided answers to many questions raised during the ToT, particularly on pedagogical and linguistic issues. The absence of these specialists in 2019 district training may affect training quality. Most respondents who commented on the subject indicated that the 2019 district trainers were better selected than the 2018 ones. This perception was not corroborated in one district, where it was reported that some teachers and coaches had “bought” their trainer’s vacancy. Overall, the observation showed—and respondents agreed—that the methodology of working with trainees at either provincial or district level is appropriate for current conditions and resources and allows for greater participation by all in the learning process, as well as a greater exchange of experiences among teachers and among ZIPs. In district training, the participants are organized in groups at the ZIP base where they work, and at provincial level the participants are organized by district of provenance. In most districts visited, it was reported that the ToT support materials were not available at the beginning of the January training. Also, teachers did not receive in May training the second trimester LPBs, which were fundamental to their work. In August, many participants, both in ToT and TTDL, had not brought material they had received in previous training that was needed for the work to be done during the August training, forcing the trainers to obtain extra copies to distribute to the teachers.22 22 It was not possible to verify if they were advised to bring the material for training. 22 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Several respondents criticized some logistical aspects of training sessions, particularly subsistence allowances at both provincial and district levels. It has been repeatedly stated that the allowances are not sufficient to cover travel and subsistence expenses. The training model predicts a massive participation of district trainers in the first PW, which takes place following the district training phase. This participation has been designed to allow monitoring of district training implementation and identify additional training challenges and needs. In practice, district trainer participation is limited and constrained for financial and work organization reasons. The PW should therefore play a central role in the teacher education model feedback process. The revised documentation and observation showed that this year, PW teacher participation has drastically declined, due primarily to lack of subsidy payments.23 In an FGD with BE teachers from one ZIP, no teacher had participated in the last PW. Also, the PW agenda is too long for the time available, especially given teachers’ delays. In the observed PW, pedagogical issues brought by the teachers were discussed, which informed the ZIP coordinator and coaches about the implementation challenges. A standardized report of each PW is made and sent to the district level, with no copies remaining at ZIP schools. The PW report focuses more on organizational and management issues than on pedagogical issues, although the latter are important for model feedback. District and provincial PW reports further synthesize the information received, diminishing their usefulness for model feedback purposes. Important feedback comes from the SDEJT and VL! supervisory visits, which also heavily emphasize logistics and management. TTI teachers are supposed to conduct a PW follow-up, but this rarely happens. In some interviews, lack of funding was cited as a reason. In Namacurra, teachers from the local TTI (Macuze Future Teacher Training Center) have not participated in teacher training or teacher supervision, although they are located in an area where BE is being implemented and students of this TTI are interning in the schools where BE is implemented. The information received through the various supervisory visits is analyzed at district and provincial levels for preparation of the next training stages and then sent to VL! at the central level. Teacher monitoring in schools, either by ZIP or by SP and cycle coordinators,24 presented a diversity of situations. In some cases, school heads were unclear about the frequency of teacher follow-up. In others, it was stated that the frequency should be quarterly. The same was true for ZIPs, where there was an awareness of the need to monitor schools but the supervision was limited, according to respondents, by the distances and time available for supervisory activity. Respondents in some interviews mentioned coaches’ limited technical capacity even though coaches received the same technical training as teachers. Some coaches do not teach in BE and others do not speak the L1. Some teachers reported consulting their peers and ZIP facilitators for clarification rather than asking their coaches. 23 The BE teacher’s manual prepared by VL! shows that the teachers receive a participation subsidy at the PW. 24 “Cycle coordinator” refers to a teacher designated to coordinate the first cycle of the primary study (grades 1 and 3). This position exists in schools where more than five classes are implementing BE in grades 1 and 2. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 23 Training Model Strengths and Weaknesses The team identified the following strengths: the model allows a large group of teachers to benefit at lower costs, facilitates the participation of qualified specialists, encourages the exchange of information and experience among various levels and regions, and builds district capacity for future support to BE teachers with available trainers at any time. The familiarity of the cascade training model to MINEDH teachers and technicians at various levels has favored its acceptance. The teachers’ interest in BE and phased training also contributed to positive appreciation. The involvement of TTIs, who are recognized for their mandate for teacher training, and the close involvement of education authorities at national, provincial, and district levels were considered positive features of the implemented model. The intervention of VL!, which gave access to specialists that would otherwise not be available, is another positive feature. The team found the following weaknesses: (1) the model assumes that the teachers to be trained have a greater mastery of L1 than they actually have; (2) the training time for teachers at district level is not sufficient for their needs, particularly in phases 2 and 3, when only three days of training are provided; (3) the model requires effective teacher monitoring by SPs (coaches) and ZIP coordinators and mentors, which has been limited in time and, in the case of coaches, quality; (4) the training program is very intensive, does not include sufficient practice, and provides little training in numeracy education; and (5) subsidies do not match trainee expectations and needs. The team also perceived (although this view was not expressed by any interviewee) that ToT needs more time to delve into BE philosophy and pedagogy. This training, as it is being implemented, focuses primarily on skills development and teaching at district level. It is assumed that ToT beneficiaries are already teacher trainers, BE technicians, and experienced teachers, not taking sufficient account of the limitations at this level. Some factors contributed to the above assessment—namely, the large number of schools covered by VL! and their dispersion, which dilutes capacity-building efforts; limited human resources to supervise and monitor at a level that will maintain the quality of education after training; the difficulty of mobilizing financial resources to monitor and follow up on teachers and to enable teachers to participate more in activities that are not part of the school’s routine program; and insufficient local capacity for supervision and follow-up due to limited SP training. Stakeholder Suggestions for Improving Training  Increase the involvement of SDEJTs, including directors and technicians, in ToT to strengthen the team of district technicians capable of monitoring and supervising BE at the local level.25  Ensure that grade 2 teachers continue to receive training at district sessions, particularly when BE content and methodologies are modified.  Increase to six the number of teacher training days in phases 2 and 3 (May and August).  Ensure that teachers starting to teach grade 3 by 2020 receive sufficient technical training to make the transition from L1 to L2 properly. 25 SDEJT involvement has been limited to logistical support and supervision, according to interviewees. 24 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV  Reorganize PW by bringing schools closer together (possibly revising the ZIP composition that in some cases includes schools that are very distant from each other).  Resume PW participation grants until the end of the program.  Increase the resources (financial/material) of schools to produce complementary BE materials.  Increase the training days for coaches so that they can effectively assist the teachers in their schools.  Offer L1 courses to teachers who teach this language (oral and written).  Develop and make available L1 dictionaries.  Introduce certificates of participation for teachers and trainers to reinforce their motivation.  Involve teachers at the Macuze Training Center in Namacurra in ToT, TTDL, and supervision. Learning Question 4. How do training outcomes vary and what factors can explain differences between capacity-building experiences in Zambézia and Nampula? Research work in the two provinces is comparable (number and type of activities, observation time, sources of information and methods used, observation levels [provincial, district, ZIP, and school]). In Nampula, activities were observed in the Emakhuwa language (three districts), and in Zambézia activities were observed in Echuwabo (two districts) and Elomwe (two districts). Comparison of Model Implementation There were no differences between Zambézia and Nampula on the type and format of teacher training activities, their duration, and the selection of participants for the courses. Thus, in both provinces ToT trainers were chosen from TTI teachers, DPEDH, and MINEDH technicians and VL!. In Zambézia, teachers from local universities were also included. ToT course graduates were chosen from SDEJT technicians, teachers selected from ZIPs, SPs and DSPs, and so forth equally in both provinces. The same applies to district- and ZIP-level activities. TTDL observation on August 19–21 showed some coordination and alignment among the various trainings that occurred at the same time in different districts of the same ZIP. There was also communication between the two provinces about specific program-related situations. Observation suggests that the availability of books for training (and teaching) is similar in the two provinces. Late book delivery in January ToT courses was mentioned in both provinces, as well as delays, shortages, and lack of student books and supplementary materials. In some places in Zambézia disagreements or changes in approach have been reported on how to apply reading and writing learning steps (letter, syllable, word, sentence) during district-level training sessions. This disagreement has created some confusion among teachers in training and criticism that some changes have been introduced without all teachers having to apply them (cohort 1 grade 2 teachers). It was not clear if the same happened in Nampula or if there were also divergences/changes in other training content areas. The desk review showed wide variation in PW teacher participation and varying degrees of implementation among districts. For example, in March 2019 no PWs were held in several districts of the two provinces, including Mogovolas and Moma in Nampula and Gilé and Mocuba in Zambézia. In the USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 25 districts where PWs were held, participation varied greatly from one PW to another—for example, 30 percent and 96 percent of expected participants participated, respectively, in Alto Molócue and Mulevale and 45 percent and 75 percent, respectively, in Lalaua and Mossuril in Nampula.26 This wide variation was evident in the interviews and FGDs: in two Zambézia districts, most teachers contacted had participated in the previous month’s PW, whereas in another district in the same province no FGD teacher had been present at the June PW. A similar pattern was found in Nampula. Overall, there was a good working connection between the two provinces between the DO and the SDEJT technicians, who jointly carried out monitoring and supervision visits and other activities. However, in one case in each of the provinces, this connection encountered difficulties, which seem to be linked to the VL! ownership processes. Comparison of Training Outcomes Few differences were found in training outcomes (application of classroom practices) between the two provinces. Two of the six teachers observed in Nampula (both from cohort II) had difficulties in teaching reading, whereas there were no such cases in Zambézia. And three of the six teachers observed in Nampula showed difficulties in managing students in the classroom, whereas in Zambézia these difficulties were not observed. However, it is not possible to attribute these differences to the training received; they may be associated with school management issues, individual teacher characteristics, or other factors.27 No differences were found in teachers’ ability to manage gender equality and classroom inclusion. Teachers made similar efforts to engage girls and boys in class and to support children with difficulties regardless of gender. The observations suggest that the amount of participation time in the program (entering in 2018 or 2019) was not a determining factor in teachers’ teaching skills. Cohort II grade 1 teachers were found doing a good job at the same level demonstrated by cohort 1 grade 2 teachers. However, duration in the program seems to influence teachers’ facility in responding to classroom challenges and using available teaching materials—those who entered with cohort 1 showed greater competence. In Nampula, unlike in Zambézia, the classrooms had no letter or word materials and no fixed posters on the walls, although in both provinces materials were produced during all observed PWs. This difference was not influenced by schools’ physical conditions, and the associated factors were not clear. For example, in a Zambézia district where the school was in very poor condition, the teacher collected wall materials at the end of each day to take with him and bring with him the next day. A higher proportion of teachers were observed in Nampula (2/6) than in Zambézia (1/6). The teachers with the most difficulties were men. There were no noticeable differences in teachers’ age groups: most were young, and the provinces had the same proportions of teachers who were about 40 years old. 26 VL! Quarterly Report II 2019, 24–25. 27 Two of the above-mentioned teachers are at the same school, which is 40km from district headquarters. The staff took 2.5 hours to cover the distance with a good vehicle. This factor certainly constrains school supervision and support actions. 26 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV There were no noticeable differences in school conditions between the two provinces (see Annex VIII). In both provinces there were very rudimentary schools with no doors, windows, or desks and other schools with desks and cement walls (Figure 2). Figure 2. Examples of School Conditions Nampula Zambézia USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 27 3. CONCLUSIONS This study findings show that the VL! cascade teacher training model is enabling teachers to teach L1 literacy. By 2019, more than 5,000 teachers in 21 districts of Nampula and Zambézia were teaching grades 1 and 2 students in BE as a result of training activities implemented by MINEDH with VL! support. This expansion has been made possible through the cascade training model, which has some limitations, however, particularly in continued technical support for BE teachers in the classroom. Observed teachers have shown skills in implementing lesson plans, although there are areas where knowledge and skills are still below those desired. A challenge posed to the team was a lack of clarity about how VL! will support the transition from L1 to L2, particularly as VL! is about two years from its end. Learning Question 1. What were the desired outcomes of VL! capacity-building activities (ToT, TTDL, and PWs) for acquisition of EGR pedagogical skills, knowledge, and practices? Specific aspects of BE philosophy and pedagogy are at the heart of the cascade teacher training model, but VL! does not neglect broader pedagogical skills in which teachers may have fewer skills. In addition to facilitating acquisition of pedagogical skills, knowledge, and practices in the BE approach established by the GRM, VL! has provided and complemented training in participatory teaching methods, assessment methods and processes, gender equality and inclusion, and other practices. Thus, the model’s desired skills for BE implementation cover not only specific areas of teaching L1 reading and writing but also more general pedagogical training. The use of native language teaching specialists was made possible with VL!’s financial support, which has contributed to the quality of activities carried out. Some of these experts are from MINEDH and subordinate institutions. Learning Question 2. Are the teachers implementing gender-responsive instructional practices that support BE EGR, in accordance with the desired outcomes established in the training model? The observed teachers are implementing inclusive teaching practices and have some gender sensitivity. The observed practices correspond to what is established in the training model, although quality varies from teacher to teacher and weaknesses were found, such as promoting gender equality in BE. The study showed that teachers had enough skills to implement BE although there are areas that need improvement. Demonstration of technical teaching skills does not ensure student learning, however. First, it does not ensure that what is observed is everyday practice: teachers and school boards were aware that they were being observed and this awareness may have influenced the dynamics implemented. Second, the training itself is not enough to ensure the desired learning results, as many other factors influence the learning process, such as school management, class conditions, student attendance and delays, and physical school conditions. 28 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Plan Compliance The planned BE program weekly learning content is generally followed. However, weekly plan activities are not always fully or correctly implemented. The teaching content and dynamics are ambitious for class conditions (space, infrastructure, student number and punctuality, etc.). These conditions constrain program implementation despite efforts by teachers and school boards to overcome some challenges. Several factors influence plan compliance. Contributing positively are the use of students’ mother tongue as a vehicle for teaching and learning; student interest, motivation, and participation; and teacher interest in the modality and availability of attractive and appropriate teaching materials. Contributing negatively are the late arrival (or even absence) of some student teaching materials and teacher support materials; teachers’ limited ability to plan classes; and some adverse teaching conditions (school infrastructure and student absences and delays). Gender-Sensitive Practices Teachers have some knowledge of gender and inclusion issues and have been given general guidelines to promote them. However, these guidelines focus on promoting equal class participation of boys and girls and distributing students appropriately throughout the classroom. Other aspects of gender equality are not addressed. The teachers have limited ability to put gender equality and inclusion processes into practice. The low proportion of female teachers also counteracts efforts toward gender equality. BE training and monitoring still pay little attention to operationalizing the gender approach (what to do and how to do it). Almost all teachers observed showed attention to integrating children with more difficulties into class dynamics. Class time and size constrain teachers’ efforts to support students individually, however, and teachers do not use interstudent help enough to address some learning difficulties. Classroom Interaction All respondents observed that classroom interaction was much better in BE classes than in monolingual28 classes. The teachers successfully and systematically used teaching methods that promote student participation and achieve a favorable learning environment. Classroom interaction is centered more on interaction with teachers than on interaction among students. Pedagogical Skills, Knowledge, and Practices Most of the observed teachers have enough BE skills, knowledge, and practices and know how to do their work. There are, however, important limitations, notably in lesson planning, phonetics, vocabulary, and numeracy teaching approaches in L1. Proficiency in L1 varies among teachers: the fact that writing in L1 is new to most teachers influences the teaching process. 28 Although no lessons were observed in the monolingual modality, all teachers and other consulted sources categorically expressed this view. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 29 Teachers demonstrated good mastery and use of some methods, but other methods (peer reading, silent reading, tale and retelling, group work, role play) were less widely used. It is not possible to determine whether this is due to lack of mastery or to other reasons. The daily and weekly student assessment process does not appear to be standardized and it is unclear how it should be done, possibly due to changes introduced during program implementation. There are well-founded concerns as to whether the investment made so far to ensure a smooth transition from L1 to L2 has been sufficient and what the impact of lack of attention to this issue will be on the continuation of BE for students already integrated into it. Learning Question 3. What are the cascade training model’s strengths and weaknesses as it is being applied? The cascade teacher training model is familiar within the Mozambican education system. Its implementation has gradually and cost-effectively trained more than 5,000 teachers in a short time and with sufficient quality for BE implementation. VL! gave access to expertise that would not otherwise be readily available because of limited local experience with large-scale BE implementation. The model, however, has some weak elements that affect its efficacy. The PWs and local teacher monitoring—key elements in outcome quality—are not yet having the desired effects. Training Model Implementation There is no shared common view on how the cascade model works because of adaptations made throughout the implementation. There have been changes in frequency, duration, participant selection, and activity methodologies between 2018 and 2019. The model is still being refined, but no records were found describing exactly what the model consists of at this time. There is a widespread perception that the current teacher training time is not sufficient for BE requirements, especially in the May and August sessions. The teachers and various respondents are aware of teachers’ linguistic and pedagogical weaknesses, to which the current model gives a limited response. The short time has led to insufficient practice in some areas, such as L1 writing, phonetics, gender equality, and inclusion. Training quality generally responds to stakeholder expectations, and there is a general awareness of time, human, and financial constraints. Except for one district, there were no significant criticisms of provincial- and district-level training quality. However, the research team considers that there are still limitations that need to be addressed. PWs are failing to fully fulfill their role in teacher improvement, capacity building, and supervision. The time allocated to PW is very short for the standardized agenda that has been set. Cutting subsidies for PW participation has reduced PW effectiveness. TTIs have participated in provincial- and district-level training and some oversight. This participation in BE implementation has not been sufficient, particularly for teacher supervision and monitoring and PW participation. Even where follow-up plans exist, they are often allegedly not implemented for lack of funds, although the TTIs have a budget line for this activity. 30 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV External support (central and provincial level) has been critical to training quality. This support translates into planning and organizing the training agenda and contents, as well as resolving doubts presented during the training sessions, particularly in relation to specific aspects of BE pedagogy. Subsidies for ToT and TTDL trainees are insufficient. Training Model Strengths and Weaknesses The cascade model corresponds to the teacher training model used by MINEDH, which makes its acceptance easier. The model has a good cost-benefit ratio, reaching a larger number of teachers at lower costs. There are, however, risks that training quality may decline as it expands to the periphery, although this risk was not recognized by most people interviewed for the study. The PWs, although central to the process of teacher education follow-up and feedback, are the model’s weakest element due to their short duration and low teacher and coach participation. Local monitoring is also limited by SPs’ preparation level for the duties they have to perform. Implementation of this model on the expected scale requires a very high level of effort on the part of those involved and has little scope for error. Human capacities are limited to ensure teacher training supervision and monitoring at this scale. The positive view of BE’s potential to promote EGR encourages stakeholder involvement and participation in the training. Stakeholders indicated three main areas for improvement: greater SDEJT involvement, PW decentralization, and enhanced training for coaches. Learning Question 4. How do training outcomes vary by province and what factors can explain differences in capacity-building experience in Nampula and Zambézia? There were no significant differences in the way the cascade training model was implemented in the two provinces, either in the activities carried out or in the inputs for program implementation. There were also no noticeable differences in school or teacher characteristics. There were more teachers with pedagogical deficiencies in Nampula than in Zambézia, but it is not possible to generalize these observations due to the small sample size. The sample size did not enable identification of factors associated with the few differences in training results found between the two provinces. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 31 4. RECOMMENDATIONS The cascade teacher training model that has been implemented has succeeded in preparing large numbers of teachers for BE implementation in Nampula and Zambézia. However, some aspects can be improved to reinforce the results achieved so far. Desired Training Outcomes  R1.1 Make the desired outcomes for PWs and in-school coaching activities more explicit and measurable.  R1.2 Formally include PW activities and in-school coaching as components of the cascade teacher training model. Gender-Sensitive Practices and Inclusion Plan Compliance  R.2.1 Ensure that the lesson plans (volume and type of activities) consider the specific conditions under which they occur, allowing teachers to apply them with the expected quality.  R.2.2 Guide teachers on what to do in specific situations where the week’s plan cannot be fully met to maintain the integrity of the learning process (letter, syllable, word, etc.).  R.2.3 Make quarterly plan manuals available before the beginning of each quarter so that teachers can plan their classes more accurately.  R.2.4 Strengthen teacher training in elaborating/adapting detailed lesson plans. Gender-Sensitive Practices and Inclusion  R.2.5 Reinforce program emphasis on gender issues by addressing them systematically in all capacities down to the school level and introducing adequate teacher materials and tools.  R.2.6 Place more emphasis in teacher training on having stronger students help weaker students to promote solidarity and enable teachers to focus on the students with the most difficulties.  R.2.7 Ensure that the program implementation monitoring system assesses gender-sensitive practices and inclusion.  R.2.8 Continue to use participatory methods to promote student interest and achieve better learning outcomes.  R.2.9 Reinforce the use of program methods that promote more student cooperation. Pedagogical Skills, Knowledge, and Practices  R.2.10 Increase training time and conduct more hands-on lessons on identified weak elements (lesson planning; phonetics; writing; numeracy; vocabulary; and methodologies, including pedagogical remediation methodologies). 32 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV  R.2.11 Increase teacher training time by revisiting the approach of excluding teachers from the previous year’s cohort from the training package for a given year.  R.2.12 Strengthen teacher training in L2 teaching methodologies.  R.2.13 Clarify the daily and weekly student assessment system to standardize its implementation. Evidence-Based Practices  R.2.14 Reinforce the training model processes and feedback elements to ensure that practical BE implementation informs the design and content of subsequent teacher training.29 Training Model Strengths and Weaknesses  R.3.1. Analyze training activity implementation from the first two years, harmonize the model, explicitly write it out, and clarify it for all stakeholders.  R.3.2. Strengthen PWs’ role in teacher training, supervision, follow-up, and feedback on the training process.  R.3.3. Improve PW documentation to better capture difficulties in pedagogy and LI.  R.3.4. Revisit logistical support elements at various levels to better respond to training and trainee needs and sustainability of results.  R.3.5. Maintain the involvement of external experts in providing training technical support throughout the program to improve the quality of local human resources and create sustainable skills.  R.3.6. Revisit the role of TTIs in supervising and monitoring BE to improve the quality of follow-up.  R.3.7. Maintain the cascade training model but devote more time to teacher training and reinforcing PWs (alignment between the PW agenda, the monitoring difficulties experienced by ZIP teachers, and the detailed planning of the following weeks’ classes), ZIP pedagogical support to schools, and follow-up within schools.  R.3.8. Revisit the issue of logistical conditions to ensure PW teacher participation.30 Differences Between Provinces  R4.1 Compare this study’s findings with the EGRA learning outcomes and deepen the analysis of any factors associated with significant differences. 29 See recommendation 3.2. 30 PW teacher participation was subsidized up to the end of 2018. Following the orientation from MINEDH, these subsides ceased being paid in 2019, with the justification that it would not be possible to maintain them after the end of the program. This reflects a concern with the activities’ sustainability. However, sustainability can also be seen as technical sustainability. In this perspective, it is important to build a wide group of better-qualified BE teachers (for which it is important to participate in the PWs). From the interviews it emerged that the teachers and other technicians are aware that the subsidies will be paid only during the life of the project and don’t have expectations about their continuity after VL!. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 33 ANNEX I: TERMS OF REFERENCE STATEMENT OF WORK Vamos Ler! (Let´s Read!) - Deep Learning & Adaptive Study #2 BACKGROUND Vamos Ler! is the five-year (2016-2020) flagship program of the U.S. Agency for International Development (USAID) Education Office in Mozambique. Implemented by Creative Associates Consortium (CAC), the Activity goal is to improve reading fluency and comprehension in the local language of primary (grades 1-3) students in the target regions of Nampula and Zambézia Provinces. Vamos Ler! in partnership with the Ministério de Educação e Desenvolvimento Humano31 (MINEDH), not only aims to improve literacy in school-going children, but also support MINEDH to lay the foundation needed to strengthen the bilingual education system as a whole. Vamos Ler! implementation activities seek to provide holistic assistance as follows: by promoting national early grade reading (EGR); introducing and delivering bilingual education pedagogical tools to teachers; strengthening national literacy policies; increasing parental and community engagement in early grade literacy and establishing functional monitoring systems. Desired outcomes of the Activity include formulation of new policies, systems, tools, and approaches that can support early grade literacy in Mozambique in years to come. The Activity Results Framework has three intermediate results (IR) areas: IR 1. Early grade reading (EGR) classroom instruction improved; IR 2. National EGR policy framework and delivery systems improved; and IR 3. Parental and community engagement in EGR increased. PURPOSE The external evaluation of Vamos Ler! relies on a combination of methodologies. The longitudinal design of this evaluation consists of a mix of computer-assisted telephone interviews (CATI) and insights gathered from periodic deep learning & adaptive studies (DLAS). Overall results of the external evaluation will be informed by these methods. The purpose of the DLAS #2 is to assess the efficacy of the VAMOS LER! cascade training model, used for cohorts I and II, in relation to teacher performance in the classroom. To date, VAMOS LER! implementation has primarily focused on delivery of IR 1 and IR 2. DLAS 1 addressed IR 2, providing key insights on progress made towards strengthening both national policies and mechanisms to support EGR. DLAS 2 will focus on IR 1. More specifically, this study will explore progress made under IR 1, only as it relates to sub-IR 1.2 and sub-IR 1.4. Progress monitoring for IR 1.1. is currently addressed through CATI and school management training is not part of this DLAS #2. The Results Framework guiding implementation of the sub-IRs is presented in Figure I below. 31Ministry of Education and Human Development 34 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Figure I. VAMOS LER! Results Framework – Intermediate Result 1 VAMOS LER! TRAINING MODEL Cascade training is commonly used in education programs to build teacher capacity, by enhancing teacher training through continuous professional development (CPD). Well-known benefits of using this model is that it does not require large blocks of time (training usually takes place for up to one week) and is cost-effective. Vamos Ler! uses a cascade training model to train grade I and grade 2 teachers in the bilingual education approach. Three training sessions are held each year, set to take place in January, May and August. Each training covers different content with subsequent sessions introducing teachers to new content. This naturally assumes that cohorts of teachers targeted from training session I will receive additional rounds of training. Based on Vamos Ler! plans to expand to cover more schools, it is expected that the total number of teachers trained will increase in each round to include new teachers. Figure 2. VAMOS LER! Cascade Training Schedule – 2019 January 2019 Training session I - A - Province level: IP & MINEDH jointly train district trainers (including ZIP coordinators and personnel of District Service) on bilingual education on how to provide training to district teachers (this is the ToT) •District level: District trainers train teachers ZIP level: ZIP coordinators provide ongoing supervision and support (coaching) to teachers May 2019* Training session II - B - Province level: IP & MINEDH jointly train district trainers (including ZIP coordinators and personnel of District Service) on bilingual education on how to provide training to district teachers (this is the ToT) •District level: District trainers train teachers ZIP level: ZIP coordinators provide ongoing supervision and support (coaching) to teachers August 2019 Training session III - C - Province level: IP & MINEDH jointly train district trainers (including ZIP coordinators and personnel of District Service) on bilingual education on how to provide training to district teachers (this is the ToT) •District level: District trainers train teachers ZIP level: ZIP coordinators provide ongoing supervision and support (coaching) to teachers Vamos Ler! Activity Goal: To Improve Early Grade Reading (EGR) Outcomes IR. 1.2. Teacher use of evidence-based, gender￾responsive instructional practices in EGR increased IR 1.3. School management and governance in support of EGR strengthened IR. 1.4. Coaching and supervision of EGR instruction IR I: EGR Classroom Instruction Improved USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 35 VAMOS LER! TRAINING APPROACH The primary aim of the training is to pass new bilingual pedagogical knowledge, skills and competencies to teachers in Nampula and Zambézia. During these training sessions, the implementing partner (Creative Associates Consortium - CAC) conducts the first of three ToT sessions at Provincial level. Trainers targeted for training include MINEDH officials, TTI, District Bilingual Education focal points, among others. Once formally trained in the bilingual approach, they are solely responsible for training teachers at district level. They also conduct supervision and mentoring of teachers at ZIP level. At school level, trainees include school directors, deputy school directors and teachers. It is important to note that at ZIP level, trainers are expected to provide continuous support and supervision to newly trained teachers on a monthly basis. LEARNING OBJECTIVES The overarching objective of DLAS 2 is to gauge overall effectiveness of Vamos Ler! capacity building efforts under IR 1 regarding teacher’s training in literacy. More specifically, MMEMS will assess the quality of teacher training. Learning objectives are listed below:  Review training materials to establish desired training outcomes. This is expected to be done for training session1 (Jan 2019), session II (May 2019) and session III (August 2019) of Vamos Ler! Activity (i.e., acquisition of new skills, knowledge, practice of teachers to support EGR for grades I and II).  Document the approach used in teacher training for session III (August 2019) at Provincial and District levels as well as in the pedagogical workshops 5 and 6 (June and July 2019), noting merits and demerits in implementation standardization to generate evidence and learnings for future programming to support EGR in Mozambique  Compare and contrast teacher training outcomes in Nampula and Zambézia for teachers drawn from Cohorts I and Cohort II  Describe the process of coaching, mentoring and supervision of teachers at school level.  Document teacher practices in classroom vis-à-vis the desired learning outcomes of the capacity building activities LEARNING QUESTIONS Question 1. What were the desired outcomes of capacity-building activities (ToT at provincial level, Training of Teachers at District Level and Pedagogical Workshops), as it relates to pedagogical skills, knowledge and practice acquisition to support EGR? The desired outcomes of the capacity building activities are defined in the training’s materials of both: trainers and teachers. Thus, in order to respond to this question, the team will review the training materials used in 2018 and 2019 in the three training levels: ToT at provincial level, training of teachers at district level and pedagogic workshops at the ZIP level. Reports of these events should also be reviewed. The tools to be used during field work, including for training observation, classroom observation, interviews and guidelines for FGDs should be developed base on this review. In particular, the tools for classroom observation should be drafted based on these desired outcomes. 36 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Question 2. Are the teachers implementing, gender-responsive, instruction practices that support BE EGR, as per the desired outcomes established in the training model? This question will be answered mostly by findings and conclusions obtained with the observation of teachers in the classroom. This observation will be done against the desired learning outcomes (literacy for teachers) of the events of the cascade training model. The team will explore the gaps that may exist in terms of pedagogical skills, knowledge and practices. Interviews and FGDs with teacher may complement the classroom observations. Question 3. What are the positives and negatives of the cascade training model as it is being applied? This question will be answered by observing the capacity building activities at all 3 levels and by interviews with key informants, as well as FGDs with teachers. The team will explore whether the competences acquired by the teachers through the cascade training model is conducive to effective, gender responsive, instruction practices by teachers. The team will also explore gaps that may exist in the cascade training model process how can they be addressed. Question 4. How do training outcomes vary by province and what factors can explain differences in capacity building experience in Nampula and Zambézia? The team will explore similarities and/or differences exist in the implementation of the VL! cascade training model in Nampula and Zambézia. The team will explore whether gaps exists and how they can be addressed? Audience and Intended Users The learning study will provide valuable insights that will be used by both internal and external audiences. Key among these include:  USAID/Mozambique  Creative Associates Consortium  MINEDH  DPEDH  EGR TWG membership  METHODOLOGY Learning questions will be answered using qualitative methods, gathering critical insights to determine both efficacy and effectiveness of the training model used by Vamos Ler! Desk review: trainings materials of both: trainers and teachers. Thus, in order to respond to this question, the team will review the training materials used in 2018 and 2019 in the three training levels: ToT at provincial level, training of teachers at district level and pedagogic workshops at the ZIP level. Reports of these events should also be reviewed. Study population: At national and sub national levels, individuals will be targeted for inclusion. Insights on what has been achieved to date under IR 1 as it relates to key thematic areas highlighted under sub IRs 1.2 and 1.4 will be documented. At sub- district level, training recipients (i.e. teachers, head teachers/school directors, deputy school directors) from select schools in Nampula and Zambézia will be targeted for inclusion. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 37 Sampling: Purposive sampling will be carried out for DLAS 2. Provincial selection is guided by where Vamos Ler! is operating, thus Nampula and Zambézia will be visited. Individual selection will be guided by a combination of purposive sampling (i.e. names drawn from lists of trainees) and convenience sampling particularly when recruiting high-level stakeholders within MINEDH to participate in the study Heterogeneity in the final sample will be critical to ensure fair representation of various education stakeholders as well as gender balance, where possible. Data collection & management: DLAS 2 will use three qualitative approaches to develop a comprehensive picture the cascade training model used by Vamos Ler! Key informant interviews (KIIs) at Provincial, District, and ZIP levels; interviews with school directors and deputy school directors; focus group discussions (FGDs) with teachers drawn from select ZIPs; and observations of training sessions at provincial, district and ZIP levels32. Teachers will also be observed in classroom settings to assess whether they are using their newly acquired pedagogical skills in practice. Observations will take place in schools surrounding select ZIP sites. 8 schools are expected to be observed in 7 selected districts of Zambézia and Nampula (districts will be proposed by MMEMS/USAID). 3 days of work can be expected at each school, which includes classroom observations (3 observations per teacher), interviews with school directors and deputy school directors and FGDs with and teachers. Co￾hort Zambézia Nampula District School Grades Observa￾tions per Teacher KII and FGD District School Grades Observa￾tions per Teacher KII and FGD 1 District 1— Echuabo School 1 Grade 1 Grade 2 3 3 8 to 10 interviews with SD, DSD and ZIP coordinators in the 4 districts 1 FGD per school with teachers District 5— Emakua School 5 Grade 1 Grade 2 3 3 8 to 10 interviews with SD, DSD and ZIP coordinators in these 3 districts 1 FGD per school with teachers District 2— Elomue School 2 Grade 1 Grade 2 3 3 District 6— Emakua School 6 Grade 1 Grade 2 3 3 2 District 3— Echuabo School 3 Grade 1 3 3 District 7— Emakua School 7 Grade 1 3 3 District 4— Elomue School 4 Grade 1 3 3 School 8 Grade 1 3 3 Note: Cohort 1 includes schools located in ZIPs which started the BE under VAMOS LER! in 2018. Cohort 2 includes schools located in ZIPs which started the BE under VAMOS LER in 2019. A team of 2 consultants will conduct classroom observations at each school. As for Cohort 1, at least 2 teachers will be observed in each school (1 of Grade 1 and 1 of Grade 2). However, schools with two shifts are expected to be selected as this can increase the number of teachers observed in each school to 2 of grade 1 and 2 of grade 2. As for Cohort II, Grade 1 teachers will be observed. 32 Province and District level in August. As for ZIP level, 2 literacy workshops should be observed in June (Oficina de Literacia 5), being one in each province and 2 literacy workshops should be observed in July (Oficina de Literacia 6), being also 1 in each province. 38 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Considerations for the selection of schools include (but are not limited to): distance to the “Sede do Distrito”, local language used, among others. Tools include interview guides, group discussion guides and observation checklists. Where possible, interviews will be audio recorded for quality control. Observation data will be captured using hard-copy checklists and later transferred into MS excel. Where possible, images will be captured to provide visual documentation of the training process including tools and materials used in training. Both hard and soft copy data will be filed and securely stored in field. Data analysis & reporting: Thematic (content) analysis will be carried out on interview data. Observation data will be used to provide some illustrative insights to corroborate and complement qualitative findings. Special attention on changes in teacher trainee competencies for grades 1 and 2 will be analyzed against standards defined in the training manuals used by the Program. A PowerPoint presentation on topline findings will be developed ahead of the final Word report. The total number of slides should not exceed 25 slides. The final (narrative) report should not exceed 30 pages, excluding cover page, table of contents, list of acronyms, executive summary, FCR and annex items. TENTATIVE TIMEFRAME DLAS 2 will start in June 2019 and end no later than September 2019. The tentative field schedule is presented below:  Pre-fieldwork preparation: • Inception report development – June 2019 • Tool development and finalization – June 2019 • Selection of districts, ZIPs and schools for classroom observation – June 2019  Field work (KIIs, FGDs, and observations): • Stage I. School – June and July 2019 • Stage II. ZIP level – June and July 2019 (Literacy Workshop 5 and 6) • Stage III. Provincial – August 2019 • Stage IV. District – August 2019  Analysis & Reporting: • Findings, Conclusions and Recommendations table – August 2019 • Reporting – September 2019 DISSEMINATION PLAN Anecdotal reports of the Vamos Ler! cascade training model suggest that much more can be done in the weeks right after intensive training to encourage adoption of pedagogical approach used to support bilingual education. At ZIP level, trainers are expected to be available for teachers to consult periodically, as a support mechanism for schools in the locality. Evidence from this study provide critical learnings about training approach, training curricula/ content, while highlighting skills gaps that persist among teacher trainees who have already received the intervention. This will be essential for the Activity to address in the training rounds of 2020. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 39 Final dissemination of results includes USAID, CAC, MINEDH, and educators involved in EGR in Mozambique will take place in September 2019. During this time, MMEMS will lead a participatory workshop using solution-focused questions to address known challenges related to technical support. Outtakes from Provincial. ZIP meetings will be synthesized into a two-page technical brief and action plan. COMBINED PROFILE OF THE TEAM  Academic or professional qualification in education (Master or PhD) or bilingual education (preferred)  Academic or professional qualification in training trainers, teachers’ educators, (bilingual education preferred).  Proven experience in research on bilanguage education (published papers)  Experience (at least 5 years) in evaluation of educational programs or projects.  Experience and proven competencies in leading and conducting research in Education (at least 5 years).  Experience in working in Mozambique with curriculum of bilanguage education (at least 5 years)  Knowledge of Portuguese and English 2019 TRAINING PLAN FOR VAMOS LER! Activity Schedule of Vamos Ler! Date Activity 7 Jan to 3 Feb Distribution of teaching-learning materials at schools 8 to 12 Jan Training of trainers in literacy 8 to 12 Jan Training of trainers of school management 14 to 19 Jan District Bilingual Education Training (1st cohort) 14 to 16 Jan District School Management Training (second cohort) 17 to 18 Jan District ZIP Coordinators Training (second cohort) 21 to 26 Jan District Bilingual Education Training (second cohort) First Quarter First day of school: 4 February 9 Feb School Management Workshop 1 (supervised by trainers) 11 to 28 Feb Follow-up visits by ZIP 1 coordinators 16 Feb Literacy workshop 1 (supervised by trainers) 40 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Date Activity 23 Feb School Council Workshop 1 9 March School Management 2 Workshop 11 to 29 March Follow-up visits by ZIP 2 coordinators 16 March Literacy 2 workshop 30 March School Council 2 Workshop 9 to 11 April LEMA training 13rd April Literacy 3 workshop 15 to 19 April LEMA data collection 20 April School Management 3 Workshop 22 April to 3rd May Follow-up visits by ZIP 3 coordinators 24 to 26 April Workshop of LEMA data analyses 27 April School Management 3 Workshop 2 to 4 May Training of trainers of literacy 2 to 4 May Training of trainers of school management School Holidays: 6 to 10 May 6 to 8 May District Bilingual Education Training (first and second cohorts) 8 May District ZIP Coordinators Training (second cohort) 9 to 11 May District School Management Training (second cohort) Second Quarter 18 May Literacy workshop 4 (supervised by trainers) 25 May School Council Workshop 4 8 June School Management Workshop 4 (supervised by trainers) 10 to 28 June Follow-up visits by ZIP 4 coordinators 22 June Literacy workshop 5 29 June School Management Workshop 5 13 July School Council Workshop 5 15 to 26 July Follow-up visits by ZIP 5 coordinators USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 41 Date Activity 20 July Literacy 6 workshop 27 July School Management 6 Workshop 29 July to 9 August Follow-up visits by ZIP 6 coordinators School Holidays: 12 to 23 August 15 to 17 August Training of trainers of literacy 15 to 17 August Training of trainers of school management 19 to 21st August District Bilingual Education Training (second cohort) 21 August District ZIP Coordinators Training (second cohort) 22 to 24 August District School Management Training (second cohort) Third Quarter 26 to 30 August Training of EGRA supervisors (EE) 2 to 6 September Training of EGRA (EE) interviewers 9 to 30 September EGRA (EE) data collection 14 September Literacy workshop 7 (supervised by trainers) 21st September School Management Workshop 7 (supervised by trainers) 18 October Presentation of preliminary EGRA results USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 43 TRAINING SCHEDULE 2019 USAID VAMOS LER! 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 2 3 4 5 6 7 8 9 10 11 15 16 17 18 19 20 21 22 23 24 Formação de formadores de educação bilingue Formadores, MINEDH, VL, SDEJT, DPEDH 1 e 2 12 L L L L L L L L L L L L Formação de formadores de gestão escolar Formadores, MINEDH, VL, SDEJT, DPEDH 1 e 2 11 GE GE GE GE GE GE GE GE GE GE GE Professores 1ª classe 1 9 L L L L L L L L L Professores 2ª classe 1 3 L L L Acompanhantes e Professores Orientadores (DE/DAE) 1 8 L L L L L L L L Formação de DEs e DAEs (gestão escolar) Responsabilidade IFP Directores e Directores Adjuntos 1 0 Formação de DEs e DAEs (gestão escolar) Directores e Directores Adjuntos 2 9 GE GE GE GE GE GE GE GE GE Formação de Coordenadores da ZIP Coordenadores da ZIP 2 4 GE GE GE GE Professores de 1ª classe 2 12 L L L L L L L L L L L L Acompanhantes e Professores Orientadores (DE/DAE) 2 12 L L L L L L L L L L L L Coorte Formação distrital educação bilingue (1ª coorte) Formação distrital educação bilingue (2ª coorte) Janeiro 1 Oficinas de Gestão 1ª e 2ª Coorte (Setembro) 1 Oficinas Pedagógica 1ª e 2ª Coorte (Setembro) CALENDÁRIO DAS FORMAÇÕES 2019 USAID VAMOS LER! Agosto 3 Oficinas Pedagógicas 1ª e 2ª Coorte (Fev, Março, Abril) 3 Oficinas de Conselho da Escola 1ª e 2ª Coorte (Fev, Março, Abril) 3 Oficinas de Gestão 1ª e 2ª Coorte (Maio, Junho, Julho) 3 Oficinas Pedagógicas 1ª e 2ª Coorte (Maio, Junho, Julho) 2 Oficinas de Conselho da Escola 1ª e 2ª Coorte (Maio, Julho) Maio 3 Oficinas de Gestão 1ª e 2ª Coorte (Fev, Março, Abril) Formação Participantes Dias 44 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX II: ANALYZED DOCUMENTS 1. PowerPoint presentations on school coaching. 2. CATI. Information about BE teachers. 10.08.2019. Available at MMEMS database. 3. INDE. Second National Evaluation of 3rd Grade – 2016, May 2017 4. MINEDH. Manual of the Trainer of Literacy. Guidance for Coach Training 5. MMEMS. Training QA_FCR - June 17, 2019 6. MMEMS. Deep Learning and Adaptive Study #1. Final Report (Draft Version). February 2019 7. USAID. Local Education Monitoring Approach (LEMA) Pilot Report – Mozambique (Draft). March 2019 8. Vamos Ler! Performance Monitoring & Evaluation Plan (PMEP). Revised October 13, 2017 9. Vamos Ler! Annual Work Plan a. Annual Work Plan 2018 – 2019. November 29, 2018 b. Annual Work Plan 2017 - 2018. December 22, 2017 10. Vamos Ler! Studies: a. Qualitative Study of Community Knowledge, Attitudes and Practices (KAP). Final Report, October 2017 b. Language Mapping Report. Final Report. October 2017 11. Vamos Ler! Reports: a. FY 18 Annual Report. October de 2017 – September de 2018. November 2018 b. FY 19 Quarterly Progress Report 1 October 2018 - December 2018. January 2019 c. FY 19 Quarterly Progress report 2. January 2019 – March 2019. April 2019 12. Vamos Ler! Scripts and Manuals for Teachers and Coaches: a. Script for the Trainer of Teachers of Bilingual Education. S/D b. Script for the Trainer of Teachers of Bilingual Education. District Training Guidelines, May 7–11, 2018 c. Script for the Trainer of Bilingual Education. N/D d. Teacher’s Manual. L1 and L2. Grades 1 and 2. 2018 e. Literacy Trainer Manual. Guidance for the Coach Training. N/D f. Coaches´ Manual at the School Level. Guide for Pedagogical Support. N/D g. Lesson Plans. Grades 1 and 2. L1 and L2. N/D h. Annex I – Operationalization of Crosscutting Themes at School and in the Classroom. N/D USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 45 13. Vamos Ler! Training Program: a. Training of Coaches and Facilitators. Nampula and Zambézia. January 17–18, 2019. January 23– 24, 2019 b. Teacher Training of Bilingual Education. Nampula and Zambézia. January 9–12, 2019 c. Bilingual Education Teacher training of trainers. Nampula and Zambézia. January 7–8, 2019 d. Grade 1 Teacher Training. Cohort 1. Nampula and Zambézia. January 14–19, 2019 e. Grade 1 Teacher Training. Cohort 2. Nampula and Zambézia. January 21–26, 2019 f. Grade 2 Teacher Training. Cohort 1. Nampula and Zambézia January 14–16, 2019 14. Vamos Ler! Pedagogical Workshops: a. Structure and Theme of Pedagogical Workshops. May 2018 b. Pedagogical Workshops Subjects. N/D c. Activity Completion Report. Nampula and Zambézia. February and March 2019. d. Activity Completion Report. Nampula and Zambézia. February and September 2018. e. Individual reports from various PWs. (Malema, Ilha de Moçambique, Lalaua, Gilé, Gurué, Lugela, Mocuba, Mulevala, Namacurra, Namarroi 15. Vamos Ler! Terms of Reference for the training of bilingual education trainers. Nampula and Zambézia. January 8–12, 2018 16. Vamos Ler! Terms of Reference for the training of bilingual education trainers. Nampula and Zambézia. January 7–26, 2019 17. Vamos Ler! Files: a. Coaching (class observation) b. Supervision of pedagogical workshops (revised) c. Weekly continuous assessment. N/D d. Monthly continuous assessment LI grade 1. N/D 46 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX III: PERSONS CONTACTED # Institution Person and Role Contacts Meeting Date MAPUTO CITY 1 DNFP Director DNFP 07/08/2019 2 Técnico do EB 07/08/2019 3 VAMOS LER! , Responsável do IR1 VL! 06/08/2019 4 Coordenador Sénior de Desenvolvimento Profissional de Formadores e Professores 06/08/2019 5 Coordenador Sénior de Desenvolvimento de Materiais 06/08/2019 6 18/06/2019 NAMPULA PROVINCE NAMPULA CITY 1 DPEDH (NAMPULA) Director Provincial-Adjunto 14/08/2019 2 , Ponto Focal Provincial do EB 25/07/2019 3 , Técnico da Repartição de Formação de Professores 15/08/2019 4 IFP DE NAMPULA Directora 16/08/2019 5 VAMOS LER! Coordenador Provincial de Formação (Leitura) 16/08/2019 MOGOVOLAS 6 SDEJT MOGOVOLAS Directora 09/07/2019 7 Chefe da REG 08/07/2019 8 Técnica de EB 08/07/2019 9 VAMOS LER! , Oficial Distrital VL! 17/07/2019 USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 47 # Institution Person and Role Contacts Meeting Date 10 EPC DE NAMETURI , Director da Escola 09/07/2019 11 Director Adjunto Pedagógico (Subst) 09/07/2019 12 Professor de EB, 1ª classe 09/07/2019 13 , Professora de EB, 2ª classe 09/07/2019 14 , Professora de EB, 2 ª classe 09/07/2019 15 , Professor de EB, 1ª classe 09/07/2019 16 , Professor de EB, 1ª classe 09/07/2019 17 Professor de EB, 1ª classe 09/07/2019 MOMA 18 SDEJT MOMA , Director SDEJT 19/07/2019 19 , Chefe da REG 11/07/2019 20 Técnico Ensino Bilingue 11/07/2019 21 VAMOS LER! Oficial Distrital VL! 12/07/2019 22 SEDE DA ZIP DE PILIVILI , Coordenador da ZIP de Pilivili 16/07/2019 23 Director Adjunto da Escola 16/07/2019 24 SEDE DA ZIP DE MILIGE Coordenador da ZIP de Milige 19/07/2019 25 ESCOLA PRIMÁRIA EDUARDO MONDLANE , Director da Escola 12/07/2019 26 Professor de EB, 1ª classe 12/07/2019 27 EPC DE CAMACULO Director da Escola 17/07/2019 28 Director Adjunto da Escola 17/07/2019 48 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV # Institution Person and Role Contacts Meeting Date 29 Professor de EB, 1ª classe e de Ensino Monolingue (EM), 2ª classe 18/07/2019 30 Professor de EB, 1ª classe e de EM, 4ª classe 18/07/2019 ERATI 31 SDEJT ERATI Director 24/07/2019 32 , Oficial Distrital VL! 22/07/2019 33 Responsável do Ensino Primário, na REG 22/07/2019 34 VAMOS LER! Oficial Distrital do VL! 22/07/2019 35 SEDE DA ZIP 25 DE JUNHO Professor Orientador do EB na ZIP 23/07/2019 36 EP1 THUMAR Professor de EB, 2ª classe 24/07/2019 37 Professor do de EB, 1ª classe 24/07/2019 ZAMBÉZIA PROVINCE QUELIMANE 1 DPEDH Coordenador do Departamento de Direcção Pedagógica 15/08/2019 2 Técnico do Departamento Pedagógico 15/08/2019 3 IFP Formador do IFP 16/08/2019 4 VAMOS LER! Coordenador de Leitura e Escrita do VL! 16/08/2019 MOCUBA 5 SDEJT MOCUBA Director 08/07/2019 6 Ponto focal da REG 08/07/2019 7 Chefe da REG 08/07/2019 8 Técnico de Supervisão, REG 08/07/2019 USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 49 # Institution Person and Role Contacts Meeting Date 9 Técnico de Supervisão, REG 08/07/2019 10 Técnica de Supervisão, REG 08/07/2019 11 Técnico de Supervisão, REG 08/07/2019 12 EPC MURRUÉ Directora EPC Murrue 09/07/2019 13 Directora Adjunta e Professora de EB 09/07/2019 14 Professora de EB 09/07/2019 15 Professora de EB 09/07/2019 ALTO MOLÓCUÉ 16 SDEJT ALTO MOLÓCUÉ Director 11/07/2019 17 Chefe da REG 12/07/2019 18 Técnico de EP 12/07/2019 19 Técnico de Ensino Secundário / Técnico de Supervisão 12/07/2019 20 EPC MICHULANE Director 12/07/2019 21 Professor de EB, 1ª Classe 15/07/2019 22 Professor de EB, 2ª Classe 15/07/2019 23 VAMOS LER! Oficial Distrital do VL! 15/07/2019 GILÉ 24 SDEJT GILÉ Director 19/07/2019 25 Chefe da REG 19/07/2019 26 Coordenador Sector Planificação 19/07/2019 27 Técnica da REG 19/07/2019 29 Técnica da Unidade de Género 19/07/2019 50 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV # Institution Person and Role Contacts Meeting Date 29 Técnica de AEA da REG 19/07/2019 30 Técnico Desporto Escolar e Saúde Escolar 19/07/2019 31 Técnica de Recursos Humanos 19/07/2019 32 Técnico de Produção Escolar 19/07/2019 33 EPC KHAYANE , Director 17/07/2019 34 Professor de EB, 1ª Classe 18/07/2019 35 Professor de EB, 1ª Classe 18/07/2019 36 Professor de EB, 1ª Classe 18/07/2019 37 VAMOS LER! , Oficial Distrital do VL! 16/07/2019 NAMACURRA 38 SDEJT NAMACURRA , Director distrital 22/07/2019 39 Chefe do DEG 22/07/2019 40 Técnico de EB (Ponto Focal) 22/07/2019 41 , Responsável de Género 22/07/2019 42 RAP 22/07/2019 43 Responsável do Ensino Secundário, REG 22/07/2019 44 RAP 22/07/2019 45 CFP DE MACUZE Directora 46 Docente de EB USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 51 # Institution Person and Role Contacts Meeting Date 47 ZIP VURUCA , Director da EPC e Coordenador da ZIP de Vuruca 23/07/2019 48 Director Adjunto, EPC Vuruca 24/07/2019 49 , EPC Vuruca, Sala Anexa, Professor de EB 24/07/2019 50 EPC Nipera, Professor de EB 24/07/2019 51 EPC Nipera, Professor EB 24/07/2019 52 , EPC Romão, Professor de EB 24/07/2019 53 , EP1 Maquita, Professora de EB 24/07/2019 54 EPC NAMACORRONE , Director 23/07/2019 55 , Professor de EB, 1ª classe 23,24 e 25/7/2019 56 Professor de EB, 1ª classe 24/07/2019 57 Professora, 2ª classe 24/07/2019 58 VAMOS LER! , Oficial Distrital do VL! 22 e 25/7/2019 52 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX IV: ANALYTICAL FRAMEWORK DATA, SOURCES, AND METHODS Data and Information Required Data Sources Data Collection Data Analysis 1. What were the desired outcomes of capacity-building activities (ToT, TTDL, and PWs) for acquisition of EGR pedagogical skills, knowledge, and practices? A. Data on expected outcomes of skills, knowledge, and practices as a result of ToT, TTDL, and PWs VL! relevant documentation MINEDH, DNEP/ BE, DNFP, DPEDH, SDEJT Selected teachers, school directors/deputy directors, and ZIP coordinators in selected sites VL! DO and experts Desk study of secondary data (documents) Key informant interviews FGDs with teachers Observation of ToT and TTDL Analyses of qualitative data from document review Data triangulation A. Data on planned and implemented activities for the ToT, TTDL, and PWs B. Perceptions of MINEDH, DPEDH, SDEJT + teachers, school officers + VL! on capacity-building outcomes 2. Are the teachers implementing gender-responsive instruction practices that support BE EGR, in accordance with the desired outcomes established in the training model? A. Plan compliance (lesson schedule and contents) Relevant documents at the school level Class lecturing Document review Class observation PW observation FGDs with teachers Analyses of qualitative data Data triangulation B. Application of gender-responsive practices by teachers in the classroom (or lack of) C. Application of skills to promote student interaction D. Specific technical skills and competencies Reading teaching methods Writing teaching methods Reading and writing evaluation Planning classroom activity Didactic material management 3. What are the cascade training model’s strengths and weaknesses as it is being applied? A. Description of the cascade model (ToT, TTDL, PW, and coaching/supervision) Relevant documentation Document review Analyses of qualitative data B. Description of how model is implemented, including the feedback loops that provide information to USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 53 Data and Information Required Data Sources Data Collection Data Analysis decision makers on the model’s effectiveness Stakeholders: MINEDH, DNEP/ BE, DNFP, DPEDH, SDEJT Selected teachers, principals, and ZIP coordinators in selected sites VL! DO and experts Key informant interviews PW discussions FGDs with teachers Data triangulation C. Perceived strengths and weaknesses by implementers, government officials, school directors, and teachers D. Factors associated with model strrengths and weaknessses (causes and implications) E. Stakeholder suggestions for improvement 4. How do training outcomes vary by province and what factors can explain differences in capacity-building experience in Nampula and Zambézia? A. Comparison of model implementation between Nampula and Zambézia Program documentation FGDs with teachers CATI Analysis of data from Qs 1 and 2 Key informant interviews Comparative analyses of qualitative data Data triangulation Quantitative analysis for teacher profiles and perceptions B. Comparison of specific outcomes (teacher skills and practices) in Nampula and Zambézia C. Factors associated with inputs (e.g., teacher profiles) 54 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX V: CRITERIA FOR DISTRICT, ZIP, AND SCHOOL SELECTION Criteria for district selection 1. Represent differentiated socioeconomic and cultural characteristics: a. Rural/urban b. Coastal/interior c. Different languages (Elomwe and Echuwabo in Zambézia and Emakhuwa in Nampula) 2. Dimension of the district (large geographic area, e.g., in NPL Monapo, Moma, Mogovolas, Erati, Malema) Criteria for ZIP selection 1. Headquarters of ZIP, with large number of schools with cohort I and cohort 2 2. Headquarters of ZIP, far from district headquarters, up to 60 km Criteria for school selection 1. Schools with more than one shift of bilingual grades 1 and 2 2. Random choices between two-shift schools, within 60km of district headquarters June 11, 2019, vs2 USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 55 ANNEX VI: WORK PLAN AND DETAILED SCHEDULE Activity June July August Sept Oct 10-14 17-21 24-28 1-5 8-12 15-19 22-26 29-2 5-9 12-16 19-23 26-30 Planning meeting Presentation MMEMS/USAID 19 Desk review Tool preparation Submission of final work plan 1 Data collection 22 Data analysis FCR 27 Presentation of FCR Draft report 16 Comments/ USAID 21 Final report 31 Dissemination TBD Note: Dates in matrix represent specific product delivery dates. 56 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV ANNEX VII: DETAILED FIELDWORK PLAN DLAS 2 - Trabalho de campo - Observação de aulas + KII (SD+DSD) + FGDs (Professores) + Oficinas Pedagógicas ZIP Província Districtos ZIP Escolas Coorte Classe Língua L1 21 22 23 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 I 1ª Classe Echuwabo I 2ª Classe Echuwabo Namacurra Vuruca EPC Namacarrone II 1ª Classe Echuwabo I 1ª Classe Elomwe I 2ª Classe Elomwe Gilé Khayane EPC Khayane II 1ª Classe Elomwe OP ZIP Síntese I 1ª Classe Emakhuwa I 2ª Classe Emakhuwa Moma Pilivili EP1 Eduardo Mondlane II 1ª Classe Emakhuwa Moma Milige EPC de Camaculo II 1ª Classe Emakhuwa I 1ª Classe Emakhuwa I 2ª Classe Emakhuwa OP ZIP Síntese June July Nampula Mogovolas Moquito EPC de Nameturi Eráti 25 de Junho EP1 de Thumar Zambézia Mocuba Derruba EP1 de Murrué Alto Molócue Namicopo EPC Michulane USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 57 ANNEX VIII: SCHOOL CHARACTERISTICS School, ZIP, and District Physical Facilities Safety No. of Class￾rooms Teachers BE Teachers Students of Observed Classes Total Women Total Share Women Registered (total = H+M) Present (daily average and %) Students present, % of students Namacorrone, Vuruca, Namacurra Walls: clay Roof: zinc Floor: hard-packed dirt floor Desks: trunk + seats Without doors or windows 4 6 (+ 1 half time) 2 (+1 half time) 3 1/333 Grade 1 48 = 32 + 16 33=68.7 79% Grade 2 There is none —2nd cohort EPC Nameturi, Moquito, Mogovolas Block 1 Walls: block and concrete Roof: zinc sheets Floor: concrete Desks: of wood Block 2 and 3 Walls: clay Roof: reed Floor: hard-packed dirt floor Desks: of wood Window grilles and wood door without lock, and a door grille fixed only at the bottom Block 2 and 3: Open windows and no door 12 Total: 9 Grade 1: 4 Grade 2: 5 Total: Grade 1: 3 Grade 2 : 4 Total: 7 Grade 1: 4 Grade 2: 3 Total: 5 Grade 1: 3 Grade 2: 2 Grade 1 75 = 40 + 35 48.4 = 64.4 24.6 = 70.74 Grade 2 52 = 25 + 27 36.6 = 70.51 14.6 = 54.32 EP1 Eduardo Mondlane, Pilivili, Moma Walls: clay Roof: reed Without doors or windows 3 3 0 1 0/1 Grade 1 60 = 36 + 24 54 = 90% 21 = 87.5 33 The only qualified teacher was not teaching BE but grade 2 monolingual teaching. 58 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV School, ZIP, and District Physical Facilities Safety No. of Class￾rooms Teachers BE Teachers Students of Observed Classes Total Women Total Share Women Registered (total = H+M) Present (daily average and %) Students present, % of students Floor: hard-packed dirt floor, with many holes Desks: some trunks and no desks EPC Camaculo, Milige, Moma Main block Walls: block and concrete Roof: zinc sheets Floor: hard-packed dirt floor Desks: of wood Attached room Walls: clay Roof: reed Floor: hard-packed dirt floor Desks: of wood No doors, but with cement grilles in the windows 5 5 0 2 0/2 Grade 1 121=67 + 54 72.6 = 60.05% 27.33 = 50.68% EP1 de Thumar, 25 de Junho, Eráti Walls: clay Roof: grass Floor: hard-packed dirt floor Desks: no desks; the students sit on the floor No doors or windows 4 4 0 3 0/3 Grade 1 – 166 [63 = 35 + 28] x 2 classes34 68=53.9 64% Grade 2 – 94 = 60 + 34 61=64.9 49% 34 In the observed class, there were students of two classes, with the same number of students. One of the grade 1 classes is taught by the SP, who has BE training, but at the days of observation, she was not at school because she had lost her sister. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 59 School, ZIP, and District Physical Facilities Safety No. of Class￾rooms Teachers BE Teachers Students of Observed Classes Total Women Total Share Women Registered (total = H+M) Present (daily average and %) Students present, % of students EPC Murrué, Derruba, Mocuba Walls: clay Roof: grass Floor: hard-packed dirt floor Desks: there are none; the students sit on the floor No doors or windows 4 3 1/3 Grade 1: 37 (17+20) 24,5=66,2 70% Grade 2: 61(30+31) 50=81,9 75,8% EPC Michulane, Namicopo, Alto Molócue Walls: clay Roof: grass and part in rehabilitation with zinc sheets Floor: hard-packed dirt floor Desks: there are none; the students sit on the floor No doors or windows 4 2 0/2 Grade 1: 71 (34+37) 47,6=67,1 83,7% Grade 2: 62 (27+35) 42=67,7 75,2% EPC Khayane, Khayane, Gilé Walls: masonry Roof: zinc Floor: paved in cement Desks: two desks and bench seats; the students sit in the seats or on the floor when there is no space on the seats No doors or windows 4 3 0/3 Grade 1: 98 (44+54) 82,3=84 82,7% USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 61 ANNEX IX: FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Question 1: What were the desired outcomes of VL! capacity-building activities (ToT, TTDL, and PW) for acquisition of EGR pedagogical skills, knowledge, and practices? ToT Provincial trainers should have pedagogical skills, knowledge, and practices to:  Disseminate the BE philosophy and model adopted in Mozambique.  Distinguish and explain the tasks of various actors in teacher training/qualification and accompaniment at the provincial, district, ZIP, and school levels.  Train the district trainers in the contents, organization, and functions of the main training/qualification instruments at provincial, district, ZIP, and school levels.  Train the district trainers in BE procedures and training routines.  Train the district trainers in training methods for teachers, coaches, and supervisors.  Teach how to model and apply reading and writing teaching and learning methods and routines.  Train the district trainers to explore crosscutting themes in the classrooms.  Train the district trainers to use and produce BE teaching and learning materials.  Train the district trainers in district supervisory routines.  Train the district trainers to evaluate students based on standard objectives and defined learning results monthly and quarterly.  Describe PW objectives, organize and implement the PWs, and know the provincial trainer’s role in facilitating them.  Describe and teach pedagogical accompaniment techniques in L1 and L2 classes.  Identify, describe, and perform procedures and monitoring routines for BE implementation.  Describe Rapid Alert System components, role, and operation. TTDL District trainers should have the pedagogical skills, knowledge, and practices to:  Disseminate the BE philosophy and model adopted in Mozambique.  Train teachers in planning grades 1 and 2 L1 reading and writing classes, based on appropriate curriculum maps, content, objectives, methods, and materials.  Train teachers in using the main teaching and learning materials— L1 and L2 teacher’s manual and student book—and producing additional materials. 62 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV  Train teachers in applying teaching methods for grades 1 and 2 L1 speaking, reading, and writing: games/songs, tale/retelling, image reading, role play, song and dance and vocabulary (oral development methods), phonological awareness, phonetics, reading fluency, reading comprehension, knowledge of printed and written material (reading, writing and oral development methods).  Train teachers in using the gradual progression model in class: "I Do, We Do, You Do."  Train teachers in using the L1 alphabet scope and sequence progression pyramid for teaching grades 1 and 2 reading and writing.  Train teachers in addressing crosscutting themes in the classroom.  Train teachers in using continuous student assessment sheets.  Train teachers to identify students with learning disabilities and implement pedagogical remediation routines and activities.  Train teachers in operating the Rapid Alert System.  Train teachers in managing student attendance and punctuality.  Train teachers in managing student behavior in the classroom.  Train teachers in managing student class participation, ensuring inclusiveness (gender, special educational needs, etc.).  Distinguish and explain the tasks of various stakeholders in training/coaching and school monotoring at district, ZIP, and school levels: district trainer, coordinator, and ZIP facilitator and coaches.  Train the ZIP coordinator, facilitator, and coaches in the contents, organization, and functions of the main training, supervision, and monitoring tools at ZIP and school levels: terms of reference for training BE trainers and teachers, trainer’s guide, school-level coach manual.  Train the ZIP coordinator, facilitator, and coach in BE training and supervision procedures and routines.  Train the ZIP coordinator, facilitator, and coach in teacher training methods.  Describe PW objectives, organization, and implementation and know the mentor’s role in facilitating them.  Train the ZIP coordinator, facilitator, and coach in identifying, describing, and performing procedures and monitoring routines for implementing the BE program at school level. PWs The PWs are intended to enable:  The identification of teachers’ teaching difficulties.  The supplementation of district-level training to respond to teachers’ difficulties.  The detailed planning of classes for the next period.  The production of materials needed for classes in the next period.  The mutual aid and exchange of experience among teachers.  The simulation of routines suggested by teachers. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 63 Findings Conclusions Recommendations Question 2: Are the teachers implementing gender-responsive instructional practices that support BE EGR, in accordance with the desired outcomes established in the training model? 2A: Plan compliance F2.A.1. All teachers observed taught what was planned in the curriculum (the topic/letter assigned for the week). F2.A.2. All teachers obeyed the weekly map except one in Nampula.35 F2.A.3. All teachers had lesson plans except one in Nampula,36 who started making lesson plans only after being asked by the research team. But the plans rarely had the necessary quality. F2.A.4. Many teachers in both Nampula and Zambézia reported not having sufficient training to draw up detailed daily plans. F2.A.5. No teacher had the second quarter lesson plan book. F2.A.6. In a case where there was no school in the first two days, a teacher implemented the plan drawn up for Wednesday without the students having done the exercises on Monday and Tuesday.37 F2.A.7. Most teachers do not fulfill the daily plan.38 Time is limited for student numbers and Plan compliance C.1. The teaching program is generally being followed in terms of planned learning content for a given week. However, the weekly plan activities are not always fully or properly implemented. (F2.A.1-3,6,8). C.2. The contents and dynamics of teaching are ambitious for the objective conditions of the places where the classes are taught (space, infrastructure, number and punctuality of students, etc.). (F2.A.7,10-11,13). C.3. Several factors influence plan fulfillment. Positive influences include the use of the mother tongue in teaching, student interest, the environment, student participation, and the presence of teachers during the class period. Negative influences include teaching materials, particularly for grade 2, not arriving on time; the lesson plan book for the current quarter not yet being available; insufficient teacher skills to plan classes; students arriving late; and infrastructure conditions constraining the use of some methodologies. (F2.C.1-3; Plan compliance R.1. Ensure that the lesson plans (activity volume and type) consider class conditions. (C.3) R.2. Guide teachers on what to do in situations when the full weekly plan cannot be met. (C.1,2) R.3. Make quarterly plan manuals available before the start of each quarter. (C.3) R.4. Strengthen teacher training in elaborating/adapting detailed lesson plans. (C.1, C.3) 35 Refers to the same teacher indicated in F2.A.3. 36 Refers to the same teacher indicated in F2.A.2. 37 The weekly plan deals with one new letter each week. It has a logical progression of learning that implies assistance in all classes. 38 The weekly plan unfolds into daily lesson plans. The daily plans show in detail the activities to be performed on each day. 64 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations participatory methodologies. F2.A.8. Unlike in Zambézia, in Nampula crosscutting issues were not addressed in any of the classes observed. F2.A.9. Everywhere observed the teachers were present at the scheduled class start time and they ended class at the scheduled time. Exceptions are two teachers at a school in Nampula, who took breaks off schedule. F2.A.10. In all observed locations the vast majority of students (60–70 percent) were not present at the beginning of the L1 class when it started at 7 a.m. Delays ranged on average between 0.5 and 1 hour, with cases of 2/2.5-hour delays in some schools. There was an exception in a place where the school is right in the center of the village and almost all the students arrived on time. F2.A.11. Overall and on average over the three days, student attendance ranged from 54 to 90 percent of enrolled students. In Zambézia girls had a higher attendance than boys, unlike in Nampula. F2.A.12. Reasons for student absence were distances to schools, cold weather, and illnesses (associated with the cold). In Nampula, student participation in family activities was indicated as a factor in school absenteeism. F2.A.13. Material teaching conditions (classrooms, desks, physical F2.D19-23, F2.A.4-5, F2.A.9- 11, F2.A.13) USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 65 Findings Conclusions Recommendations infrastructure) were very precarious almost everywhere. 2B: Application of gender￾sensitive practices/social inclusion F2.B.1. The teaching materials give visibility to girls and women. However, no particular efforts have been observed to use the full potential of these materials to encourage more gender equality and social inclusion. F2.B.2. In teacher training sessions observed in the two districts, there was no attention to gender issues in teaching content. F2.B.3. It was unclear whether the regular monitoring activities included monitoring of gender mainstreaming in education. The August ToT introduced a gender assessment and inclusion sheet in practice, but in the observed TTDL it was not presented. F2.B.4. Both ToT and TTDL addressed gender issues in student participation. F2.B.5. Class teaching materials addressed crosscutting subjects. F2.B.6. Teachers called/challenged boys and girls in a relatively balanced way. Efforts have been noted to do this. F2.B.7. Some teachers mentioned that they received guidance to pay attention to gender issues but no specific training in how to do this. The ToT and TTDL observation confirmed this point. F2.B.8. There were only three women teachers among the 12 teachers observed in this sample. This very Application of gender-sensitive practices/social inclusion C.4. Teachers have some knowledge about gender and social inclusion and received guidance to promote these considerations. However, there are some skill limitations in practicing social inclusion and gender equality. (F2.B,1,4-7,9-10,12) C.5. The training, and possibly the monitoring, pay little attention to implementing the gender approach (what and how to do). (F2.B.2-3,7) C.6. In the schools observed, the women teachers were a minority of BE teachers, counteracting gender equity. (F2.B.8) C.7. Class time constraints limit individual support for students with learning disabilities. (F2.B.10-11) Application of gender-sensitive practices/social inclusion R.5. Address gender issues systematically in all capacities at the school level and introduce materials and sufficient tools for teachers. (C.4, C.5) R.6. Strengthen teacher training in using the strongest students to help the weaker students, promoting solidarity and enabling teachers to concentrate on those with greater difficulty. (C.7) R.7. Ensure that the implementation monitoring system assesses gender￾sensitive and social inclusion practices. (C.5) 66 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations low proportion also corresponds to the low proportion of women teachers in the places observed. F2.B.9. In almost all the 34 classes observed, teachers made efforts to assist students with learning disabilities. The ToT and TTDL observed called attention to the need for individual work with students with difficulties. F2.B.10. It was not always clear whether teachers had sufficient knowledge and skills to support all children. Classroom time was scarce for this purpose. In the observed trainings, the development of techniques to respond to these situations was limited. F2.B.11. There was no routine implementation of stronger students helping weaker students, contrary to program recommendations. F2.B.12. Most of the classes observed did not have children with visible disabilities. But in Eráti there was an albino child in a classroom; no discrimination was noted, and the child seemed well integrated. 2C: Promotion of classroom interaction Teacher-student interaction: F2.C.1. There was good teacher￾student interaction everywhere. There was a lot of student participation. The children were very comfortable, relaxed, and comfortable with the teachers. F2.C.2. Respondents noted that using the students’ native languages creates a much Promotion of classroom interaction C.8. The teachers successfully and systematically use teaching methodologies that promote student participation in class and achieve a learning-friendly environment. (F2.C.1,5-6,8, F2.D.2-3) C.9. Classroom interaction is centered more on student interaction with teachers Promotion of classroom interaction R.8. Continue using participatory methods to promote student interest and achieve better learning outcomes. (C.8) R.9. Reinforce the use of methods that promote more student cooperation. (C.9) USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 67 Findings Conclusions Recommendations more open and spontaneous environment that supports student participation. F2.C.3. Teachers say there is a big difference in the classroom environment between monolingual and bilingual teaching. On the basis of previous work experiences, the team agrees with this observation. F2.C.4. Both provincial- and district-level training fostered an environment in which learners experienced mutually supportive relationships. However, in the simulations, application of approaches to promote student cooperation was not visible. Interaction among students: F2.C.5. In most of the observations, there was very active interaction between the students and the teachers. F2.C.6. The technique of writing on the back was routinely used. F2.C.7. The peer reading method (2 observations out of 34) and the group work method were little used. Objective39 material conditions did not encourage use of the latter method. F2.C.8. The team observed examples of mutual help among students, with variations among schools. Teachers were not observed to promote mutual help among students, even in cases than on interaction among students. (F2.C.4-5,7-9) 39 Many classrooms are small for the number of students; sometimes they have benches and/or tree trunks that are not easy to move to organize group work. See Annex VIII – School Characteristics. 68 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations where it happened (spontaneous). F2.C.9. There were few examples of teachers seeking to promote student interaction during class. Most of the interaction on class subjects was between students and teachers, not among students themselves. 2D: Pedagogical skills, knowledge, and practices F2.D.1. Almost all teachers know the materials (books, posters, etc.) made available by VL! to BE (1 in 12 teachers did not master the materials well). F2.D.2. All teachers used various methods of teaching orality: songs, image reading, vocabulary, phonetic awareness (up/ down play), clapping in the syllable division. F2.D.3. All teachers understand and have some mastery in the routine, “I do, we do….” (sometimes they don’t complete the cycle [don’t do what you do]). F2.D.4. Teachers had a good command of the interconnection of language skills: listening, speaking, reading, and writing. F2.D.5. Continuous assessment routines are applied in class. But the use of continuous assessment forms was not observed; teachers said they had no copies. F2.D.6. Teachers have some limitations in identifying pedagogical remediation activities for students with learning disabilities.40 Pedagogical skills, knowledge, and practices C.10. Most of the observed teachers have sufficient pedagogical skills, knowledge, and practices for BE and know how to do their work. However, this does not guarantee that these skills will be used systematically in the absence of observation. (F2.D.1-5,9-11,14,16) C.11. Teachers have limitations in planning lessons in phonetics (sound of the letters) and in vocabulary and numeracy teaching approaches. (F2.A.4,6, F2.D.7-8,13, 15) C.12. There are limitations in the use of some methodologies by some teachers—namely, peer reading, silent reading, tale and retelling, group work, role play, and others. (F2.C.6, F2.D.6-7) C.13. The student assessment process does not appear to be standardized, in particular the daily and weekly assessment. (F2.D.5,17-18) C.14. The materials available are not sufficient for the enrolled students; they Pedagogical skills, knowledge, and practices R.10. Increase time and perform more teacher training practices on identified weak elements (lesson planning; phonetics; writing; numeracy; vocabulary; and methodologies, including pedagogical remediation methodologies). [C.11- 13,15] R.11. Increase teacher training time by revisiting the approach of excluding teachers from the previous year’s cohort from the training of a given year. (C.11-13,15) R.12. Strengthen teacher training in L2 teaching methodologies, including how to use physical education classes to facilitate L2 teaching. (C.15) R.13. Clarify the student assessment system to standardize its implementation. (C.13) 40 Continuous assessment routines are used to identify students with learning disabilities. However, teachers are not always able to identify the most appropriate pedagogical remediation activities to help some students with difficulties. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 69 Findings Conclusions Recommendations F2.D.7. TTDL observation showed that teachers still have many doubts in the pedagogical area. The training served to resolve these doubts. F2.D.8. Some teachers reported that they need more L1 training (writing, phonics, etc.).41 F2.D.9. All teachers manage attendance. F2.D.10. Teacher ability to manage classes varies greatly. Three observed teachers had difficulty managing student behavior in the classroom, not always because of class size. Class size varied widely (from 31 to 92 students per class). F2.D.11. All teachers observed had L1 as their mother tongue. F2.D.12. Several respondents expressed concerns about the transition from L1 to L2, stating that teachers do not have sufficient skills in this area. Physical education classes are not always being properly explored to help introduce L2. F2.D.13. In two districts of Zambézia, interviews and a focal group of teachers indicated difficulty in teaching mathematics in L1 due to the lack of specific vocabulary. Reading practice F2.D.14. In 25 of 34 observations, reading techniques were applied correctly. F2.D.15. In most of the classes observed, some cases of difficulties in phonetics were observed (different sounds for the same letter, phonetic variations among the various L1 dialects). Writing practice arrived late and in phases, which hindered the teachers’ work. (F2.D.20- 21) C.15. The investment (teacher training) made so far is not enough to ensure a smooth transition from L1 to L2. (F2.D.12) 41 Although all the teachers observed spoke L1, none had learned to write and read in that language. 70 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations F2.D.16. In general, teachers use the prescribed techniques (gestures, back, floor, board, notebook). Reading and writing assessment F2.D.17. Teachers do not correct every student’s work daily, nor is this required by the program. F2.D.18. Situations were found where incorrect student work was rated as correct. The reason for this was not explored, given the nonintrusive observation method. Lesson planning F2.D.19. Stakeholder positions vary on the need to have detailed daily plans. Materials management F2.D.20. The basic grade 1 materials (student reading books and notebooks) existed in almost every classroom observed, although in some classes there were not enough for the number of students. F2.D.21. There were significant delays, and in some cases grade 2 student books and notebooks were not available (the observation was made midway through the second trimester). F2.D.22. In most cases, the students’ books were kept in the school at night. However, in one school, students took the books home. At a meeting with teachers of one ZIP, the teachers complained about the time it takes to deliver and collect books, which is linked to USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 71 Findings Conclusions Recommendations attendance calls and impaired teaching.42 F2.D.23. In some schools, there are no safe conditions for storing books so the books are kept at teachers’ houses. Question 3: What are the cascade training model’s strengths and weaknesses as it is being applied? 3A: Description of the Cascade Training Model  The cascade training model has three components: ToT, TTDL, and PWs.  ToT is held in the provincial capitals in three phases: a six-day training in January, a three-day training in May, and a three-day training in August. The January training is divided into two stages: a two-day training of TTI teachers and a four-day training of district trainers by the TTI teachers. The May and August trainings lack this first stage. All training is guided by experts from Maputo, provincial technicians, and TTI trainers.  Teacher training takes place in the week after ToT at district headquarters. The six-day January training covers all grade 1 teachers and cohort 1 grade 2 teachers. The three-day May training covers only grade 1 teachers from both cohorts. The three-day August training covers cohort II grade 1 teachers. These trainings are guided by trained trainers at the provincial level with the support of national-level DPEDH and TTI experts.  The half-day educational workshops are held at the ZIP level seven times a year on Saturdays. They are facilitated by trained facilitators and participating BE teachers, head teachers (coaches), and the ZIP coordinator.  The model includes monthly in-school supervision and monitoring at the ZIP level by ZIP coordinators and coaches and monthly in-school follow-ups by the SP for each teacher. This follow￾up is complemented by quarterly supervisory visits by SDEJT, including the VL! DO. Findings Conclusions Recommendations 3B: Model implementation F3.B.1. Activity frequency, duration, and participants have changed since the model’s original design. F3.B.2. Teacher training time has been reduced since the original design. F3.B.3. Experts from Maputo (VL! and MINEDH) provided answers to many questions raised during ToT, Model implementation C.16. There is no shared common view on how the model works (activities, frequency, duration, participants, methodologies) due to adaptations made during the Model implementation R.14. Analyze training activity implementation from the first two years, harmonize the model, and carry out clarification actions for all stakeholders. (C.16- 22) 42 The program recommends that these materials remain in schools, to be distributed at the beginning of class and returned at the end of class. 72 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations particularly on pedagogical and linguistic issues. F3.B.4. The division of learners by district and ZIP in ToT and TTDL, respectively, affords the the greatest participation of learners. F3.B.5. In most districts, ToT support materials were not available at the beginning of training. In August, some ToT or TTDL participants did not bring the materials they had received in previous training. F3.B.6. About half of the interviewees criticized trainee logistical support, saying that it does not cover accommodation and food expenses in the district and provincial capitals. F3.B.7. There is widespread positive appreciation of trainer capacity at both provincial and district levels. However, one district reported a reduction in quality at the district level. F3.B.8. It was reported that ToT trainer selection was better this year; the process started at the school and ZIP levels. However, in one district it was reported that there are teachers who “buy” vacancies. F3.B.9. In one district of Zambézia, the local CFP TTI trainers have not been involved in teacher training. F3.B.10. Most respondents (teachers, SPs, district directors, and VL! staff) reported that BE teachers’ training time does not meet their needs. F3.B.11. The feedback mechanism is centered on PWs whose reports are reviewed at the district level. Provincial-level reports summarize the district reports and are then discussed with the DPEDH and sent to VL! at the central level. The reports follow a standardized model and have a very strong focus on workshop management and administration. F3.B.12. Interviews at the national level (MINEDH and VL!) indicated that the synthesized PW reports are used to inform planning of subsequent trainings. F3.B.13. PWs are not working as planned. Teacher participation in the 2019 implementation. (F2.D.19, F3.B.1) C.17. There is a widespread perception that the current training time for teachers is not sufficient for BE requirements, especially in the May and August sessions. (F3.B.1,2,10, F3.C.6-10, F3.D.7,12) C.18. Training quality generally responds to stakeholder expectations, which take into account limitations of time and resources. (F3.B.4,7-8, 17, F3.D.4) C.19. PWs are failing to fully fulfill their role in teacher improvement, capacity building, and supervision. (F3.B.13- 16) C.20. External support at central and provincial levels has been fundamental for the quality of the observed training. (F3.B.3) C.21. TTI/CFP participation in BE implementation has not been sufficient, particularly in supervising and monitoring teachers. (F3.B.9,18) C.22. Logistical support to trainees (ToT, TTDL) is insufficient (F3.B.5-6) R.15. Strengthen the PWs’ role in training, supervising, and monitoring teachers and in the training feedback process. (C.19, 21) R.16. Revisit logistical support at various levels to better meet training and trainer needs. (C.22) R.17. Maintain external experts’ involvement in training technical support throughout the program to improve the quality of local human resources. (C.20) R.18. Revisit the role of TTI/CFP in BE supervision and monitoring to improve monitoring quality. (C.21) USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 73 Findings Conclusions Recommendations PW dropped significantly and the work agenda is not fulfilled due to the brevity of sessions (four hours). F3.B.14. All interviews and PW reports cite the lack of subsidies for PW participation as a constraint to teacher participation. F3.B.15. The PW reports are not shared with the schools. F3.B.16. The teachers clarify their duties by calling on people they feel can support them. They don’t always get the coaches’ response. F3.B.17. Training contents are appropriate and useful for BE purposes. F3.B.18. It was often reported that the TTI teacher trainers did not conduct teacher supervision and monitoring as planned due to lack of funds. 3C: Model strengths and weaknesses The respondents reported the following strengths: F3.C.1.The model benefits a large group of teachers. F3.C.2.The model enables access to quality specialists at lower costs. F3.C.3.The model enables exchange of information and experiences among levels and regions. F3.C.4.The model builds district capacity for future support to BE teachers; they have trainers available at any time. Respondents reported the following weaknesses: F3.C.5.The model requires local teacher monitoring by the school directors (coaches) and ZIP coordinators. This monitoring is limited because of coach quality and time constraints associated with distance. F3.C.6.TTDL time was insufficient, particularly in phases 2 and 3, which have only three training days. F3.C.7.Some respondents reported that the daily schedule is very long and tiring. F3.C.8.The program offered few practices to each participant. F3.C.9.Cohort I grade 2 teachers received only three days of training in 2019, which is considered insufficient. F3.C.10. The program does not sufficiently address vocabulary to teach mathematics. Model strengths and weaknesses C.23. The model presents a good cost-benefit ratio, reaching more teachers at lower costs. However, there are risks that training quality may decline as it expands. (F3.C.1-4, F3.C.11) C.24. The PWs, although they are central to monitoring and feedback of teacher training, are the model’s weakest element because of their brevity and low teacher and coach participation. (F3.B.11.13, F3.D.10, F4.A.3) C.25. Coaches’ preparation level is limited to the functions they have to perform. (F3.B.16, F3.C.5, F3.D.14) Model strengths and weaknesses R.19. Maintain the cascade training model but devote more time to teacher training and strengthening PW components, pedagogical support from ZIP to schools, and in-school monitoring. (C.17,19,23 - 26)R.20. R.20. Revisit logistical conditions for PW teacher participation. (C.24) 74 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations F3.C.11. There is a risk that trainer quality will decline as they move from provincial to district level (reported by a small number of respondents). 3D: Factors associated with model strengths and weaknesses According to respondents and stakeholders contacted, the strengths and weaknesses are as follows: Strengths F3.C.12. The cascade model is familiar to the respondents. F3.C.13. TTI/CFP involvement favors appropriation and creates a local resource base to support BE. F3.C.14. VL! support and education specialists contributed to increasing training quality, particularly in BE technical aspects. F3.C.15. The training in three phases allows for a less loaded daily schedule and integration of new elements as implementation progresses. F3.C.16. The close involvement of education authorities at national, provincial, and district levels. F3.C.17. The positive perception of BE by communities, teachers, and authorities. F3.D.1. The interest of BE teachers. Weaknesses F3.C.18. Large number of schools covered by the program. F3.C.19. Limited human resources available for capacity building and supervision. F3.C.20. The suspension of subsidies for teachers to participate in the PWs. F3.C.21. Limited availability of teachers’ time to participate in training. F3.C.22. The weak level of implementation of the supervision system and education monitoring. F3.C.23. The dispersion and distances between schools of the same ZIP. F3.C.24. Competence level of the coaches, who have the same training as the teachers they should supervise. Factors associated with model strengths and weaknesses C.26. The cascade model corresponds to the teacher training model used by MINEDH, which makes its acceptance easier. (F3.D.1) C.27. External support from either VL! or MINEDH and from TTI/CFP is fundamental for achieving the desired results. (F3.B.3, F3.D.2-3, 5) C.28. The positive image of BE’s potential to promote EGR encourages stakeholder involvement and participation in the training. (F3.D.6-7) C.29. The limited human capacity to develop the program with the current amplitude contributes negatively to the results. (F3.D.8-9.11) 3E: Suggestions for improvement Respondents and stakeholders made the following suggestions to improve the training model: Suggestions for improvement C.30. Three main areas were identified: greater SDEJT involvement, PW decentralization, and USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 75 Findings Conclusions Recommendations F3.E.1. Include district directors in training to encourage greater follow-up at local level. F3.E.2. Increase the number of SDEJT technicians to participate in ToT. F3.E.3. Increase the number of teachers to reduce the number of students per class. F3.E.4. Ensure complete quality training for grade 3 teachers. F3.E.5. Perform six days of training in each phase. F3.E.6. Resume PW subsidies. F3.E.7. In each ZIP, organize PWs by groups of schools that are closest to each other. F3.E.8. Increase the number of training days for coaches. One day is not enough. F3.E.9. Sensitize the schools to the production/reproduction of BE materials using own resources, including direct support to schools. F3.E.10. Introduce certificates of participation for teachers and trainers as a way to reinforce their motivation. F3.E.11. Offer L1 courses to teachers who teach this language (oral and written). F3.E.12. Prepare and make available L1 dictionaries. strengthening the training of coaches. (F3.E.1-12) Question 4: How do training outcomes vary by province and what factors can explain differences in capacity-building experience in Nampula and Zambézia? Findings Conclusions Recommendations 4A: Comparison of model implementation between Nampula and Zambézia F4.A.1. Activity type, format, and duration and training participant selection are the same in Zambézia and Nampula. F4.A.2. The provinces were similar in availability of books for training and teaching. F4.A.3. There are some indications that teacher PW participation is Comparison of model implementation between Nampula and Zambézia C.31. No major differences were found in training model implementation. (F4.A.I-4) C.32. More teachers with pedagogical limitations were Comparison of model implementation between Nampula and Zambézia R.21. Compare this study’s findings with the EGRA learning outcomes and further analyze any factors associated with significant differences in outcomes. (C32-35) 76 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Findings Conclusions Recommendations lower in Zambézia than in Nampula. F4.A.4. Both provinces demonstrated some tensions between VL! and district service technicians.43 found in Nampula than in Zambézia, but it is not possible to generalize these observations due to the small sample size. (F4.B.I-2.4) C.33. There are no noticeable differences in training process inputs. (F4.C.I-3) C.34. No factors associated with the few differences in training results were found. (F4.B.I-4) 4B: Comparison of model implementation between Nampula and Zambézia F4.B.1. Two of six teachers observed in Nampula (both from cohort II) had difficulty teaching reading. There were no such cases in Zambézia. F4.B.2. Three of six teachers observed in Nampula showed difficulties in controlling the class. In Zambézia there were no such difficulties.44 F4.B.3. No differences were found in teachers’ ability to manage gender equality in class. F4.B.4. In Nampula, unlike in Zambézia, the classrooms did not have materials with letters and fixed posters on the walls. 4C: Associated factors F4.C.1. Among those observed, there was a higher proportion of female teachers in Nampula (2/6) than in Zambézia (1/6).45 F4.C.2. There were no considerable differences in teachers’ age groups. F4.C.3. There were no notable differences in school infrastructure conditions. 43 In three districts there was some tension that appeared related to program appropriation (decision makers) and the availability of resources. 44 The teachers already mentioned above. 45 The two female teachers are from the same school in Mogovolas, where of a total of seven teachers, five are women. In the other schools there were no female teachers teaching BE, except for Mocuba, where there was a female teacher. There was a female teacher trained in Namacurra, but she wasn’t teaching BE. See Annex No. VIII – School Characteristics. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 77 ANNEX X: INTERVIEW SCRIPTS AND FOCUS GROUP DISCUSSIONS NATIONAL/CENTRAL LEVEL Introduction Eurosis was contracted by MMEMS, a mechanism created by USAID to monitor and evaluate the projects that it finances, to conduct DLAS 2, a qualitative study of the training of teachers under VL!. DLAS 2 aims to evaluate the efficacy of the cascade training model used by VL!, to train teachers in bilingual education and generate information that will improve this training. DLAS 2 will take place between June and September of 2019 in the two provinces where VL! is being implemented (Nampula and Zambézia) and in Maputo. The following scripts will be used for interviews at the national level:  Interview scripts for MINEDH  Interview scripts for VL! Note: Put the start and end date of each interview, meeting, etc., as well as the names of the interviewers/observers. Interview Script for MINEDH Interview with Direcção Nacional de Formação de Professores (DNFP) 1. What has been the contribution of VL! to the DNFP agenda? 2. What is your overall impression of the level, quality, and results of implementation of cascade bilingual teacher training supported by VL!? 3. More specifically, how do you assess the level of preparedness of those who train teachers of bilingual education at provincial level (ToT) and those who train at district level (TTDL) under the VL! program? 4. What strengths and weaknesses can be identified in terms of competencies of provincial-trained trainers? 5. What is the impact of training of trainers at the province level supported by VL! (considering that some teachers had or had not received training in this area in the initial courses)? 6. What information do you have about the impact of the training supported by VL! on the skills and outcomes of teachers’ work in class? 7. How do you evaluate the VL! cascade training model as it is being implemented? What are its strengths and weaknesses? (You can talk about the level of competence of the trainers, the competencies developed in the teachers, the nature of the themes/topics addressed or the competencies developed, the duration of the trainings, the efficiency and efficacy of the model and if it is producing the expected results, etc.) 78 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 8. How do you get information about the implementation of training (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 9. What recommendations do you make to VL! in relation to the training model adopted (e.g., in terms of aspects or training practices you consider can be strengthened or improved, and how)? 10. What have been, in general, the challenges of teacher training for bilingual education? Is VL! responding to these challenges? What could be done better? 11. Instead of the cascade training model, is there any other form of training or professional development for bilingual education teachers that you believe that could be more efficient and effective? If yes, say which and why. If not, justify why. 12. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects you consider relevant to share with us. Interview with Direcção Nacional de Ensino Primário (DNEP) Department of Bilingual Education 1. What has been VL!’s contribution to the bilingual education agenda in Mozambique? 2. How has the collaboration with VL been, in the implementation of bilingual education in Nampula and Zambézia provinces? 3. How do you evaluate VL! results in terms of student learning in the schools where the program is being implemented? What evidence do you have to illustrate your assessment? 4. How do you assess the level of preparedness of bilingual teacher trainers at the different levels of the cascade? What strengths and weaknesses can you identify in terms of skills of trainers at the different levels? 5. How do you evaluate the impact of training of trainers at provincial level and of teachers at district level? 6. How do you evaluate the level of preparation of teachers for the implementation of bilingual education? What strengths and weaknesses can you identify in terms of teachers’ skills to teach children in bilingual education? 7. How do you evaluate the VL! cascade training model? What are its strengths and weaknesses? (You can talk about the level of competence of the trainers, competencies developed in the teachers, nature of the themes/topics addressed or developed competencies, duration of the training, efficiency and efficacy of the model and if it is producing the expected results, etc.) 8. How do you get information on training implementation (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 9. What recommendations do you make to VL! in relation to the training model adopted (e.g., in terms of aspects or training practices that you consider can be strengthened or improved and how)? 10. Instead of the cascade model, is there any other form of training or professional development of bilingual teachers that you consider could be more efficient and effective? If so, say which and why. If not, justify why. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 79 11. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects that you consider relevant to share with us. Interview Script for Vamos Ler! Interview with head of intermediate result 1 (IR1) 1. What is the teacher education approach used by VL!? What specific activities do they perform at central, provincial, district, and local levels? 2. How does VL! cascade training model work? 3. How do you evaluate up to now the impact of the VL! cascade training model? (You can talk about the level of competence of the trainers, the level of competence of the teachers, the nature of the themes /topics covered, and their impact on the development of teacher competencies, etc.) 4. In your opinion, what are the main challenges in implementing the cascade training model? (You can speak from the point of view of the organization, logistics, technique, or training itself.) How are you responding to these challenges/how could they be tackled? 5. What mechanisms and processes do you use to gather information on the implementation of the various teacher training activities (training, coaching) (from classroom level through ZIP, district, etc., up to national level)? Is the information you obtain used to make the necessary adjustments to improve the quality of training? Can you give examples? 6. What are the other entities involved in implementing VL!? What role does each of these entities play? 7. How does collaboration with other entities/partners work for the implementation of VL!? 8. Do you consider that the time devoted to training at the three levels is adequate to ensure preparation and development of teachers? Why? What would be necessary to achieve better results? 9. How do you assess the competence, the quality, and the frequency of the support provided by facilitators and coaches? 10. In your opinion, what is needed to facilitate the strengthening of the skills of teachers, facilitators, and coaches? 11. How have the issues related to absenteeism level of teachers and coaches in Pedagogical Workshops been addressed? 12. Instead of the cascade training model, is there any other form of training or professional development of teacher’s bilingual education that you think could be more efficient and effective? If yes, say which and why. If not, justify why. 13. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects you consider relevant to share with us. 80 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Interview with those responsible for training at national level 1. How do you evaluate up to now the impact of the VL! cascade training model? (You can talk about the level of competence of the trainers, competencies developed in the teachers, the nature of the themes/topics covered, and their impact on the development of teacher competencies, etc.) 2. How do you get information about the implementation of training (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 3. In your opinion, what are the main challenges in implementing of the cascade training model? (You can speak from the point of view of the organization, logistics, technique, or training itself). How are or could these challenges be addressed? 4. Do you consider that the time devoted to training at the three levels is adequate to ensure the preparation and development of the teachers? Why? 5. How do you evaluate the support provided by facilitators and coaches to the teachers? 6. How have the issues related to the level of absenteeism of teachers and coaches in Pedagogical Workshops been addressed? 7. In your opinion, what is necessary to do to facilitate the strengthening of skills of teachers, facilitators, and coaches? 8. Other than what you have said before, what other suggestions do you have to improve the skills of the teachers you work with? 9. Instead of the cascade training model, is there any other form of training or professional development of teacher bilingual education which you think could be more efficient and effective? If yes, say which and why. If not, justify why. 10. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects you consider relevant to share with us. PROVINCIAL LEVEL The following scripts will be used for interviews at provincial level:  Interview script for DPEDH  Interview script for TTI  Interview script for VL! Note: Put the start and end date of each interview, meeting, etc., as well as the names of the interviewers/observers. Interview Script for DPEDH Interview with director/deputy director 11. Can you explain which teacher training model is used by VL! to empower the teachers of bilingual education? (How do you train at various levels, including the course lengths?) USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 81 12. What are the differences between the way training is done under VL! and the training of bilingual education, in places where there is no VL!? 13. Can you roughly give the percentage of teachers teaching in EP1 who received bilingual teaching during initial training? 14. In your opinion, what is the impact of the training model being implemented under VL! for bilingual education? Can you give evidence/examples? What factors influenced this impact? How do you assess the level of preparedness of the trainers of teachers of bilingual teaching at the different levels of the cascade? What strengths and weaknesses can you identify in terms of skills of trainers at different levels? 15. How do you get information about the implementation of training (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 16. How do you assess the teachers’ level of preparation for the implementation of bilingual education? What strengths and weaknesses can you identify in terms of teachers’ skills to teach in bilingual education? 17. What recommendations do you make to the program to improve teachers training in bilingual education, in terms of skills, abilities, and knowledge (for example, in terms of aspects or training practices which you consider that can be strengthened or improved, and how)? 18. Instead of the cascade training model, is there any other form of training or professional development of teacher bilingual education that you think could be more efficient and effective? If yes, say which and why. If not, justify why. 19. Is there a working group on bilingual education at provincial level? Who constitutes it? How often have you met and for what purposes? Have you been able to promote the coordination of bilingual education between the various parties? Can you give evidence of the impact/results of this group’s work? 20. Looking to the future, after the end of VL!, how sustainable is the work done in the districts where VL! is supporting the implementation of bilingual education? What do you think is necessary to improve so that bilingual education is maintained and develops to a good level in these districts? 21. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects you consider relevant to share with us. Interview with pedagogical department/BE technician and focal point 1. What has the collaboration been between your sector and VL!? 2. How is the coordination of training at different levels (provincial, district, and ZIP) under VL! done? Do you think the established coordination mechanisms work as expected? Why? 3. What teacher trainings were done this year? How many and who were the participants and what is the program and the duration of the courses? 4. How do you evaluate the impact of VL! cascade training model in bilingual education? Can you give examples/evidence of this impact? What factors contributed to this? 82 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 5. How do you get information on training implementation (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 6. How do you assess the level of preparation of the teacher trainers of bilingual education in the different levels of the cascade? What strengths and weaknesses can you identify in terms of trainers’ skills in different levels? 7. How do you assess the level of preparation of the teachers for the implementation of bilingual teaching? What strengths and weaknesses can you identify in terms of teachers’ skills to teach in bilingual education? Do you consider that the time devoted to training is sufficient to prepare the trainers, teachers, and coaches for the implementation of bilingual education? Why? 8. What is your appreciation about the organization and implementation of Pedagogical Workshops under VL!? Are they reaching their intended ends? Can you give concrete examples? How do you evaluate the competence of the Pedagogical Workshop supervisors to perform the duties assigned to them? What are the positives and negatives of the Pedagogical Workshops carried out under the VL!? How could the work and impact of the Pedagogical Workshops be improved? 9. What is your appreciation on how ZIP is supporting teachers of bilingual education in their area of influence? Can you give examples? What are the positive and negative aspects? Is there enough competence at ZIP to monitor bilingual education in schools? How do schools present their teaching difficulties to ZIP and what support have they received? What could be done to improve the monitoring activity of bilingual education in schools and by ZIP? 10. How is the monitoring of teachers by the schools’ coaches (SP / DSP / cycle coordinators, other) being done? What is the frequency and content of these sessions of orientation? What is your appreciation of this coaching? Do coaches in schools have sufficient skills for the role they should perform? (What are the necessary skills and those that are insufficient?) How could it improve? 11. What are the challenges faced in implementing the current training model adopted in the VL! program? How are these challenges being or should they be addressed? 12. What recommendations do you make for VL! in relation to the training model adopted (e.g., in terms of aspects or training practices that can be strengthened or improved, and how)? 13. Is there a working group on bilingual teaching at province level? Who constitutes it? How often have you met and for what purposes? Have you been able to promote the coordination of bilingual education between the various parties? Can you give evidence of the impact/results of this group’s work? 14. Looking to the future, after the end of VL! how sustainable is the work done in the districts where VL! is supporting the implementation of bilingual education? What do you think needs to be improved so that bilingual education is maintained and develops to a good level in these districts? 15. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects you consider relevant to share with us. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 83 Interview Script for TTI Interview with TTI representative 1. In the current teachers’ initial training program for primary school, what has been the training that the learners receive that enable them to implement bilingual education (subjects, content, etc.)? 2. In the teachers’ initial training program for primary school, which will start now in July, which subjects, contents, methodologies exist to prepare the teachers for bilingual education? 3. What is the role of the TTI in training of teachers of bilingual education under the VL! program? 4. What is your appreciation of the cascade training model used under VL!? Which results have been achieved and not achieved? Why (factors)? Could the TTI have another training program? 5. What suggestions do you make to improve the training under VL!? 6. How do you get information about the implementation of training (training, coaching)? Is that information used to make the necessary adjustments to improve the quality of training? Can you give examples? 7. Have there been lessons from the VL! experience that are being used in TTI? Can you give examples of these lessons and changes? Interview Script with Vamos Ler! Interview with those responsible for training at provincial level 1. What is your role in the implementation of VL!, in particular in relation to training? 2. How do you assess the level of preparedness of the teacher training of bilingual education at the different levels of the cascade? What strengths and weaknesses can you identify in terms of skills of trainers at different levels? 3. How do you assess the level of preparedness of the teachers for the implementation of bilingual education? What strengths and weaknesses can you identify in terms of skills of teachers to teach bilingual education? 4. Do you consider that the level of training and experience of coaches and facilitators is adequate to provide the necessary support to teachers? 5. Do you consider that the time devoted to training is sufficient to prepare trainers, teachers, and coaches for the implementation of bilingual education? Why? 6. What is your appreciation about the organization and implementation of Pedagogical Workshops under VL!? Are they reaching their expected ends? Can you give concrete examples? How do you evaluate the competence of the Pedagogical Workshop supervisors to perform the duties assigned to them? What are the positive and negative aspects of the Pedagogical Workshops carried out under VL!? How could the work and impact of the Pedagogical Workshops be improved? 7. What is your appreciation of how ZIP is supporting bilingual education teachers in their area of influence? Can you give examples? What are the positive and negative aspects? Is there enough competence at ZIP to track BE in schools? How do schools present their teaching difficulties to ZIP 84 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV and what support have they received? What could be done to improve the monitoring activity of BE in schools by the ZIP? 8. How is the monitoring of teachers by the coaches in schools (SP/DSP/ Cycle Coordinator, other) being done? What are the frequency and content of these orientation sessions? What is your appreciation of the coaching? Do mentors in schools have sufficient skills for the role they should perform? (What are the necessary skills and those that are insufficient?) How could it be improved? 9. How do you assess the impact of the cascade training model of VL! within bilingual teaching? What positive and negative aspects can you identify? Can you give examples/evidence? 10. How do you get information on training implementation (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 11. What are the challenges faced in implementing the current training model adopted in the VL! program? How are these challenges being or should they be addressed? 12. We are at the end of our conversation. But before we finish, we would like to leave some space to address any other aspects that you consider relevant to share with us. DISTRICT LEVEL The following scripts will be used for interviews at the district level:  Interview scripts for schools  Interview scripts for SDEJT  Interview script for the VL! DO Note: Put the start and end date of each interview, meeting, etc., as well as the names of the interviewers/observers. Interview Scripts for schools Interview script for FGD with the teachers 1. What support have you received to implement bilingual education? Who has given this support? How do you evaluate this support? • Do you think that, at school level, you have had the necessary didactic and pedagogical support to teach your classes successfully? (You can speak about the availability and skills of coaches, regularity of support actions, etc.) • Do you think that, at ZIP level, you have had the didactic, pedagogical support necessary to teach your classes successfully? (You can speak about the availability and skills of coaches, regularity of support actions, etc.) 2. What assessment do you do in relation to your training process to teach in bilingual education under VL!? (You can talk about the length of training, content addressed, adopted methods, skills of the facilitators, logistics aspects, etc.) 3. What are the challenges they face on a daily basis? (You can speak in terms of required skills, availability and capacity of materials use, availability and quality of follow-up, etc.) USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 85 4. How do you give feedback on training implementation (training, coaching)? Is this information used by higher bodies to make the necessary adjustments to improve the quality of training? Can you give examples? 5. Do you feel that you are prepared enough to implement the VL!’s bilingual education approach? Justify your self-assessment. 6. Do you feel that students are adapting well to the bilingual education model? Why? 7. What recommendations for improvement can you make about the training program and monitoring of VL!? (You can talk about organizational, logistical, technical aspects, etc.) 8. Before we finish, we would like to leave space to address any other aspects that you consider relevant to share with us. Interview script for in-depth teacher interview 1. How is the training received through VL! contributing to being able to teach in bilingual education? What are the positive and negative aspects of this training? 2. Was it easy to assimilate the subject of training? Explain. What were the easiest and most difficult teaching subjects and methodologies to learn? Why? 3. Does the training and follow-up you received/receive help you pay special attention to the needs of girls and children with special needs? Give examples. Explain why. 4. Do you notice any differences between girls’ and boys’ participation in the classroom? What explains this difference (or lack thereof)? 5. What are the main difficulties girls and boys face in bilingual teaching? 6. Are you happy with the support provided by the coaches in your school? Explain. 7. Are you happy with how facilitators have facilitated Pedagogical Workshops? 8. Do you have available all the necessary material to do your work? 9. What could be improved in teacher training and in the support that you receive for better implementation of the approach of bilingual education under VL!? Interview script for school directors/deputy directors 1. How does the process of monitoring teachers of bilingual education work in your school? What is going well? What needs improvement? Why? 2. In your opinion, are you sufficiently prepared to accompany and provide the necessary didactic, pedagogical support to teachers in implementing this education model? Can you justify your self￾assessment? 3. Can you tell us a little more about the monitoring and support you have given to bilingual education teachers? (You may consider aspects such as your skills, availability, regularity of your actions, etc.) 4. As a coach, what are the biggest challenges that you have faced in fulfilling your role? What has been done or thought should be done to help you overcome these challenges? 86 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 5. In your opinion, are teachers sufficiently prepared to implement this education model? Can you justify your assessment? What are the greatest strengths of teacher performance and the weakest points? 6. What are the main challenges faced by teachers? What has been done or thought should be done to help teachers overcome these challenges? 7. How do you give feedback on training implementation (training, coaching)? Is this information used by higher bodies to make the necessary adjustments to improve the quality of training? Can you give examples? Interview script for ZIP head school directors 1. What is the role of ZIP school headquarters compared with other schools with regards to bilingual education? 2. What support does your school provide to teachers from other schools? How is this support done? What are the difficulties faced and the successes achieved? 3. How have the Pedagogical Workshops of Literacy L1 been organized? Have you received support from the district? From the province? (Can you explain who/role?) How do you organize the Pedagogical Workshop agenda? How do you know the specific difficulties of teachers? In the Pedagogical Workshops, have you devoted time to strengthening teachers’ capacity in the areas they identify as weaker? 4. What recommendations for improvement do you make regarding VL! training and follow-up? (You can talk about organizational, logistical, technical aspects, etc.). 5. Before we finish, we would like to leave some space for you to address any other aspects not dealt with in our conversation that you consider relevant to share with us. Interview scripts for SDEJT Interview script for the director/deputy director 1. What is the role of SDEJT in the implementation of bilingual education under VL!? Do you think SDEJT could contribute in any other way to the implementation of this program? 2. What is the role of SDEJT in organizing and implementing teacher training at district level? 3. What teacher trainings were held this year? How many and who were the participants and what is the program and the duration of the courses? 4. How do you get feedback on the implementation of training (training, coaching) from teachers, schools, and ZIPs? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 5. Do you think there is capacity to manage the process of teacher training at the district level at this time and after the end of VL!? To what extent? (You can talk about the availability and capacity of human resources, availability of material resources, etc.) 6. How do you assess the level of preparedness of teachers’ trainers from the district level? What strengths and weaknesses can be identified in terms of skills of trainers at this level? USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 87 7. Do you think that the teacher trainings are having the desired effect? What evidence do you have to support your assessment? 8. How do you assess the level of preparedness of teachers for the implementation of bilingual education? What strengths and weaknesses can be identified in terms of skills of teachers to teach in the bilingual education? 9. What are the main challenges presented by teachers at district level? How are these challenges being or should they be addressed? 10. In your opinion, what are the main challenges regarding the training at the district level? How are these challenges being or should they be addressed? 11. What recommendations for improvement do you make regarding VL! training and follow-up? (You can talk about organizational, logistical, technical aspects, etc.). 12. Before we finish, we would like to leave some space for you to address any other aspects not dealt with in our conversation that you consider relevant to share with us. Interview script for the pedagogical department/bilingual education technician and focal point 1. What activities do you perform under VL! and specifically in relation to teacher training? 2. How do you assess the level of preparedness of the sector staff as trainer and give technical support to bilingual education and coaches? What do you consider the strengths and weaknesses of SDEJT staff that support bilingual education in the district? 3. Do you consider that the time devoted to the training of teachers and coaches is sufficient for them to successfully teach bilingual education? Why? What could be done differently to have more competent teachers? 4. Do you consider that teachers and coaches are properly trained to perform their work? What evidence do you have to justify your assessment? (Consider the various competency elements required to teachers.) 5. In your opinion, how could the ability of teachers to teach in the bilingual education program be increased? 6. What kind of support do you offer ZIPs, coaches, and teachers? 7. What are the challenges faced in working with ZIPs? How are these challenges being or should they be addressed? 8. How do you receive and give feedback on training implementation (training, coaching)? Is this information used by you and your senior bodies to make the necessary adjustments to improve the quality of training? Can you give examples? 9. What recommendations for improvement do you have regarding the training program and VL! coaching (organizational, logistical, technical aspects, etc.)? 10. Before we finish, we would like to leave some space for you to address any other aspects not dealt with in our conversation that you consider relevant to share with us. 88 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Interview script for the Vamos Ler! District Officer Interview script for the DO 1. As district officer, what is your role within VL!? 2. How do you assess the program of training and coaching of teachers under VL! in its various components? 3. How do you assess the level of preparedness of the SDEJT staff to train and give technical support to bilingual education teachers and coaches? What do you consider their strengths and weaknesses? 4. Do you consider that the time devoted to teacher training is sufficient? Why? 5. How do you assess the level of preparation of teachers for the implementation of bilingual education? What strengths and weaknesses can you identify in terms of the various skills of teachers to teach in bilingual education? 6. What are the main challenges identified in teacher’s performance? How are these challenges being or should they be addressed? 7. Do you consider that the facilitators and coaches have sufficient preparation and experience to provide didactic-pedagogical support to teachers in schools? What evidence do you have to justify your assessment? 8. What are the main challenges faced by facilitators and coaches in teacher support work? How are these challenges being or should they be addressed? 9. How have the Pedagogical Workshops of Literacy at L1 been organized? Have they received support from the district? From the province? (Can you explain who/role?) How do you organize the Pedagogical Workshop agenda? How do you know the specific difficulties of the teachers? In the Pedagogical Workshops, have you devoted time to strengthening teachers’ capacity in the areas they identify as weaker? 10. How do you give and receive feedback about the implementation of training (training, coaching)? Is this information used to make the necessary adjustments to improve the quality of training? Can you give examples? 11. What recommendations do you have regarding the VL! training and coaching program (organizational, logistical, technical aspects, etc.)? 12. Before we finish, we would like to leave some space for you to address any other aspects not dealt with in our conversation that you consider relevant to share with us. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 89 ANNEX XI: OBSERVATION FORMS OBSERVATION FORM for TRAINING OF TRAINERS A. General Data Province Nampula ☐ Zambézia ☐ District Place of training Grade 1ª ☐ 2ª ☐ Mozambican language teaching Trainer (a) H ☐ M ☐ Date of visit _____/_____/ 2019 Start and end time of training Official start time: ___hr ___min REAL start time: ___hr ___min Official end time: ___hr ___min REAL end time:___hr ___min B. Participant Management: Attendance and Absenteeism 1 Did the trainer check and record the attendance of the participants on the attendance list? Yes ☐ No ☐ 2 Number of participants in the room (count or check attendance list) Total___ H_____ M_____ 3 Expected number of participants Total___ H_____ M_____ Explore the main reasons for the recorded absences: Section I: Preparation for Class C. Student Management: Attendance and Absenteeism 4 Did the trainer instruct participants to check and record teachers’ attendance on the attendance list? Yes ☐ No ☐ D. Preparation of the Teacher for the Class 5 Are participants taught to teach the planning of class of reading and writing? Yes ☐ No ☐ 6 The participants have didactic material needed for this training:  Teacher’s Manual  Curricular Maps  Weekly Plan  Student Book  Exercise Book Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ No ☐ No ☐ No ☐ No ☐ No ☐ 90 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV D. Preparation of the Teacher for the Class  Complementary Books  Detailed Lesson Plan Model  Mathematics Lesson Plan Model  Continuous Evaluation Form Yes ☐ Yes ☐ Yes ☐ Yes ☐ No ☐ No ☐ No ☐ No ☐ 7 Do participants understand the content, organization, and function of the main teaching and learning materials? Yes ☐ No ☐ 8 Were participants taught how to prepare and make use of the Didactic Material for the classes (in addition to the materials provided by VL!)? Yes ☐ No ☐ Make your comments on the preparation of the participants for teacher training. Make notes about the materials prepared for this class by the trainer (quality of plan and materials, level of preparation of the trainer, etc.): E. Participant Materials 9 How many participants did not receive all support didactic material for the training? Total___ H___ M___ 10 Did participants demonstrate ease of understanding of the use of materials provided? Yes ☒ No ☐ N/A ☐ Make your comments regarding the ease of understanding of the use of materials. Section II: During the Lesson F. Reading Practice: The trainer explains to the participants how to do: 11 Combination of letter sounds to form syllables and words Yes ☐ No ☐ N/A ☐ 12 Reading aloud to each other (reading in pairs) Yes ☐ No ☐ N/A ☐ 13 Silent reading (individual reading to yourself Yes ☒ No ☐ N/A ☐ 14 Reading aloud at the same time (chorus reading) Yes ☐ No ☐ N/A ☐ 15 Comprehension questions from the text they read Yes ☐ No ☐ N/A ☐ 16 Interpretation questions from the text they read Yes ☒ No ☐ N/A ☐ 17 The use of prior knowledge to understand and interpret the text they read Yes ☐ No ☐ N/A ☐ 18 The exploring of the text word meanings from the texts they read Yes ☐ No ☐ N/A ☐ Make your observations about reading practice in the classroom (fluency, comprehension and interpretation skills, vocabulary, role of trainer in facilitating reading learning, etc.): Make your observations about the time devoted to reading practice. Was it adequate? Enough? Record key comments, observations, and questions from participants. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 91 G. Writing Practice: The trainer guides participants on how to teach to: 19 Draw, make pictures, or paint to practice the development of psychomotor skills (individually) Yes ☐ No ☐ N/A ☐ 20 Write letters, syllables, or words on paper, notebook, or slate (individually) Yes ☐ No ☐ N/A ☐ 21 Create or develop own texts (sentences or stories) Yes ☐ No ☐ N/A ☐ 22 Plan writing (e.g., brainstorming, word exploration, text structuring, etc.) Yes ☐ No ☐ N/A ☐ Make your comments on writing practice in the classroom (handwriting, speech clarity, linguistic correction, role of trainer in facilitating the learning of writing, etc.): Make your comments on the time devoted to writing practice. Was it adequate? Enough? H. Interaction in the Classroom: The trainer instructs participants on how to: 23 Promote student interaction on class subjects Yes ☐ No ☐ 24 Promote interaction between students when doing individual, peer, and group work Yes ☐ No ☐ 25 Encourage students to ask the trainer questions about class subjects Yes ☐ No ☐ 26 Select students in a balanced way to answer the questions Yes ☐ No ☐ 27 Encourage students to participate in class Yes ☐ No ☐ 28 Encourage students with learning disabilities to participate in class Yes ☐ No ☐ 29 Identify students with learning disabilities Yes ☐ No ☐ 30 Identify routines and educational remediation activities for students with learning disabilities Yes ☐ No ☐ 31 Manage student behavior in the classroom? Yes ☐ No ☐ How does the trainer teach participants to encourage student participation? What was done (if it only explains what it is or shows how it is done)? Make your comments on the participation of participants in the training (willingness, inclusiveness, role of trainer in facilitating the participation of all students, etc.): Make your observations about the time devoted to facilitating interaction between students. Was it adequate? Enough? I. Teaching Methods: The trainer: 32 Taught the routine application "I do," "We do," "You do," "You do" Yes ☐ No ☐ 33 Taught how to work in pairs and in groups Yes ☐ No ☐ 34 Taught the organization of discussions involving the whole class Yes ☐ No ☐ 92 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 35 Taught how to observe personalized teaching Yes ☐ No ☐ 36 Taught participants to praise the students Yes ☐ No ☐ 37 Taught participants to encourage students (what to do and how to do) Yes ☐ No ☐ 38 Taught participants to stimulate students with learning disabilities (what and how to do) Yes ☐ No ☐ 39 Taught participants to use the potential of the material to promote a more positive image of girls and women? Yes ☐ No ☐ 40 Taught and gave instruments to the participants to know how to respond to situations of gender-based discrimination and other conditions (disability, albinism) between students and with others Yes ☐ No ☐ Make your observations about the teaching methods used by the trainer (suitability for the subjects given to the participants and effectiveness in the support of learning, etc.): Make your observations about the time devoted to the practice of teaching methods. Was it adequate? Enough? Make your observations if the teaching of practical methods was only theoretical or if did simulations. Make your comments about how the trainer addressed gender discrimination/social inclusion issues. J. Student Assessment: Does the trainer teach participants how to teach students’ learning assessment by 41 Asking students questions during class? Yes ☐ No ☐ 42 Monitoring students while doing individual, pair, or group work? Yes ☐ No ☐ 43 Listening to each student reading aloud and suggesting appropriate improvements? Yes ☐ No ☐ 44 Reading the written work done by each student or in groups and suggesting appropriate improvements? Yes ☐ No ☐ 45 Using an instrument/reading assessment sheet, giving a survey, or testing the class? Yes ☐ No ☐ 46 Monitoring the pedagogical performance of the students? Yes ☐ No ☐ 47 Planning and implementing pedagogical remediation routines and activities according to student achievement? Yes ☐ No ☐ Make your observations about student assessment methods (systematics, comprehensiveness, relevance in supporting student learning, etc.): Make your observations about whether the teaching of the practice was only theoretical or if simulations. USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 93 Section III: Profile and performance of the trainer K. Performance of the Trainer From your observation, do you consider that the trainer: 48 Speaks fluently the LI of teaching used in this province? Yes ☐ No ☐ 49 Reads fluently in this LI? Yes ☐ No ☐ 50 Writes fluently in this LI? Yes ☐ No ☐ 51 Has mastered the main reading teaching and learning methods and routines in L1, recommended in VL!? Yes ☐ No ☐ 52 Has mastered the main writing teaching and learning methods and routines in L1, recommended in VL!? Yes ☐ No ☐ 53 Has mastered the main reading and writing assessment methods and routines? Yes ☐ No ☐ 54 Promoted a good interaction and kept trainees interested/engaged in work? Yes ☐ No ☐ 55 Answered the questions raised by participants? Yes ☐ No ☐ Make your comments on the profile and performance of the trainer (do you think he/she has the ability and profile to be a trainer of bilingual teaching trainers? What are his/her strengths and weaknesses?) OBSERVATION FORM for TRAINING OF TEACHERs A. General Data Province Nampula ☐ Zambézia ☐ District Training site Grade 1ª ☐ 2ª ☐ Mozambican language teaching Trainer H ☐ M ☐ Date of visit _____/_____/ 2019 Start and end time of class Official start time: ___hr ___min REAL start time: ___hr ___min Official end time: ___hr ___min REAL end time:___hr ___min B. Participant Management: Attendance and Absenteeism 1 Did the trainer check and record the attendance of the participants on the attendance list? Yes ☐ No ☐ 2 Number of attendees in the room (count or check attendance list) Total___ H_____ M_____ 94 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV 3 Expected number of participants Total___ H_____ M_____ 4 Number of teachers present Total___ H_____ M_____ 5 Number of School Directors (DE) present Total___ H_____ M_____ 6 Number of Assistant Principals (DEA) present Total___ H_____ M_____ 7 Number of District Officers present Total___ H_____ M_____ 8 Number of ZIP Guiding Teachers (PO) present Total___ H_____ M_____ 9 Number of Cycle Coordinators (CC) present Total___ H_____ M_____ Explore the main reasons for recorded absences: Section IV: Sustainability 56 Was there and what was the intervention, support, or contribution of the VL! team during the training? Yes ☐ No ☐ Present evidence: Section I: Preparation for Class C. Student Management: Attendance and Absenteeism 10 Did the trainer instruct participants to check and record teachers’ attendance on the attendance list? Yes ☐ No ☒ D. Preparation of the Teacher for the Class 11 Are participants taught to teach the planning of class of reading and writing? Yes ☐ No ☐ 12 Did the participants have didactic material needed for this training?  Teacher’s Manual  Curricular Maps  Weekly Plan  Student book  Exercise Book  Complementary Books  Detailed Lesson Plan Model  Mathematics Lesson Plan Model  Continuous Evaluation Form Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ Yes ☐ No ☐ No ☐ No ☐ No ☐ No ☐ No ☐ No ☐ No ☐ No ☐ 13 Did teachers understand the content, organization, and function of the main teaching and learning materials? Yes ☐ No ☐ USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 95 D. Preparation of the Teacher for the Class 14 Were teachers taught how to prepare and make use of the Didactic Material for the classes (in addition to the materials provided by VL!)? Yes ☐ No ☐ Make your observations about teacher preparation for class. Make notes about the materials prepared for this class by the trainer (quality of plan and materials, level of preparation of the trainer, etc.): E. Teacher Materials 15 How many teachers did not receive all the support material for the training? Total___ H___ M___ 16 Did the teachers demonstrate easy understanding of the use of the materials provided? Yes ☒ No ☐ N/A ☐ Make your comments regarding the ease of understanding of the use of materials. Section II: During the Lesson F. Reading Practice: The trainer explains to the teachers how to do: 17 Combination of letter sounds to form syllables and words Yes ☐ No ☐ N/A ☐ 18 Reading aloud to each other (reading in pairs) Yes ☐ No ☐ N/A ☐ 19 Silent reading (individual reading to yourself) Yes ☐ No ☐ N/A ☐ 20 Reading aloud at the same time (chorus reading) Yes ☐ No ☐ N/A ☐ 21 Comprehension questions from the text they read Yes ☐ No ☐ N/A ☐ 22 Interpretation questions from the text they read Yes ☒ No ☐ N/A ☐ 23 The use of prior knowledge to understand and interpret the text they read Yes ☐ No ☐ N/A ☐ 24 The exploring of the text word meanings from the texts that they read Yes ☐ No ☐ N/A ☐ Make your observations about reading practice in the classroom (fluency, comprehension and interpretation skills, vocabulary, role of trainer in facilitating reading learning, etc.): Make your observations about the time devoted to reading practice. Was it adequate? Enough? Record key comments, observations, and questions from teachers. G. Reading Practice: The trainer guides the teachers on how to teach to: 25 Draw, make pictures, or paint to practice the development of psychomotor skills (individually) Yes ☐ No ☐ N/A ☐ 26 Write letters, syllables, or words on paper, notebook, or slate (individually) Yes ☐ No ☐ N/A ☐ 27 Create or develop own texts (sentences or stories) Yes ☐ No ☐ N/A ☐ 28 Plan writing (e.g., brainstorming, word exploration, text structuring, etc.) Yes ☐ No ☐ N/A ☐ 96 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV Make your observations about classroom writing practice (handwriting, clarity of speech, language correction, role of trainer in facilitating learning writing, etc.): Make your observations about the time devoted to writing practice. Was it adequate? Enough? H. Interaction in the Classroom: Does the trainer instruct the teachers on how to: 29 Promote interaction between students about class subjects? Yes ☐ No ☐ 30 Promote interaction between students when doing individual, pair, and group work? Yes ☐ No ☐ 31 Encourage students to ask questions to the trainer about class subjects? Yes ☐ No ☐ 32 Select students in a balanced way to respond to questions? Yes ☐ No ☐ 33 Encourage students to participate in class? Yes ☐ No ☐ 34 Encourage students with learning disabilities to participate in class? Yes ☐ No ☐ 35 Identify students with learning disabilities? Yes ☐ No ☐ 36 Identify routines and educational remediation activities for students with learning disabilities? Yes ☐ No ☐ 37 Manage student behavior in the classroom? Yes ☐ No ☐ How does the trainer teach the teachers to encourage student participation? What was done (if it only explains what it is, or shows how it is done)? Make your comments on the participation of trainers in the training (willingness, inclusiveness, role of trainer in facilitating the participation of all students, etc.): I. Teaching Methods: The trainer: 38 Taught the routine application "I do," "We do," "You do," "You do." Yes ☐ No ☐ 39 Taught how to organize work in pairs and in groups. Yes ☐ No ☐ 40 Taught how to organize discussions involving the whole class. Yes ☐ No ☐ 41 Taught how to observe personalized teaching. Yes ☐ No ☐ 42 Taught the teachers to praise the students. Yes ☐ No ☐ 43 Taught the teachers to encourage students (what to do and how to do). Yes ☐ No ☐ 44 Taught the teachers to stimulate students with learning disabilities (what and how to do). Yes ☐ No ☐ 45 Taught the teachers to use the potential of the material to promote a more positive image of girls and women. Yes ☐ No ☐ 46 Taught and gave instruments to teachers to know how to respond to situations of gender-based discrimination and other conditions (disability, albinism) between students, or students with the others. Yes ☐ No ☒ Make your observations about the teaching methods used by the trainer (suitability for the subjects given to the students, effectiveness in the support of learning of the teachers, etc.): USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 97 I. Teaching Methods: The trainer: Make your observations about the time devoted to the practice of teaching methods. Was it adequate? Enough? Make your observations if the practice was only theoretical or if simulations were done. Make your observations about how the trainer addresses gender discrimination/social inclusion issues. J. Student Assessment: Does the trainer teach teachers to evaluate student learning by: 47 Asking students questions during class? Yes ☐ No ☐ 48 Monitoring students while doing individual, pair, or group work? Yes ☐ No ☐ 49 Listening to each student’s reading aloud and suggesting appropriate improvements? Yes ☐ No ☐ 50 Reading the written work done by each student or in groups and suggesting appropriate improvements? Yes ☐ No ☐ 51 Using an instrument/reading assessment sheet, giving a survey, or testing the class? Yes ☐ No ☐ 52 Monitoring the pedagogical performance of the students? Yes ☐ No ☐ 53 Planning and implementing pedagogical remediation routines and activities according to student achievement? Yes ☐ No ☐ Make your observations about student assessment methods (systematics, comprehensiveness, relevance in supporting student learning, etc.): Make your observations if practice was only theoretical or if simulations were done. Section III: Profile and performance of the trainer K. Trainer Performance From your observation, do you consider that the trainer: 54 Speaks fluently the LI of teaching used in this province? Yes ☐ No ☐ 55 Reads fluently in this L1? Yes ☐ No ☐ 56 Writes fluently in this L1? Yes ☐ No ☐ 57 Has mastered the main reading teaching and learning methods and routines in L1, recommended in VL!? Yes ☐ No ☐ 58 Has mastered the main writing teaching and learning methods and routines in L1, recommended in VL !? Yes ☐ No ☐ 59 Has mastered the main reading and writing assessment methods and routines? Yes ☐ No ☐ 60 Promoted good interaction and kept trainees interested/engaged in work? Yes ☐ No ☐ 98 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV K. Trainer Performance From your observation, do you consider that the trainer: 61 Answered the questions raised by participants? Yes ☐ No ☐ Make your comments about the profile and performance of the trainer (do you think they have the ability and profile to be a trainer of bilingual teachers? What are their strengths and weaknesses? Were the trainers concerned about maintaining the quality of training they themselves had received? Comment. Section IV: Sustainability 62 Was there any intervention, support, or contribution of the VL! team during the training? Yes ☐ No ☐ Present evidence: USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 99 Observation Form for L1 Reading/Writing Class A. General Data Province Nampula ☐ Zambézia ☐ District ZIP name School Grade 1ª ☐ 2ª ☐ Mozambican language teaching Teacher H ☐ M ☐ Date of visit _____/_____/ 2019 Start and end time of class Official start time: ___hr ___min REAL start time: ___hr ___min Official end time: ___hr ___min REAL end time:___hr ___min Observer Name Instructions: 1. This form is intended to capture specific data on the environment and course of the observed L1 reading/writing classes. 2. The form applies to both grades 1 and 2. 3. This form must be completed throughout the observation of the class, although it may be improved immediately after the observed class. 4. The worksheet comprises three main parts, each part having different components: Part I: Preparation for the Class; Part II: Course Proceedings; and Part III: Profile and Performance of the Teacher. 5. Fill in the spaces indicated as instructed. In most cases, you should answer the questions you have raised by ticking “x” in the space reserved for YES or NO. YES means that the procedure or practice indicated was observed in class and NO means it was not observed. However, there are cases of procedures or practices provided in the form that, depending on the topic and objectives, may not be relevant to the observed class. In such cases, put an “X” in the N/A indication, meaning “Not Applicable.” Section I: Preparation for Class B. Participant Management: Attendance and Absenteeism 1 Did the teacher track and record student attendance in the class book? Yes ☐ No ☐ 2 Number of students in class (count) Total_______ H_____ M_____ 3 Number of students registered in the class book (see list) Total_______ H_____ M_____ Explore the main reasons given for absences recorded: 100 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV C. Preparation of Teacher for Class Yes No 4 Does the teacher have a Lesson Plan for the class (he) or (she) will teach? 5 Does the lesson given correspond to the lesson planned for this date in the weekly plan? 6 Does the teacher have the L1 and L2 Teacher Manual? 7 Does the teacher have the grade 1/ grade 2 L1 Student Book, as relevant? 8 Does the teacher have other Didactic Material prepared for this class, besides the materials provided by VL!? Make your observations about the materials prepared for this class (quality of the plan and materials, the teacher’s level of preparation, reasons the given class does not match the one planned for this date (if applicable), etc.): D. Student Materials 9 How many students don’t have L1 Student’s Book of Grade 1 or 2, as relevant? Total___ H___ M___ 10 How many students don’t have L1 Exercise Book? Total___ H___ M___ 11 Is there Complementary L1 Reading Material available in the classroom? Yes ☐ No ☐ Make your observations about student materials. Section II: During the Lesson E. Reading Practice: Does the teacher guide students to: Yes No N/A 12 Combine letter sounds to form syllables and words? 13 Read aloud to each other (peer reading)? 14 Read silently (individual reading to yourself)? 15 Read aloud at the same time (chorus reading)? 16 Answer comprehension questions of the text they read? 17 Answer questions of interpretation of the text they read? 18 Use previous knowledge to understand and interpret the text they read? 19 Explore the meanings of words in the text you read? Make your observations about classroom reading practice (fluency, comprehension and interpretation skills, vocabulary mastery, role of teacher in facilitating reading learning, etc.): F. Writing Practice: Does the teacher lead the students to: Yes No N/A 20 Draw, make figures, or paint to practice the development of psychomotor skills (individually)? 21 Write letters, syllables, or words on paper, notebook, or slate (individually)? 22 Create or elaborate own texts (sentences or stories)? USAID.GOV VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 | 101 F. Writing Practice: Does the teacher lead the students to: Yes No N/A 23 Plan text writing (e.g. brainstorming, word exploration, text structuring, etc.)? 24 Correct texts before submitting them to the teacher? 25 Rewrite texts after teacher’s correction? Make observations about writing practices in the classroom (handwriting, clarity of speech, language correction, role of the teacher in facilitation of writing learning, etc.): G. Interaction in the Classroom Yes No N/A 26 Do students interact with each other about class subjects? 27 Does the teacher interact with students when doing individual, pair, and group work? 28 Do students ask the teacher questions about class subjects? 29 Does the teacher select male and female students in a balanced way to respond to questions (e.g., does not favor the selection of male students to the detriment of female students)? 30 Does the teacher encourage the female students to participate in class? 31 Does the teacher encourage students with learning disabilities to participate in class? Make observations about student participation in the classroom (willingness, inclusiveness, teacher role in facilitating the participation of all students in the classroom, etc.): H. Teaching Method: Does the teacher: Yes No N/A 32 Apply the routine ‘I do,’ ‘We do,’ ‘You do,’ ‘You do’? 33 Organize works in pairs? 34 Organize work in groups? 35 Organize discussions involving all the class? 36 Make personalized teaching (e.g., individually help students who show difficulty in understanding any matter or performing any activity)? 37 Praise the students? 38 Call on students with learning difficulties? 39 Use materials’ potential to promote a more positive image of girls and women? Make observations about teaching methods used by the teacher (adequacy of the material given to the students, effectiveness in supporting student’s learning, etc.): I. Student Assessment: Does the teacher evaluate student learning by: Yes No N/A 40 Asking questions to students during the class? 41 Monitoring the students while performing individual, pair, or group work? 102 | VAMOS LER! DEEP LEARNING AND ADAPTIVE STUDY #2 USAID.GOV I. Student Assessment: Does the teacher evaluate student learning by: Yes No N/A 42 Listening to the reading that each student does aloud and suggesting suitable improvements? 43 Reading the written works made by each student, or in groups, and suggesting suitable improvements? 44 Using instruments/relevant assessment forms, giving a survey, or testing the class? Make observations about teacher’s evaluation methods (consistency, relevance to supporting student’s learning, etc.): Section III: Profile and Performance of the Teacher J. Performance of the Teacher From your observation, do you consider that the teacher: Yes No N/A 45 Speaks fluently the LI used for teaching in this school? 46 Reads fluently in this L1? 47 Writes fluently in this L1? 48 Has mastered the main methods and routines of teaching and learning reading in L1, recommended on the VL!? 49 Has mastered the main methods and routines of teaching and learning writing in L1, recommended on the VL!? 50 Has mastered the main methods and routines of reading and writing assessment? Make observations on the profile and performance of the teacher. Do you consider that he/she has the ability and profile to be a bilingual education teacher? What are their strengths and weaknesses?