MOZAMBIQUE McGovern-Dole International Food for Education and Child Nutrition Educating Children Together Phase 2 (ECT2) Final Evaluation 1/13/2022 DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Educating Children Together Phase 2 (ECT2) Final Evaluation Report Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-656-2015-011-00 Funding Year: Fiscal Year 2021 Project Duration: FY2016 – FY2021 Implemented by: WORLD VISION Evaluation Authored by: INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI) FELICIANO CHIMBUTANE, CARLOS LAUCHANDE, CATALINA HERRERA￾ALMANZA, NAUREEN KARACHIWALLA, JESSICA LEIGHT, DANIEL MAGGIO ii Table of Contents 1 Introduction................................................................................................................................. 4 1.1 Background ............................................................................................................................... 4 1.2 Interventions ............................................................................................................................. 5 1.4 Evidence on School Feeding.................................................................................................. 7 1.5 Evidence on Teacher Pedagogy, Training, and Remedial Education ..........................7 1.6 Objectives and Research Questions .................................................................................... 9 2 Methodology................................................................................................................................... 10 2.1 Quantitative Methods ........................................................................................................... 10 2.1.1 EGRA Description ............................................................................................................. 14 2.2.2 Reliability of the EGRA Test............................................................................................15 2.2 Qualitative Methods .............................................................................................................. 16 2.3. Data Collection Methods: ECT2 Endline ........................................................................17 2.3.1 Training and Data Collection..........................................................................................18 2.3.2 Data Management and Security.....................................................................................19 2.3.3 COVID-19 Specific Precautions.......................................................................................19 2.4 Evaluation Constraints and Limitations ...........................................................................20 3 Results on ECT2 Indicators ........................................................................................................ 20 3.1 Summary Statistics................................................................................................................ 21 3.2 Improved Literacy of School-Aged Children (SO1) ......................................................26 3.3 Literacy Instruction (1.1) ..................................................................................................... 35 3.4 Teacher Attendance (1.1.1)................................................................................................. 37 3.5 Access to School Supplies and Materials (1.1.2).............................................................39 3.6 Improved Attentiveness (1.2).............................................................................................41 3.7 Reduced Short-Term Hunger (1.2.1)................................................................................43 3.8 Increased Access to Food (School Feeding) (1.2 & 1.3)................................................45 3.9 Improved Student Attendance (1.3).................................................................................48 3.10 Reduced Health Related Absences (1.3.2).....................................................................50 3.11 Increased Knowledge and Use of Health, Hygiene, and Dietary Practices (2.1)..52 3.12 Increased Knowledge of Safe Food Preparation and Storage Practices (2.2).......55 3.13 Increased Access to Clean Water and Sanitation Services (2.4).............................57 4 COVID-19 Experiences................................................................................................................ 59 4.1 Lockdown Hardship and Coping Strategies ....................................................................59 4.2 Education in the Age of COVID-19 ...................................................................................62 ii 4.3 School Reopening................................................................................................................... 63 5 Conclusions and Recommendations.........................................................................................66 References .......................................................................................................................................... 70 Appendix 1: Performance Indicators...........................................................................................74 Appendix 2: Evaluation Terms of Reference (TOR)................................................................79 Appendix 3: Survey Instruments .................................................................................................. 90 Appendix 4: Conflict of Interest Statement.............................................................................130 List of Tables Table 1: Data availability by district across treatment and control groups in baseline, midline and endline................................................................................................................................................... 11 Table 2: Cronbach’s alpha reliability coefficient ...................................................................................16 Table 3: Qualitative Data Collected, by District ..................................................................................16 Table 4: Characteristics of Students in the Sample ..............................................................................22 Table 5: Characteristics of Teachers in the Sample .............................................................................24 Table 6: Characteristics of Deputy Directors and Schools in the Sample...........................................25 Table 7: Summary of EGRA mean scores by subtest...........................................................................28 Table 8: Summary of EGRA mean scores by subtest by gender..........................................................31 Table 9: Percentage of students who by the end of two grades of primary schooling demonstrate that they can read and understand the meaning of grade level text – Baseline, Midline and Endline vs. Target.................................................................................................................................................... 33 Table 10: Percentage of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors or coaches – Baseline, Midline and Endline vs. Target .................36 Table 11: Percentage of teachers in target schools who attend and teach school at least 80% of scheduled days per school year – Baseline, Midline and Endline vs. Target ........................................38 Table 12: Percentage of Teachers who received textbooks and other teaching and learning materials provided as a result of USDA assistance – Baseline, Midline and Endline vs. Target ..........................40 Table 13: Percentage of students in target schools attentive during class/instruction – Baseline, Midline and Endline vs. Target.............................................................................................................. 42 Table 14: Percentage of students in target schools who indicate that they are not hungry during school day – Baseline, Midline and Endline vs. Target..........................................................................44 Table 15: Number of school aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance – Baseline, Midline and Endline vs. Target.................................................47 Table 16: Percentage of students that attend school at least 80% of the school days – Baseline, Midline and Endline vs. Target.............................................................................................................. 49 Table 17: Percent of students in target schools who miss more than 10 school days/year due to illness – Baseline, Midline and Endline vs. Target .................................................................................51 Table 18: Percentage of children in target communities who can identify at least 3 important health/hygiene practices – Baseline, Midline and Endline vs. Target ....................................................53 iii Table 19: Percent of target beneficiaries who use appropriate hand washing practices – Baseline, Midline and Endline vs. Target.............................................................................................................. 54 Table 20: Percentage of food preparers at target schools who can identify at least 3 key practices aimed at safe food preparation – Baseline, Midline and Endline vs. Target .........................................56 Table 21: Number of schools using an improved water source and number of schools with improved sanitary facilities – Baseline, Midline and Endline vs. Target.................................................................58 List of Figures Figure 1: Difference in Differences....................................................................................................... 12 Figure 2: Correct answers on Each EGRA subtask..............................................................................30 Figure 3: Student reading ability in treatment and control schools at baseline, midline and endline.34 Figure 4: COVID-19 Hardships and Coping Strategies........................................................................60 Figure 5: Main Reasons Children did not Return to School.................................................................61 Figure 6: Pre- COVID-19 Educational Measures..................................................................................62 Figure 7: Educational Measures during Lockdowns..............................................................................63 Figure 8: COVID-19 Mitigation during School Reopening ...................................................................64 Figure 9: COVID Mitigation Shortcomings...........................................................................................65 iv List of Acronyms CESC Centro de Aprendizagem e Capacitação da Sociedade Civil CI Confidence interval CNBS Comité Nacional de Bioética em Saúde COVID-19 Coronavirus disease CSB Corn soy blend DID Difference in Difference ECT Educating Children Together EGRA Early Grade Reading Assessment ES Effect size IFPRI International Food Policy Research Institute INDE Instituto Nacional de Educação / Institute for the Development of Education IRT Item Response Theory LB Literacy Boost MINEDH Ministério de Educação e Desenvolvimento Humano / Ministry of Education and human Development NGO Non-Governmental Organization RCT Randomized controlled trial SD Standard deviation SE Standard error SDEJT District Services of Education, Youth and Technology TARL Teaching at the Right Level UL Unlock Literacy UNESCO United Nations Educational, Scientific and Cultural Organization UNICEF United Nations Children’s Fund USDA United States Department of Agriculture WV World Vision ZIP Zona de Influência Pedagógica / Pedagogical Influence Zone v Acknowledgements The authors acknowledge the contributions of several actors. We first thank World Vision for support on this project, both with project design and with the facilitation of several steps of this work. We also thank ELIM Serviçios for excellent quantitative data collection and Jessemusse Cacinda for excellent qualitative data collection. We acknowledge funding from World Vision (grant number FFE￾656-2019/018-00-IFPRI) as well as funding from the Consultative Group on International Agricultural Research (CGIAR) Program Policy, Institutions, and Markets (PIM) research program. 1 Executive Summary This report presents the results of the endline evaluation of the Educating Children Together Phase 2 (ECT2) program in Nampula province, Mozambique, implemented by World Vision. The program addresses multiple dimensions of the low levels of human capital in Mozambique by providing school meals and early grade literacy programming with improvements in the learning environment and health/nutrition education. The evaluation conducted for this report measures the impacts of the school meals and literacy program in 175 rural public primary schools in Nampula (160 treatment schools in the target districts, and 15 control schools). Quantitative and qualitative data was conducted by a survey firm working under the supervision of the International Food Policy Research Institute (IFPRI) and was analyzed in conjunction with earlier waves of data collected separately at baseline and midline, as well as programmatic data provided by World Vision. The program included two treatment districts (Nacarôa and Muecate), which were targeted because they had very low levels of human capital, and a control district (Murrupula), which was selected purposively as a comparison. We provide three types of quantitative analysis in this report. The first is difference in differences (DID) estimation for literacy scores between midline and endline only as that is the only raw, individual-level data that we have available. This analysis method compares the changes over time between the treatment and control groups. Second, we present trends over time in the treatment groups at baseline, midline, and endline. Tests for significant differences between baseline and endline are presented. Finally, we test for significant differences between the treatment and control districts at endline. The quantitative analysis is supplemented by qualitative data from observations and quotations from students, teachers, deputy school directors, farmers, parents, and World Vision and education partners. We begin by noting that there are two important challenges to interpreting the results in this report. The first is that Murrupula district was different from Nacarôa and Muecate districts from the beginning of the study. Murrupula was employed as the control district for the midline evaluation, and accordingly has continued to serve as the control district in this analysis for consistency. The socio￾economic status of households and quality of schools in Murrupula was higher, and as such, does not constitute the ideal control group. Additionally, schools in the control group in Murrupula did receive some literacy-related and other schooling interventions. Consequently, the results presented in this report should not be interpreted as causal impact estimates. Second, the COVID-19 pandemic severely affected schools and other outcomes in Mozambique and around the world. We can ex-ante expect that education and nutritional outcomes would decrease across the board. Thus, while the program may have had very beneficial effects (as suggested by the midline data and by numerous stakeholders), the data presented here appear as if that is not the case. In almost all cases, decreases in the level of achievement of indicators can likely be attributed to disruptions caused by COVID-19. However, the main conclusion of this report is that there were likely beneficial effects. Student nutrition, attentiveness, and attendance increased, access to teaching and learning materials and teacher pedagogical practices increased, and school facilities were bolstered. The results suggest that, though ECT2 may have contributed to improve literacy levels of early grade students in Nacarôa and Muecate, the difference-in-difference estimate of the impact of the project on the Early Grade Reading Assessment (EGRA) is negative. This result can be attributed to the shocks linked to the COVID-19 pandemic, including the closure of schools for the entire 2020 school year, which was the final year of ECT2. The findings indicate that although students in control schools performed better than their peers in treatment schools, both groups are still struggling to read with comprehension. The progress by students in Murrupula may also reflect in part the launch of other interventions (Vamos Ler!) targeting early school literacy in this district following its initial selection as a control district at baseline, a point elaborated in further detail below. Vamos Ler is a USAID 2 funded project targeted at improving early grade reading outcomes and is serving 2,800 schools in Nampula and Zambézia through teaching young children to read in local languages while simultaneously building Portuguese language acquisition. In addition, though stakeholders and ECT2 monitoring data indicate that teacher attendance had substantially improved, this study found some decrease when comparing with the midline results. The decrease can be attributed to the disruptions in teaching and learning processes associated with school closures from March to December 2020 and to the irregular functioning of schools in 2021 due to the ongoing COVID-19 pandemic. (Schools reopened only in February 2021, and have operated under new restrictions on the numbers of students attending each day, leading to a restricted schedule in which students attend only two to three days a week.) Despite prevailing challenges, ECT2 has contributed to improving access to school supplies and materials in treatment schools via acquisition and production of books stocked in book banks and school libraries. Project reports and accounts from different stakeholders also indicate that ECT2 contributed to improve students’ attentiveness in Nacarôa and Muecate districts. However, the endline study found that the impact of school feeding on students’ attentiveness has dropped. This decline can be attributed to COVID-19 effects associated with the end of ECT2 and preparation of a new school feeding cycle. The number of school-aged children receiving daily school meals in Nacarôa and Muecate increased by 42% when comparing the baseline and endline results (from 57,501 to 81,589 students). This in part reflects increased enrolment that may be driven by the benefits provided by the project. This increase means that the USDA program improved the well-being and predisposition to learn for a growing number of children in need, thus enhancing their opportunities to learn and stay longer in schools. In addition, ECT2 contributed to improving students’ attendance and retention. However, by the time of the endline survey the percentage of students who attended school at least 80% of the school days had dropped relative to baseline data. This decline can be attributed to COVID-19 related school closures and health fears, which prevented students from attending school on a regular basis. In addition, schools are generally recommending a reduced schedule of attendance (students attending only 2-3 days per week) in order to maintain reduced class sizes and compliance with COVID-19 protocols, and this has inevitably negatively affected attendance. Finally, the ECT2 project contributed to increased knowledge of safe food preparation and storage practices in Nacarôa and Muecate treatment schools and contributed to increased access to clean water sources and the number of schools with improved sanitary facilities. To sum up, ECT2 had some notable achievements in increasing literacy, enhancing student and teacher performance, and reducing hunger, particularly at the point of the midline evaluation. Subsequently, the onset of the COVID-19 pandemic generated substantial disruptions across Mozambique; evidence suggests these effects were particularly acute in the districts served by ECT2 (Nacarôa and Muecate), who were worse hit than the comparison district (Murrupula). These disruptions, in conjunction with ex-ante differences between the treatment and control districts, render the interpretation of patterns at the point of the endline survey challenging. Despite these disruptions, there is evidence of some substantial progress toward evaluation goals in the treatment schools. 3 We make three main recommendations. 1) First, the primary recommendation of this report is to continue the school meals in a self￾sustaining manner, building up community infrastructure to provide the meals is highly recommended. 2) Second, learning in Portuguese was identified as a large constraint to reading in Nampula. The introduction of bilingual education in Nacarôa and Muecate could be a way to enhance the opportunities of children to develop early grade literacy skills and providing more books and teaching and learning materials, particularly in Macua, would benefit children’s learning. While the government supports bilingual education, this policy may require more reinforcement at the school level. 3) Third, measures that were in place before the school closures, such as school council monitoring of teacher attendance and public displays of teacher attendance on school buildings should be continued. 4 1 Introduction 1.1 Background Increasing human capital in both the health and education spheres is crucial for building resilience to shocks and hardships, as well as for households to graduate out of poverty. Human capital investments in early years have important implications for resilience and poverty reduction in the future; both better nutrition and higher levels of learning improve access to and productivity in work and thus incomes, with intergenerational impacts as well. However, relatively little is known about how combinations of nutrition and learning interventions can improve human capital outcomes to build resilience and reduce poverty. In Mozambique, both types of human capital outcomes are either low or stagnant. While chronic malnutrition is very high across all of Mozambique (43%), it is more pronounced in the provinces of Nampula, Cabo Delgado, Niassa, and Zambezia (UNICEF 2016). Anemia prevalence was 66 percent in rural areas among children in 2015. In addition, Mozambique has one of the lowest educational attainment rates in the world, characterized by primary completion below 40 percent (Mambo et al., 2019), and less than one third of students progressing to secondary school (UNESCO Institute of Statistics). In 2015, the youth literacy rate was only 71 percent, and 90 percent of second graders could not read two words in Portuguese. According to the last grade 3 national reading assessment (INDE/MINEDH 2017), Nampula is one of the provinces that performed below the national average. Children from remote and less developed districts such as Nacarôa and Muecate (our study sites) are more affected. World Vison, Inc. (WV) has been implementing the Educating Children Together (ECT) Program in the Muecate and Nacaroa Districts of the Nampula Province in Mozambique since 2013 with funding from the McGovern-Dole International Food for Education and Nutrition Program under the International Food Assistance Division of USDA. The second phase of Educating Children Together (ECT2) program originally covered the period from December 2016 to December 2020 but a nine months No-Cost Extension (NCE) was approved to allow the program continue until September 30, 2021. During the period of approximately six years of ECT2 project, WV deployed USDA donated commodities (CSB+) and cash resources in the target districts for achieving the following objectives: • Improve the quality of literacy instruction through better-trained and more-available instructors and administrators, and greater access to literacy materials; • Improve student attendance and attentiveness by reducing short-term hunger through provision of a daily school meal; • Increase student attendance by promoting the benefits of education and enrollment; • Improving school infrastructure, and raising awareness on the barriers that can affect school attendance and; • Improve student, community, and school administration knowledge and practices around nutrition, health, hygiene, food safety and storage. This report presents the results of the endline evaluation of the Educating Children Together Phase 2 (ECT2) program in Nampula province, Mozambique, implemented by World Vision (WV) in partnership with Save the Children. The program addresses multiple dimensions of human capital by providing school meals and early grade literacy programming with improvements in the learning environment and health/nutrition education. The evaluation conducted for this report measures the 5 impacts of the school meals and literacy program in 175 rural public primary schools in Nampula; 160 of these schools are treatment schools, and 15 are control schools. Note that ECT2 has been followed by a new round of implementation in the program called ECT3, which is implemented by World Vision and Centro de Aprendizagem e Capacitação da Sociedade Civil (CESC) with continued USDA funding. While ECT2 programming concluded in 2019 and ECT3 was scheduled to start during the 2020 school year, due to the COVID-19 pandemic, ECT3 will only launch fully in the 2021 school year. ECT2 received a nine-month extension, and the first year of ECT3 was implemented in parallel. 1.2 Interventions The ECT2 study design consists of a control district (Murrupula) and two treatment districts (Nacarôa and Muecate). Murrupula received no WV or Save the Children interventions, while Nacarôa and Muecate received school meals (implemented jointly with a water, sanitation, and hygiene program) and an early grade literacy program. World Vision partnered with Save the Children to implement an early grade literacy program called “Literacy Boost” (LB). USDA, through McGovern-Dole , funds school feeding in Mozambique. The school meals consist of a porridge with micronutrients added and are provided to all teachers and students at the school, every school day. Students from different grades take turns to have their meal together. Each school has school cooks to prepare the meals who are trained in sanitary cooking practice, and the cook receives a take-home ration as an incentive. Some of the food served is sourced from the local community from farmer groups. LB included two components: teacher training and community action. Although grade 1-3 teachers were the target of literacy component of ECT2, teachers in upper grades also benefited from training. The training method adopted in this project followed a cascade model, in which trainers of teacher training institutes trained trainers of trainers at the provincial level, which included education technicians from the Provincial Directorate of Education. These trainers trained district level trainers, which included education technicians from the District Services of Education and those trainers then trained local level trainers, which included ZIP coordinators, in charge of training teachers in a more regular basis. During the project, teachers received training three to four times a year, comprising different sections of the LB approach. The teacher training comprised training of teachers on pedagogical techniques for early-grade reading. Teachers were taught the five phases of reading: letter knowledge, sounding out words, reading fluency, vocabulary, and comprehension. Teachers learned to create a print-rich environment in their classrooms and ensure that children remained motivated while learning to read. They were also provided with materials including books and classroom aids. These materials are in the local language, using locally relevant exercises, and are targeted at the appropriate grade level. Teachers also conducted literacy assessments of students at the beginning and end of the school year to assess the level at which students were and measure their progress. This practice also helped the school and government track progress and target resources. In 2016 the project introduced teacher performance awards based on teacher attendance and student academic performance. Awards included bicycles, mattresses, linen, and kitchenware. School councils were trained to monitor teacher attendance in a regular basis, thus complementing the work of school managers. In fact, based on this method, teachers were not only accountable to school managers but also to the communities served by their schools. Lists with results from teacher attendance tracking were publicly displayed in school walls, which, according to project managers, teachers, and other 6 stakeholders, discouraged teachers from being absent as none wanted to see his/her name on those lists. The community action component of the LB program had several elements. First, parents were brought together in groups to learn about and discuss the importance of schooling, and especially of reading. They were taught ways to help their children with reading at home, for example, incorporating reading exercises in daily life, such as during cooking or cleaning. Second, reading camps were established, whereby a literate teenager in the community met with children weekly outside of school (usually on the weekends) to create fun reading exercises and motivate children to read. They were supposed to teach children in the camps according to the reading level at which they are, use fun learning materials (they are provided with some), and engage the children in a different way from learning inside the classroom. There were also community read-a-thons and other community events. The project also established one library in each school and one book bank in each community surrounding the project schools. While school libraries were mainly meant to serve grade 1-3 students and teachers, book banks aimed to serve students and other community members, including emergent, beginner, and experienced readers. School Council members (school managers, teachers or administrative staff) were trained to look after the libraries. Book banks were a key component of the Reading Camps, thus preferably kept in the communities to allow easy access by the readers. ECT2 also trained key stakeholders in health and hygiene knowledge and practices, including immunization, de-worming, pregnancy, food and nutrition, drinking water storage and treatment, use of latrines, personal hygiene (taking a bath and washing hands) and environmental hygiene. The trainees included volunteer teachers and members of school boards, who in turn trained other members of the communities in which treatment schools are located. As a result, during ECT2 implementation, health education sessions were conducted involving mainly students but also community members. The ECT3 project will continue many of these interventions and will include some additional community and school interventions. Additionally, WV will be implementing a slightly updated early grade literacy programme called “Unlock Literacy” (UL), taking over from Save the Children. 1.3 Other programs implemented Following the launch of ECT2, a separate project funded by USAID, Vamos Ler!, launched in 2016. Vamos Ler! is a five-year program that is also targeting early grade literacy. The budget is $73,500,000, and the objective is to serve 800,000 children across 2,800 schools, supporting 11,000 teachers. One of the participating districts is Murrupula district; this fact was not yet public at the point when Murrupula was selected as the control district for this evaluation. Vamos Ler! seeks to build capacity for bilingual education in the target districts, enhancing early grade pedagogical practices that allow students to learn to read in their local language while simultaneously building their knowledge of Portuguese. Specific goals include improving early grade reading classroom instruction, developing early grade literacy materials in local languages, strengthening school management and governance, strengthening early grade reading assessment systems, and increasing parental, family and community engagement in early grade reading. The implementation of this program is relevant for this evaluation as it presumably led to substantial investment and progress in early grade literacy in the control districts to which the ECT2 districts are compared. This is an important point in the interpretation of the findings that will be highlighted at various points in the below report. 7 1.4 Evidence on School Feeding There is a large literature that relates child health and nutrition to the school performance of children. In particular, early childhood stunting (Mendez and Adair 1999) and anemia (Soemantri et al., 1985) have been associated with poor performance on tests and cognitive assessments. Acknowledging this association, many governments have established school feeding programs with the intent of improving school performance through reduced undernutrition. However, despite the popularity of these programs in both developed and developing settings, while there is abundant evidence on their positive impacts on nutrition and enrolment (Alderman and Bundy 2012; Drake et al. 2017), there is a relative dearth of evidence regarding their effectiveness on learning outcomes (Aurino et al. 2020). A randomized trial in 16 rural Jamaican schools found that school breakfasts improved students’ math achievements, and the program had larger effects among undernourished children (Grantham￾McGregor, Chang, and Walker 1998; Powell et al. 1998). A breakfast program in Peru also improved performance on a vocabulary assessment among heavier children (Jacoby, Cueto, and Pollitt 1996). An evaluation of different implementation modalities of the World Food Programme school feeding program in primary schools in camps for internally displaced people in Northern Uganda found that the school feeding increased math scores for girls only (Alderman, Gilligan, and Lehrer 2012). Studies of a school breakfast program in Kenya found that the program improved arithmetic and curricula scores (Neumann et al. 2007; Hulett et al. 2014; Whaley et al. 2003; Vermeersch and Kremer 2005). A school feeding program in India had no average effects on learning (Berry et al. 2018; Krämer, Kumar, and Vollmer 2018), but there was evidence of a 0.2 standard deviation treatment effect on math and reading for students with high attendance. Most of these evaluations have been small in scale, not implemented by government but rather by NGOs, and have not been scaled up. However, in Bangladesh, Ahmed (2004) evaluated the impact of a World Food Program mid-morning snack program for one million children in approximately 6,000 primary schools in highly food-insecure areas. The snack improved test scores by 15.7 percentage points, with larger effects on math. In another large-scale study, Aurino et al. (2020) exploit a re-targeting of the Government of Ghana’s school feeding program to conduct a randomized control trial (RCT) to identify the causal impact of school feeding programs. In the Ghana School Feeding Program, private caterers are awarded contracts to procure, prepare, and serve food to pupils in the targeted schools. Cash transfers (and, recently, electronic payments) are made from the District Assemblies to caterers based on 54 Ghana pesewas per child per day (roughly US$0.33) every two weeks. The authors find that students in schools enrolled in the school feeding program performed 0.15 standard deviations better on standardized math and literacy exams (Aurino et al., 2020). Overall, these studies suggest that while school meals do raise learning outcomes, they may not be sufficient when complementary educational inputs such as teachers, pedagogy, or infrastructure are lacking or are of poor quality. Consequently, the potential for the addition of an early child literacy program to a school meals program may be substantial. 1.5 Evidence on Teacher Pedagogy, Training, and Remedial Education Within many schools in developing countries, teachers still use a “chalk and talk” style of teaching in which instruction takes the form of a lecture with little student interaction (Glewwe and Muralidharan 2016). This method offers little scope to differentiate instruction to account for the large heterogeneity in preparation levels often observed in early grade classrooms. It is likely that such a pedagogy leaves many children behind, and reviews have shown that “teaching at the right level” (TARL) pedagogical interventions generally have positive impacts on test scores in both reading and math. However, there is often some difficulty in getting teachers to adopt different teaching methods (Glewwe and Muralidharan, 2016; Muralidharan, 2017; Beg et al. 2020). 8 TARL interventions provide teachers with instruction and materials to better teach to the level of the students’ ability levels and have been shown to be quite effective. LB’s inclusion of assessment of students to determine reading levels follows this successful practice. One evaluation conducted in Tarlac Province in the Philippines evaluated a read-a-thon program in 100 elementary schools. As part of the program, teachers in government schools were offered age-appropriate reading material, training on how to use the materials, and 31 days of support using the materials. The authors found that at the end of the 31 days of support, children in grade 4 scored 0.13 SD on a reading exam and had read 1.24 more books outside of school in the past month. While these results persisted past the month of support, the effects fell to 0.06 SD and 0.87 books, respectively, 3 months after the read-a￾thon (Abeberese et al. 2014) suggesting that either longer-term of continuous support is needed for gains in learning to persist. The weight of the evidence on teacher training implies that programs can be effective but also shows that choices in program design can have large effects on program efficacy. In a recent review, Popova et al. (2021) conducted a systematic review of 39 separate teacher training programs. Their review shows that teacher training programs generally display gains in child test scores, but that these gains are substantially larger if there are incentives associated with the training, the training is focused on a specific subject, the training takes place in a face-to-face setting, or is focused on lesson enhancement. Longer trainings and those accompanied by follow-up and coaching tend to be quite effective. To complement the recent evidence on in-classroom techniques, there have also been several out-of￾classroom interventions that aim to complement the learning that children achieve within the classroom. This evidence is relevant for understanding the potential effects of reading camps. One seminal work that has had influence beyond education literature tested a remedial education program in which young women, referred to as “balsakhis”, were hired by an NGO called Pratham in India to offer remedial reading education to children in 3rd or 4th grade who were identified as falling behind. The balsakhis met with groups of 15-20 students for roughly two hours a day. During their sessions, the balsakhis taught a curriculum developed by Pratham that was designed to reinforce the skills that children were taught during 1st and 2nd grade. During a trial conducted in Vadodara, India 98 government schools were assigned to receive the balsakhi program in either grade 3 or grade 4 classrooms. Results show that children who received the remedial education scored 0.14 SD higher on a literacy test one year into the program and 0.28 SD higher after two years of implementation. Potentially even more promising for the potential of remedial education is the fact that the results seem to have been driven by children in the low end of the ability distribution (A. V. Banerjee et al. 2007). In a large-scale experiment that took place throughout Kenya during the 2016 school year, villages were randomly assigned to take part in a “cross-age tutoring” intervention in which upper grade volunteers were chosen to tutor lower grade students in either Math or English. The tutoring sessions occurred after school each day for 40 minutes and the tutor assigned was at least 5 years older than the tutee. At the end of the school year, researchers identified a small but positive increase in math test scores among students who received math tutoring (0.063 SD) but do not find positive effects on English test scores among children who received English tutoring (Romero et al. 2021). An objective of reading camps is to increase involvement by parents and communities in literacy promotion. Evidence about the effectiveness of interventions targeting increased parental involvement in the education system is mixed. One trial in Mexican public schools found that an intervention teaching parents ways to be involved in schools did not improve academic achievement (Barrera￾Osorio et al. 2020). One possibility for why these interventions often do not improve academic performance is that even if parents are involved in the child’s education, they are not themselves educated to a level that allows them to aid in the child’s development. Indeed, the literature on intergenerational transmission of human capital suggests that children born to more educated and literate parents have higher academic achievement (Andrabi et al. 2012). Informed by this evidence, a 9 trial in Bihar, India conducted by Pratham aimed to improve child achievement by improving the literacy of mothers. The trial consisted of two separate treatments. In the first, mothers were given access to daily instruction on math and literacy. These training sessions placed greater focus on math and were designed based off Pratham’s Read India program. In a separate treatment arm, mothers were given access to instructional activities to be completed with their children in the home. The activities were designed for children 5-8 years old and intended to improve mothers’ involvement in their children’s education. Finally, in a third treatment arm, mothers were offered access to both the maternal literacy trainings and the instructional activities. During their endline survey, researchers found that children in both the literacy and materials treatment arms scored 0.035 SD better on math tests, and that children born to mothers who received both interventions performed 0.042 SD better on a reading exam and 0.056 SD better on a math exam (Banerji et al. 2017). There have been few studies that combine both a pedagogical intervention with remedial instruction as conducted in Literacy Boost. However, results for one similar intervention are presented in Björkman and Guariso (2021). The intervention, again implemented by Pratham, combined the TARL intervention previously discussed through Banerjee et al. (2016) with community managed study groups. These study groups consisted of roughly 7 students each and were designed to cover topics from previous classes that students were struggling to grasp. To test the program, 200 villages in Assam Province, India were randomly assigned to 4 equally sized groups, in which 50 villages were randomly assigned to receive both the TARL and study group interventions, 50 were assigned to receive only the TARL program, 50 were assigned to receive only the study group program, and the final 50 were assigned to serve as the control group. Upon their analysis, the authors found that when combined, the programs induced a 0.09-0.12 SD improvement on test scores in math and English (Björkman Nyqvist and Guariso 2021). While these results are promising for the prospects of the LB program, it is important to note that the authors do not find any significant effects of either of the stand-alone interventions and are able to rule out the possibility of large effects. 1.6 Objectives and Research Questions The objective of this project is to analyze the effects of a combined school meals and early grade literacy intervention on literacy, nutrition, and other outcomes for children and primary schools in rural Mozambique. The primary research questions are as follows: 1. What is the effect of a school meals and early grade literacy program on the literacy of students who have been exposed to two years of reading instruction? 2. What is the effect of a school meals and early grade literacy program on nutritional outcomes of students? 3. How do effects differ by student gender? The secondary research questions are as follows: 1. What is the effect of the program on teachers’ pedagogical practices with regards to early grade reading; attendance; and access to school supplies and materials? 2. What is the effect of the program on students’ attendance; attentiveness in the classroom; short-term hunger; access to food; and knowledge and use of health, dietary, and sanitation practices? 10 3. What is the effect of the program on school cooks’ knowledge of safe food preparation and storage practices and schools’ access to sanitary facilities? In this report, we present the results of the evaluation of the ECT2 program on each of these indicators. We also note throughout the report that the period of implementation and evaluation for this program encompasses the onset of the COVID-19 pandemic and the associated school closures during 2020-2021. This generates some substantial challenges in interpreting the observed empirical patterns and their relationship with ECT2 implementation, and we have highlighted these challenges throughout our discussion. Chapter 2 describes the methodologies used in the study, Chapter 3 presents the results on program indicators, Chapter 4 presents results on how students, teachers, and schools were affected by COVID-19, and Chapter 5 concludes and provides some recommendations. 2 Methodology We use a mixed-methods approach to evaluate ECT2. First, we provide a quantitative analysis of the ECT2 indicators of interest using data from baseline, midline, and endline points of the program evaluation. Conditional on data availability, we conduct a quantitative analysis either by studying the trends over time of the indicators or testing their differences across treatment and control groups or by using a quasi-experimental approach (difference-in-difference). This last method is only applicable for the EGRA indicator. It is worth noting that we also perform Cohen tests to estimate effect sizes and analyse the reliability of the EGRA indicator. Second, to complement our quantitative analysis, we also use data from semi-structured interviews and focus groups with the main stakeholders of the program, including students, teachers, school principals, parents, farmers, and personnel from World Vision and the Ministry of Education. In the following sub-sections, we explain in detail each of these quantitative and qualitative methods. 2.1 Quantitative Methods We exploit three different data points in our quantitative analysis: baseline, midline, and endline surveys. i). Baseline Data: These school-based surveys were collected by RTI International in 2017 to assess the impact evaluation of the World Vision school feeding and literacy program. The data considered three types of schools: i) those receiving school feeding and literacy interventions; ii) those receiving only the school feeding intervention; and iii) comparison schools not receiving any of these interventions. The data included 448 schools located in Nacarôa, Mucuete, and Mongicual (in Nampula province), and Maputo districts. These data included information from grade 2 and 4 students and teachers, deputy school directors and cooks, and warehouse managers. The final sample consisted of 429 head teachers, 214 cooks, 239 warehouse managers, 412 grade 2 teachers, 394 grade 4 teachers, 3,420 grade 2 students, and 3,289 grade 4 students. ii). Midline Data: These school-based surveys were part of the midline evaluation to assess the implementation and early impact of Educating Children Together (ECT2)-McGovern-Dole program. The data collection was conducted by Ernst & Young in 2018. The surveys were conducted in 18 schools in Muecate and Nacarôa districts in Nampula province. These 18 schools were selected as a representative sample of the 150 schools where the program was implemented at that time. The data included 382 surveys with grade 3 students and 18 interviews with teachers, volunteers, and school￾administrators. Furthermore, 66 additional students’ surveys were conducted in 3 schools located in Murrupula district, selected as a comparison group. The evaluation also included secondary data from school and the District Services, Youth and Technology (SDEJT) records. 11 iii). Endline Data: These data were collected by this research team and comprised school-based surveys conducted by ELIM Serviços, in partnership with IFPRI. These data included 175 schools in Nampula province: 160 in Nacarôa and Muecete districts, where the program is currently present, and 15 schools in Murrupula district, which were chosen as a comparison group for the ECT2 program. A random sample of 10 grade four students was selected within each school, conditional on parental consent. In addition, we also interviewed the grade 4 teachers, the deputy school directors (equivalent to the school principal), and the school cooks, who prepares the school meals. The target cohort for this evaluation was grade three students. However, given that endline data was collected in the summer of 2021, following a nearly year-long interruption in schooling, the baseline survey was conducted with grade four students. Grade four students graduated from grade two but received only minimal instruction in grade three in the 2020 school year due to COVID-related closures. Like students entering grade 3, they only had 2 years of literacy instruction and accordingly, their literacy level should proxy for grade three literacy in future cohorts. The final sample included 5,582 grade 4 students. In addition to the students, 175 headteachers, 175 teachers, and 157 school cooks were interviewed. Table 1: Data availability by district across treatment and control groups in baseline, midline and endline Panel A. Data availability by District Treatment Districts Muecate, Nacarôa Muecate, Nacarôa Muecate, Nacarôa Control Districts Maputo districts, Mogincual and Meconata in Nampula Murrupula Murrupula Panel B. Final Survey Sample Baseline Midline Endline No. schools in sample 448 18 175 No. of students interviewed Grade 2: 3,420 Grade 4: 3,289 Grade 3: 448 Grade 4: 5,582 No. of teachers and deputy school directors interviewed Head teachers: 429 Grade 2 teachers: 412 Grade 4 teachers: 394 School Administrators: 18 Grade 3 teachers: 18 Headteachers: 175 Grade 4 teachers :175 No. of other stakeholders School cooks: 214 Warehouse managers: 239 School volunteers:18 School Cooks: 157 As is evident in this table, one of the main challenges of the quantitative analysis is that we do not have information on the same districts in the comparison group across the three points of time; baseline, midline, and endline. For instance, we lack information on Murrupula district at baseline. Furthermore, for some indicators, their measurement and construction are not consistent across the three points of time. Due to these data limitations, we perform the following three types of analysis: a. Difference-in-Difference Method: We can implement this quasi-experimental approach only for the EGRA indicators for which we have raw data in both midline and endline across the same treatment 12 and control districts. Where raw data is not available, variable means from the ECT2 baseline and midline reports are used (see b and c). This approach captures the average learning effect based on a difference-in-differences coefficient (DID). DID is an approach that compares the average changes in the indicators before and after implementation, for schools and students between the intervention and control groups. DID essentially compares intervention and comparison groups in terms of change (if any) in the outcome variables that occur between the midline and the endline. Figure 1 below summarizes the study design. Figure 1: Difference in Differences Outcome In such a DID design, the difference in outcomes before and after the intervention for the comparison group (D-C) is subtracted from the change in outcome for the treatment group (B-A); or equivalently, the difference in outcomes between treatment and comparison schools at baseline is subtracted from the difference in outcomes between treatment and comparison schools at follow-up, i.e., DID Program Impact = (B-A) - (D-C). As Table 1 shows, we lack the same comparison districts in the baseline and endline points, so we compare endline and midline. Changes in treatment school outcomes between endline and midline are compared to changes in outcomes of control schools between endline and midline. In effect, the DID method isolates the differential change in outcomes due to the intervention by controlling for any changes in the absence of the intervention. The estimation follows the setup: (T endline – C endline) – (T midline – C midline), where T stands for treatment group and C stands for the control group. For treatment estimates to be causal, the assumption is that the outcomes of the treatment and control groups would have evolved similarly over time without any treatment. This common trend assumption cannot be directly tested in these data (especially in data with only two points in time). A potential concern is that if pre-program observable characteristics of the treatment and comparison Timeline Program impact Baseline Time of intervention A Treatment schools B C Control schools D Endline 13 schools are very different, they might experience dissimilar changes in their outcomes, independent of the ETC2 program. We have some information to estimate whether it is plausible to assume that comparison and treatment districts were following a parallel trend in the absence of the treatment and the data suggest that they were likely not following a parallel trend (see Section 3.1, below). Consequently, the DD results presented here do not estimate causal impacts. Combining midline and endline data being collected for the treatment and comparison group, DID can be estimated using the following regression approach (which is the multivariate regression equivalent to Figure 1): 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂 = 𝛼𝛼 + 𝛽𝛽𝛽𝛽 + 𝛾𝛾𝛾𝛾 + 𝛿𝛿( ∙ ) + 𝜀𝜀. The left-hand side of the equation is the outcome variable of interest (for example, reading fluency). The variables on the right-hand side include:  A dummy variable which equals 1 if the observation is a treatment school and zero otherwise. The estimate of 𝛽𝛽 captures the treatment effect. In other words, controls for any differences in the outcome variable that are associated with being in the treatment group.  A dummy variable which equals 1 in the follow-up year (endline) and zero in the midline year. The estimate of 𝛾𝛾 captures the time effect. In other words, controls for any changes in the outcome variable that occurs over time and is common for treatment and comparison group schools.  An interaction term ( ∙ ) which equals 1 if the observation is in the treatment school and in the follow-up year and zero otherwise. The estimate of 𝛿𝛿 captures the impact of the project on the outcome variable—this is the parameter of interest. b. Analysis of trend indicators: We track indicators over time only in the treatment districts (Nacarôa and Muecate) for baseline, midline, and endline. It is worth noting that this analysis is limited to describing the trends of the indicators in the treatment group. Thus, we are not able to infer any causality for such trends as we lack the comparison group data. The pre and post-test (only treatment group) approach was used to evaluate the trend of the project indicators. Due to lack of raw data for the baseline of some indicators, means of indicators from the ECT2 baseline and midline reports were used. Mean differences between the baseline and endline were estimated, and the standard error of the mean difference was computed using the following formula: SE = (SE2 baseline +SE2 endline )1/2 assuming independence of the variables between the baseline and endline. A statistical significance of p<0.05 was considered if the difference taken in absolute terms was more than twice (1.96) its standard error (SE). Confidence intervals (95%) around the mean and Cohen’s d was used to assess the statistical significance and the magnitude of change of variables related to the indicator1. 1 Cohen’s d is simply a measure of the distance between two means, measured in standard deviations. The formula used to calculate the Cohen’s d looks like this: . For proportions, the Cohen’s d is estimated by the 14 c. Test of Differences: Using only the endline data, we test for statistically significant differences between Nacarôa and Muecate (the ECT2 treatment districts) and Murrupula (the ECT2 control district) at endline. It is important to note that we cannot infer a causal effect of the program using this analysis. Because the allocation to treatment and control was not determined randomly, the comparison between treatment and control groups at endline does not rule out any selection bias; the estimated difference between groups might be confounded with other observed and unobserved variables. In fact, as we will see in Section 3, Murrupula district is quite different from Nacarôa and Muecate in many ways. Parametric (Independent Samples T-Test) and non-parametric tests (Mann-whitney and Chi-square) were used to compare the intervention and control groups. When these tests were statistically significant, Cohen’s (1988) standardized effect size (also known as Cohen’s d) was used to assess the magnitude of change in variables related to the indicator. According to Cohen’s estimates, d > =0.8 is regarded as a large effect, d between 0.5-0.7 is a medium effect, and d <0.5 is a small effect. 2.1.1 EGRA Description The Early Grade Reading Assessment (EGRA) was developed in 2016 by RTI international to measure children’s progress towards reading mastery. In our study, we use the EGRA to measure differences in the effectiveness of the ECT2 school meals and pedagogical techniques interventions. The EGRA is a 15-minute assessment that measures a student’s aptitude in 5 crucial reading subdomains. The EGRA offers multiple instruments based on purpose, but in the version of the EGRA that we use, we test student’s oral vocabulary, listening comprehension, letter recognition, familiar word reading, and reading comprehension. The different sections of the instrument are described below. Oral vocabulary: This subtask measures children’s oral receptive and production language skills of individual words and phrases related to body parts, common objects, and spatial relationships. The sub-task included two sub-parts: in the first subpart, eight prompts that required children to perform an action (e.g., “show me your knee”) were used to determine their level of understanding of basic Portuguese oral vocabulary; in the second subpart, children were requested to follow six instructions given orally (e.g., “place the pencil next to the paper”) and perform the required actions. The score is the number of prompts and instructions correctly performed (maximum score of 14). Concepts about printed material: This subtask measures children’s emergent literacy skills by asking them to demonstrate their readiness to handle and read print material. The children were asked 10 questions assessing recognition of the front and back covers of a book, the direction in which to read, the title of a story in a book, the page numbers in which a given story is located, etc. The score is the number of correct answers given. Letter name identification and reading: This subtask is used to measure whether children can identify and read letters presented in random order, both lowercase and uppercase. In this subtask, children were presented with a chart of 100 letters and asked to read as many of these letters as they could within one minute. The score is the number of letters correctly identified and read in one minute. Familiar words reading: This subtask assesses the ability of children to decode printed words and read them correctly. The task reflects both the accuracy and fluency of reading, which are fundamental skills for developing the ability to read and comprehend what is read. The children were presented with a card containing 30 words common to their daily life, including their school life, and asked to following formula: If we assume that P1 and P2 represent the two proportions. The effect size is represented by the difference h formed as follows ℎ = 𝜑𝜑1 − 𝜑𝜑2 where 𝜑𝜑𝜑𝜑 = 2 arcsine Pi 0.5. 15 read as many words as possible in one minute. The score is the number of words correctly read in one minute. Listening comprehension: This subtask is used to measure whether children have basic knowledge of the language in question and whether they can process what they hear in that language. In this subtask, the enumerators read aloud a short text comprising 56 words for the children and then asked them four questions to check their comprehension. The children were not given a copy of the text to refer to when answering the questions. The score is the number of correct answers given by the children to the four questions asked. Oral reading fluency: This subtask assesses the speed, accuracy, and expressivity at which children read texts. The task reflects the ability to translate letters into sounds, recognize familiar words, decode unfamiliar words, and make sense of the text’s meaning. The children were given a card with a narrative text of 120 words and asked to read as many words as possible in one minute. The metric of oral reading fluency was the number of correct words per minute (cwpm) read by the student. Reading comprehension: This subtask assesses the ability of the children to extract and construct meaning out of the texts they read. Studies have shown that oral reading fluency is a predictor of reading comprehension (e.g., Daane et al., 2005; Abadzi, 2011), hence the relationship between EGRA oral reading fluency and reading comprehension subtasks. After reading the narrative text used as the stimulus for the oral reading fluency subtask, the children were asked up to five questions based on how much text they had read. For example, the enumerator only asked the first question to those children who managed to read at least nine words of the text given. The score is the proportion of comprehension questions correctly answered. As a tool to assess early reading skills, EGRA is often administered to early grades (1-3). However, grade 4 students were tested for the ECT2 endline. It should be noted that grade 2 and 4 students took the test for ECT2 baseline study, while grade 3 students took the test for midline evaluation. As mentioned before, the decision to administer the test to 4th graders in the endline evaluation was because these students did not attend grade 3 classes in 2020 due to school closures in the context of COVID-19. For various reasons, for some grades that do not undergo national examinations (e.g. grades 1, 2, 3, 4 and 5), the government automatically promoted all students to the next grade. As a result, the students in grade 4, when the 2021 school year started, had only been exposed to literacy classes for two years (grade 1 and 2). So, in practice, they can be compared to grade 3 students in a normal situation. WV requested that the students tested had had at least two years of exposure to literacy development in the school context. 2.2.2 Reliability of the EGRA Test Reliability refers to how dependably or consistently a test measures a characteristic. Reliability in item response theory (IRT) measures the extent to which the measure is independent (free) from groups (samples) as well as from the test items; in other words, if we apply many versions of the test for the same group, they must get the same score and same ranking (Lord, 1980). The reliability of a test is indicated by the reliability coefficient and is denoted by "α". It is expressed as a number ranging between 0 and 1.00, with α = 0 indicating no reliability, and α = 1.00 indicating perfect reliability. In other words, the higher the α coefficient, the more likely the items measure the same underlying concept. A minimum α coefficient between 0.65 and 0.8 (or higher in many cases) are considered indicative of a reliable assessment; α coefficients that are less than 0.5 are usually unacceptable. Table 2 shows the reliability coefficients for each subtask and the entire EGRA. In all subtasks except reading compression, in the intervention group, the reliability is within the range of the recommended values. The α reliability varies from 0.64 to 0.98, in the intervention and control groups, except for 16 reading comprehension in the intervention group, which is below 0.5 (but it is quite close at 0.45). The α coefficient for the entire EGRA test is around 0.60 for both the control and intervention groups (0.59 and 0.58 respectively, see Table 2). One can argue that the measurement is independent (free) from groups (samples) as well as from the test items. Table 2: Cronbach’s alpha reliability coefficient Subtask Intervention Control Letter name identification and reading (100 items) 92.5 92.0 Familiar words reading (30 items) 88.7 82.7 Listening comprehension (4 items) 0.64 0.75 Oral reading fluency (161 items) 0.93 0.98 Reading comprehension (4 item) 0.45 0.71 Full EGRA 0.58 0.59 2.2 Qualitative Methods The qualitative component of the ECT2 endline evaluation included focus groups and structured interviews from grade 4 students and teachers, deputy school directors, parents, farmers, and World Vision and government employees affiliated with the program. The school-based sample for this qualitative component consisted of the schools that ELIM had already visited for the endline survey data collection in the districts of Nacarôa, Muecate, and Murrupula. Five schools were chosen among this school sample based on geographic convenience and all other schools in the ZIP having been surveyed. There were six focus groups with grade four students (three focus groups with boys and three focus groups with girls). In each selected school, 5 to 6 students in grade 4 were randomly selected for the focus groups. These students had prior parental consent to participate in the study. This qualitative data collection also included three focus groups with school teachers and three focus groups with deputy school directors, one in each district; each focus group included five staff members, each staff member from a different school. Furthermore, three focus groups were conducted with farmers’ groups, one in each district of interest. In-depth interviews with World Vision stakeholders were conducted, including two education specialists, a monitoring and evaluation officer, and a project coordinator. Data collected was as listed below. Table 3: Qualitative Data Collected, by District Nacarôa: Number of focus groups Population category Male Female Total 2 Students 6 6 12 1 Deputy Directors 4 1 5 1 Teachers 3 2 5 1 Farmers 5 4 9 5 Total 18 13 31 Muecate: Number of focus groups Population category Male Female Total 2 Students 6 6 12 1 Deputy Directors 2 3 5 1 Teachers 1 4 5 1 Farmers 3 2 5 17 1 Parents 3 2 5 6 Total 15 17 32 Murrupula: Number of focus groups Population category Male Female Total 2 Students 6 6 12 1 Deputy Directors 3 1 4 1 Teachers 4 0 4 1 Parents 3 5 8 5 Total 16 12 28 Semi-structured interviews: 3 World Vision Employees (1 Project Coordinator, 1 Education Specialist and 1 M&E Manager) 3 Government Officials (2 Education Officials and 1 Health Official) 2.3. Data Collection Methods: ECT2 Endline The endline surveys for the ECT2 program were collected between June 30th, 2021 and August 6th, 2021 by ELIM Serviços Lda, a survey firm based in South Africa with offices in Mozambique, in partnership with IFPRI. The overall sampling frame for this study is constituted by World Vision’s target schools in the study districts. The surveys included 175 schools throughout Nampula province, 83 in Muecate, 77 in Nacarôa, and 15 in Murrupula as the control district. Murrupula district was chosen as the control district back in 2015 when it did not have any interventions similar to the interventions under the ECT program. At that time, it was socio-economically not very different from where the project is being implemented. However, over the past 5 to 6 years there have been significant interventions in the district especially in the education sector with funding from USAID. It has continued to serve as the control district though currently contaminated because it could not be changed technically. In the future, WV would have to proactively monitor changes in such control districts and where necessary make some changes by identifying alternative control districts as much as feasible within the target province. The survey collected information on four key respondents for each of the 175 school: i) 4th grade students, ii) 4th grade teachers, iii) deputy school directors, and iv) school cooks. The surveys had approval by the IFPRI Institutional Review Board as well as the Comité Nacional Bioética em Saúde (CNBS) in Mozambique. A class list was collected for all children in grade 4 to obtain information on class size and attendance. This list also included information on the child’s age and biological sex, as well as information on school closings due to COVID-19 and attendance before the lockdown. If a student was absent on the day of the class list, information was collected from those present about the student’s enrolment status. Following the listing, 10 students who had parental consent from every grade 4 class were randomly selected to respond to a more in-depth survey. Each student in the class list among those who could take the survey (i.e., with parental consent and with no disabilities) was assigned a randomly generated number. The 10 students with the lowest numbers assigned were administered the survey. If one of the children selected for the assessment was no longer present or removed consent, the child with the next lowest random number was administered the exam. These grade 4 students’ surveys collected information on family demographics, socioeconomic and sociolinguistic background, students’ schooling, literacy activities out of school, food sufficiency and nutrition, hygiene practices, and child labor activities. The survey also collected data on the socioeconomic impacts of COVID-19 on schooling and households. After the socio-demographic information, the students took the EGRA test, described in detail in section 2.2.2. The survey of the deputy school director collected information on the demographic characteristics of the deputy director, student and teacher attendance, school facilities and characteristics, and the 18 school’s participation in previous programs. From teachers, information was collected on demographics, training, participation in school programs, the presence of literary materials in the teacher’s classroom, and the impacts of COVID-19. Finally, where schools employed a school cook and offered school meals, information was collected on the school’s meal program from the school cook. In total, data was collected on 5,582 grade 4 students. Of the students in attendance, 1,671 grade 4 students were administered the EGRA. In addition to the students, 175 head teachers, 175 teachers, and 157 school cooks were interviewed2. 2.3.1 Training and Data Collection Data collection was carried out by a team of 20 interviewers from ELIM. These enumerators were divided into 4 teams with 4 enumerators and one supervisor each. Each team will collect data in 2 schools per day. Enumerators were trained on the questionnaires in a 7-day training event, which included one day in the field in non-study sites to practice interviewing. During the training, all enumerators were trained on how to administer each questionnaire, the logistics of the school visits, and on research and data collection ethics. They were trained in how to carry out the interview, including line-by-line explanation and interpretation of the questionnaire, the flow and skip-patterns, definitions, and explanations of how to handle unusual cases and when to contact the supervisor for assistance. Supervisors participated in the enumerator training but also received additional training related to their supervisory role. This included detailed and special training on how to obtain informed consent for child respondents – including a detailed protocol for obtaining parental approval. Moreover, we emphasized that when interviewing respondents, enumerators should emphasize that the respondent does not need to answer any particular question should they not want to. All required World Vision training on child protection was also provided. Careful quality assurance protocols were used to ensure fidelity to high-quality data collection principles. Supervisors monitored enumerators’ work and directly observed interviews as appropriate. Regular data checks were conducted by ELIM and by IFPRI to identify any anomalies in the data. In addition, there was careful adherence to consent procedures to ensure that all households have the opportunity to provide informed consent for the participation of their children. As earlier mentioned, these procedures were specified in submissions to the IFPRI Institutional Review Board as well as the Comité Nacional Bioética em Saúde in Mozambique and approved by both committees. Consent was sought from the parents of students as follows. Before starting fieldwork, a verification exercise took place whereby the survey firm visited schools to introduce themselves and collect information on school and class timings to organize fieldwork better. At that time, consent forms were dropped off with the school director, who sent the forms home with students. Students then returned the signed forms (if the parent consented) before the school visit. For adults who were surveyed, written consent was sought on the day of the survey and were provided with a consent statement that included: 1) objectives of the study; 2) study procedures; 3) risks and benefits of participating in the study; 4) strategies used by researchers to minimize risks; 5) costs/compensation associated with participating in the study; 6) the duration of the interviews, 7) the voluntary nature of the study and the participant's right to refuse to answer questions or leave the study; 8) that all information would be confidential, that nobody would be able to identify any particular individual's responses, and that their data would be kept securely; and 9) contact information for study staff. The consent statement was also written in Emakhuwa, the local language, to facilitate comprehension. The signed copy was retained by the supervisor, and a copy was provided to the respondent to take home. Their consent was recorded in the CSPro software, whereby a box was checked if the respondent consents to 2 Note that schools in Murrupula did not have school cooks since they did not have a school meals program. 19 participate. Respondents had the opportunity to ask any questions and raise any concerns, and the enumerators were prepared to address any issues. 2.3.2 Data Management and Security The enumerators interviewed the respondents individually. All data was collected on tablets and provided to the project manager and research team each night via a password-protected Dropbox folder. Interviews were not recorded as consent was not sought for recordings, but rather enumerators typed responses into the tablet. The project manager conducted quality checks of the data collected. The data was kept archived with the main IT specialist of the survey firm. Only the project manager and the Director of the company had computer access to these files. Only authorized individuals have access to the dataset with identifiers, which are secured through a combination of restricted dissemination of information and storage in a password protected file. Original data containing the identities of the respondents is not shared with any other institutions apart from IFPRI. Team members involved with this project are made aware of this provision. Names and other easily recognizable identifiers were entered with IDs in a separate electronic file from all other data. This electronic file containing names is held separately from all other data files and is kept only by managers of the project at IFPRI. This information is kept in electronic format so that it can be used easily to help find respondents for the next round of data collection. Study identifiers (school and individual IDs) are included in each data file so that data from the several instruments collected within a school may be linked together and with future survey rounds. However, these are not meaningful to casual observers without access to the original study logs. All data files are always maintained under password protection. Public use data will include no identifiers. Protection of personally identifiable information is a key priority for this evaluation. As noted above, PII is only accessible to the project managers and the research assistant engaged in data cleaning and is stored in a secure electronic environment. No PII has been shared with external stakeholders, policymakers, or with World Vision, and these provisions will hold for the duration of the project. 2.3.3 COVID-19 Specific Precautions Following approval to conduct the survey, we implemented several measures to protect enumerators and participants. Enumerator training was conducted in a large space that ensured that enumerators were 6 feet apart. Everyone was required to wear masks. When possible, training was moved outside. We trained enumerators on measures to be taken and the importance of compliance to these COVID￾19 safety measures. There was strong supervision of compliance by the supervisors and by the field manager. Enumerators were required to terminate an interview in case of non-compliance. All enumerators were tested daily with a no-touch thermometer (thermo-gun) to determine whether they had a fever. Those showing signs of illness were not permitted to work and returned home until they were well. All enumerators were guaranteed paid sick leave. All enumerators used Personal Protective Equipment (PPE). They were provided with masks that they changed daily as well as gloves and hand sanitizer. They cleaned their hands with sanitizer before and after each interview. Enumerators were also instructed not to shake hands with anyone and to use an alternative form of greeting instead. Precautions being made for COVID-19 were explained to respondents, as well as potential risks. Enumerators emphasized that participation is voluntary if there is any discomfort. 20 All interviews took place outside when possible or in a well-ventilated space. For the class lists, the students were moved outside the classroom and the enumerator remained 6 feet away from the teacher and students. For the one-on-one interviews, teachers, students, and school cooks were interviewed outside, so that enumerators could remain 6 feet apart from respondents. The enumerators emphasized the need for distancing during the interviews to other people who may be nearby. Testing students on the EGRA requires that the enumerator and student be in relatively close proximity because the enumerator must be able to see what students are pointing to on the cards. Students were provided with separate cards from which they read letters, words, and texts; the cards were laminated, and the enumerator followed the student reading while entering information on the tablet. In addition, we acquired clear plastic screens to set up on a table to separate the enumerator and the student, while also enabling them both to see the tablet and cards clearly. Tablets and laminated cards were sanitized between each interview and students also sanitized their hands after the interview. 2.4 Evaluation Constraints and Limitations The final impact evaluation of the ECT2 program is limited by the following factors. First, the estimates presented here cannot be interpreted as "causal" effects of the program on the outcomes of interest. The lack of a random comparison group across the treatment and control groups prevented fully implementing a quasi-experimental evaluation method, such as difference-in-differences to estimate a counterfactual scenario; i.e., what would have been the trajectory of the indicators in the absence of the program. While DID estimation is conducted, it is not possible to test the assumption of parallel trends (which is necessary to interpret effects as causal) with the data. Consequently, we caution against interpreting the results presented in this report as causal impacts of the program without any potential confounders. Second, the endline data was collected at the beginning of the 2021 school year, after more than one year of COVID-19 related school closures. Thus, after returning to school, children were potentially not exposed to the ECT2 program for long enough for the data to capture the potential mitigating effect of the program on the learning loss due to the pandemic. Much previous learning would have been lost while schools were closed, hampering the student’s ability to learn when schools reopened. Furthermore, the impact of COVID-19 on children's well-being goes beyond the school interruption as the pandemic has increased their households' food insecurity and poverty conditions. Therefore, the detrimental effects of COVID-19 on the target children and their households' socio-economic conditions need to be considered when interpreting the ECT2 evaluation results. 3 Results on ECT2 Indicators This chapter will present the results of the evaluation of the ECT2 program. We will report on indicators that were requested by WV in the Terms of Reference (TOR) signed, including literacy, nutrition, and water, sanitation, and hygiene (WASH) indicators. We will tie indicators to the McGovern-Dole Results Framework, grouping indicators by Strategic Objectives. In addition to comparing indicators to their targets, we also report cumulative numbers for items provided. As mentioned in chapter 2, we present three types of analysis. For the EGRA tests cores only, we calculate the differences between the treatment and control groups between midline and endline using DID analysis. The estimation follows the setup: (T endline – C endline) – (T midline – C midline), where T stands for treatment group and C stands for control group. This is because we have the raw, 21 individual level data only for the EGRA and not for the other indicators. The baseline and midline indicators for other outcomes are derived from the baseline and midline reports. Second, we examine trends over time in the two treatment districts between midline and endline. We compute statistically significant differences over time. Finally, we calculate differences between the treatment and control groups at endline. It is important to note two key points that are relevant to the interpretation of the results. First, Murrupula was not chosen randomly as a comparison district in the framework of a randomized trial; it was purposively selected. Nacarôa and Muecate were chosen as targets for the ECT2 program precisely because they were characterized by more challenges in nutrition, literacy, and school performance for children. Accordingly, we can expect there to be differences between the treatment and control districts that are not attributable to the ECT2 intervention. Second, the COVID-19 pandemic and associated school closures in 2020 constitute a huge disruption between the midline and endline of this evaluation, and presumably had a range of effects on students’ achievement as well as other indicators. The relevance of these two key points will be highlighted throughout this analysis and discussion. 3.1 Summary Statistics We begin by summarizing characteristics of students, teachers, and schools in the three districts of interest. The tables below provide information on demographics of students, teachers, and deputy school directors, as well as teacher practices and school facilities. The student characteristics also provide a preliminary view of the EGRA scores of students in the three districts. The data in Table 4 column 4 shows that, on average, the students are about 12 years old, 47 percent are female, and one third of students work. Unfortunately, the average number of meals consumed per day is 2.3, inclusive of the school meal; 42% report being hungry during the day. Also notable is that less than 3 percent of students speak Portuguese at home, instead using a local language (predominantly Emakhuwa); this indicates a mismatch between the language of instruction in school versus that spoken at home. There is also an immediately noticeable trend in Table 4 comparing column 1 with columns 2 and 3. In Murrupula district, many students appear to be better off. Students in Murrupula score higher on the EGRA and report going hungry due to COVID-19 less frequently. 22 Table 4: Characteristics of Students in the Sample (1) (2) (3) (4) t-test t-test t-test Murrupula Nacarôa Muecate Total Difference Difference Difference N=147 N=724 N=798 Mean/SE (1)-(2) (1)-(3) (2)-(3) Child's Age 12.254 11.332 11.581 11.534 0.922*** 0.672** -0.249 [0.313] [0.139] [0.129] [0.092] Child's Biological Sex 1=Male 0.507 0.448 0.489 0.473 0.059 0.018 -0.042 [0.053] [0.019] [0.022] [0.014] Correct Answers on Body Part EGRA Section 4.626 3.419 2.271 2.976 1.207*** 2.355*** 1.148*** [0.328] [0.178] [0.136] [0.119] Correct Answers on Pencil EGRA Section 4.136 2.700 2.001 2.493 1.436*** 2.135*** 0.699*** [0.240] [0.148] [0.130] [0.103] Correct Answers on Letters EGRA Section 27.585 16.140 13.555 15.912 11.446** 14.030*** 2.584 [4.654] [1.474] [1.205] [0.990] Correct Answers on Words EGRA Section 6.769 2.536 2.190 2.744 4.233*** 4.578*** 0.345 [1.267] [0.259] [0.228] [0.213] Correct Answers on Tale EGRA Section 0.429 0.095 0.085 0.120 0.333*** 0.343*** 0.010 [0.098] [0.015] [0.016] [0.015] School Attendance Over Last Week 3.122 3.613 3.618 3.572 -0.491** -0.495** -0.005 [0.186] [0.084] [0.089] [0.059] Child Works (1=Yes) 0.293 0.322 0.308 0.313 -0.029 -0.016 0.014 [0.059] [0.025] [0.023] [0.016] Average Number of Meals 2.381 2.235 2.365 2.310 0.146 0.016 -0.130** [0.100] [0.041] [0.036] [0.027] Hungry During the School Day (1=Yes) 0.524 0.436 0.396 0.425 0.087 0.128** 0.040 [0.053] [0.024] [0.025] [0.017] Has Books to Read at School (1=Yes) 0.660 0.686 0.654 0.669 -0.027 0.006 0.032 [0.068] [0.024] [0.024] [0.017] Parents Help with Homework (1=Yes) 0.646 0.508 0.605 0.567 0.138*** 0.041 -0.097*** [0.047] [0.022] [0.022] [0.015] Belongs to a Reading Camp (1=Yes) 0.204 0.298 0.252 0.268 -0.094*** -0.048 0.046 23 (1) (2) (3) (4) t-test t-test t-test Murrupula Nacarôa Muecate Total Difference Difference Difference N=147 N=724 N=798 Mean/SE (1)-(2) (1)-(3) (2)-(3) [0.025] [0.021] [0.024] [0.015] Went Hungry Due to COVID (1=Yes) 0.510 0.610 0.640 0.616 -0.100* -0.130** -0.030 [0.054] [0.026] [0.027] [0.018] Lives with Mother (1=Yes) 0.803 0.855 0.811 0.829 -0.052 -0.008 0.044** [0.028] [0.016] [0.015] [0.010] Lives with Father (1=Yes) 0.735 0.749 0.701 0.724 -0.014 0.034 0.048* [0.028] [0.018] [0.020] [0.013] Speaks Portugese at Home (1=Yes) 0.014 0.035 0.025 0.028 -0.021 -0.011 0.009 [0.009] [0.011] [0.006] [0.005] F-test of joint significance (F-stat) 3.287*** 4.022*** 5.616*** F-test, number of observations 829 924 1469 Notes: The value displayed for t-tests are the differences in the means across the groups. The value displayed for F-tests are the F-statistics. Standard errors are clustered at the school level. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 24 Table 5: Characteristics of Teachers in the Sample (1) (2) (3) (4) t-test t-test t-test Murrupula Nacarôa Muecate Total Difference Difference Difference N=15 N=77 N=83 N=175 (1)-(2) (1)-(3) (2)-(3) Teacher's Gender (1=Male) 0.200 0.195 0.301 0.246 0.005 -0.101 -0.106 [0.107] [0.045] [0.051] [0.033] Teacher's age 34.000 32.078 31.578 32.006 1.922 2.422 0.500 [2.584] [0.850] [0.754] [0.561] Teacher has High School Education 0.867 0.818 0.759 0.794 0.048 0.108 0.059 [0.091] [0.044] [0.047] [0.031] Years as a Teacher 9.467 8.208 7.434 7.949 1.259 2.033 0.774 [1.729] [0.708] [0.714] [0.483] Teacher Groups Students by Ability 0.267 0.104 0.060 0.097 0.163 0.206* 0.044 [0.118] [0.035] [0.026] [0.022] Teacher has Access to Reading Curriculum (1=Yes) 0.667 0.779 0.675 0.720 -0.113 -0.008 0.105 [0.126] [0.048] [0.052] [0.034] Teacher Participated in Literacy Boost (1=Yes) 0.067 0.662 0.602 0.583 -0.596*** -0.536*** 0.060 [0.067] [0.054] [0.054] [0.037] Number of Males in Class 29.467 36.325 34.687 34.960 -6.858 -5.220 1.638 [3.150] [3.880] [3.709] [2.461] Number of Females in Class 24.200 30.792 29.229 29.486 -6.592* -5.029 1.563 [3.128] [1.705] [3.221] [1.721] School Can Practice COVID Protection (1=Yes) 0.667 0.571 0.699 0.640 0.095 -0.032 -0.127* [0.126] [0.057] [0.051] [0.036] F-test of joint significance (F-stat) 3.588*** 2.745*** 1.076 F-test, number of observations 92 98 160 Notes: The value displayed for t-tests are the differences in the means across the groups. The value displayed for F-tests are the F-statistics. Standard errors are clustered at the level of the school. ***, **, and * indicate significance at the 1, 5, and 10 percent critical levels. 25 Table 6: Characteristics of Deputy Directors and Schools in the Sample (1) (2) (3) (4) t-test t-test t-test Murrupula Nacarôa Muecate Total Difference Difference Difference N=15 N=77 N=83 N=175 (1)-(2) (1)-(3) (2)-(3) Director's Age 44.267 40.519 36.349 38.863 3.747 7.917*** 4.170*** [2.150] [0.864] [0.734] [0.578] Director's Gender (1=Male) 1.000 0.883 0.855 0.880 0.117*** 0.145*** 0.028 [0.000] [0.037] [0.039] [0.025] Director has High School Education 0.867 0.922 0.867 0.891 -0.055 -0.001 0.055 [0.091] [0.031] [0.037] [0.024] Director Teaches at School 1.000 0.974 0.928 0.954 0.026 0.072** 0.046 [0.000] [0.018] [0.029] [0.016] Years as a Teacher 17.067 16.247 12.386 14.486 0.820 4.681*** 3.861*** [1.520] [0.844] [0.608] [0.509] Director observes classes (1=Yes) 0.867 0.974 0.952 0.954 -0.107 -0.085 0.022 [0.091] [0.018] [0.024] [0.016] Grade 4 Teacher Attended 80% of Previous 5 days 0.933 0.714 0.795 0.771 0.219** 0.138* -0.081 [0.067] [0.052] [0.045] [0.032] School Participated in Literacy Boost (1=Yes) 0.000 0.974 0.964 0.886 -0.974*** -0.964*** 0.010 [0.000] [0.018] [0.021] [0.024] Number of Males in Grade 4 35.667 44.805 37.313 40.469 -9.139 -1.647 7.492 [5.225] [3.720] [3.123] [2.262] Number of Females in Grade 4 31.067 42.065 32.422 36.549 -10.998 -1.355 9.643** [5.691] [3.924] [2.828] [2.260] Percent of Pupils Who miss more than 10 days 5.800 8.130 6.627 7.217 -2.330 -0.827 1.503 [1.789] [1.507] [0.874] [0.796] Number of Teachers in School 6.733 8.701 7.169 7.806 -1.968 -0.435 1.533 [1.395] [0.959] [0.576] [0.518] F-test of joint significance (F-stat) 70.710*** 41.636*** 3.500*** F-test, number of observations 92 98 160 Notes: The value displayed for t-tests are the differences in the means across the groups. The value displayed for F-tests are the F-statistics. Standard errors are clustered at the level of the school. ***, **, and * indicate significance at the 1, 5, and 10 percent critical levels. 26 Additionally, our data shows that 63% of the students interviewed reported household hunger due to COVID-19, and 45% had to sell goods to buy food, against 51% and 29% in control schools, respectively. While we did not collect data that would allow us to identify the precise reason that household hunger have increased, plausible channels include the loss of non-agricultural employment, limited access to markets due to restrictions on free movement (either externally imposed via government regulations, or self-imposed), and the disruption in the provision of other forms of social support services. These statistics demonstrate that Nacarôa and Muecate districts may have been harder hit by COVID-19. Table 5 shows that 75 percent of teachers are female. Teachers on average are 32 years old, 80% have a high school education and they have about 8 years of experience teaching. We also see some trends that differ in Murrupula. Teachers are slightly more likely to have completed high school, have slightly more experience, and are more likely to employ higher quality pedagogical techniques (like grouping students by ability). Turning to Table 6 we see that deputy school directors are about 38 years old on average, are overwhelmingly male (88 percent), and have approximately 14 years of experience. Almost 90 percent have a high school education and 95 percent also teach a class at the school. Similar trends regarding Murrupula emerge in the school data as well. School directors in Murrupula have more years of experience, teachers have higher attendance rates, and school sizes are smaller (indicating that there may be fewer teachers per student, which enables learning). These observations comparing Murrupula to Nacarôa and Muecate will be helpful in interpreting the results that we describe below. In the next sections, we will present results from the McGovern-Dole Results Framework on Strategic Objectives 1 and 2. Objective one concerns literacy and the inputs that improve literacy, and objective 2 concerns school feeding and the inputs that shape the success of it. 3.2 Improved Literacy of School-Aged Children (SO1) We now turn to reporting differences over time, between treatment arms, and between treatment arms over time. We begin with the EGRA scores, for which we have raw data at midline and endline, with the same districts in the control (Murrupula) and treatment (Nacarôa and Muecate) arms. With these data, we can conduct difference in difference analysis. Table 7 includes EGRA scores on five subtasks: (1) letter name identification and reading, whose score is the number of letters correctly identified and read in one minute (nlpm); (2) familiar words reading, whose score is the number of words correctly read in one minute (nwpm), (3) listening comprehension, whose score is the number of correct answers given by the children, (4) oral reading fluency (ORF), whose score is the number of correct words read per minute (nwpm) and (5) reading comprehension, whose score is the number of correct answers given with regards to a story. In Table 7, Panel (1) reports the number of observations, mean, and standard deviation for Nacarôa and Muecate (treatment group) at baseline. Panel (2) reports number of observations, mean, and standard deviation first for Nacarôa and Muecate and then for Murrupula at midline. The last column in Panel (2) reports the difference between Nacarôa and Muecate compared to Murrupula at midline. Panel (3) follows the same format as Panel (2), except the last column reports the difference between Nacarôa and Muecate compared to Murrupula at endline. The final column of the table (Panel 4) reports the difference in differences estimate. This calculation is only available when both midline and 27 endline data are available, but we report figures from all available rounds. Asterisks denote statistically significant differences. Comparison between treatment and control between midline and endline In all five EGRA subtasks we observe that there are ex-ante differences between the treatment and control districts. In most sub-tasks, students in Murrupula scored higher than those in Nacarôa and Muecate at midline (though not all differences are statistically significant). This indicates that students in Murrupula were better at reading from the start. Additionally, students in Murrupula continued to have higher reading scores at endline compared to those in Nacarôa and Muecate. Consequently, the DID estimate is generally negative (for continuous scores) and positive (for proportion scoring zero). The percentage of students with zero scores has increased significantly in all subtasks, presumably reflecting disruptions in learning following the substantial school closings in 2020. For instance, the DID of letter name identification and reading shows a statistically significant decrease (10 percentage points, p <1%) in treatment when compared to control schools. A similar pattern of the results can be seen in the familiar words reading subtask, where the DID is -5.1 and statistically significant (p<1%). The proportion of students who could not read a single word has increased by 21.2 percentage points and is statistically significant (p<1%). While Table 7 may suggest that the ECT2 program did not improve reading scores for students in the treatment districts, we do not believe this is the right interpretation. As stated earlier, students in Murrupla had advantages in reading, teaching, and school characteristics at midline. Additionally, Nacarôa and Muecate were harder hit by COVID-19 compared to Murrupula, and a substantial USAID-funded literacy program (Vamos Ler!) was launched in Murrupula following its selection as the control district for this evaluation. While acknowledging that COVID-19 disruptions and associated school closures have a bearing on the low scores obtained in this assignment, we note that this outcome is comparable to that obtained in other EGRA tests administrated in Mozambique (cf. Raupp, Newman and Revés, 2013; Raupp et al., 2016; Turney et al., 2018; Stark et al., 2020). Comparison between treatment and control at endline (Panel 3) Overall, the endline results show that students in the control group performed far better than those in the treatment group in all subtasks. Indeed, on average students in the control group identified and read 27.6 letters per minute (lpm), read 6.77 words per minute (wpm), answered 1.02 questions out of 4 in listening comprehension subtask, read fluently 33.2 wpm from a connected text, and answered 0.43 correct questions out of 4 in the reading comprehension subtask. In contrast, students in the treatment group identified and read 14.8 lpm, read 2.4 wpm, answered 0.56 questions out of 4 in listening comprehension subtask, read fluently 7.52 wpm from a connected text, and answered 0.09 correct questions out of 4 in the reading comprehension subtask. The control group also had a lower proportion of zero scores in all subtasks when compared to the treatment group. For all subtasks the differences between treatment and control group are statistically significant. Again, this pattern reflects the fact that the treatment districts were chosen to correspond to those experiencing the most significant challenges linked to student literacy and that the treatment districts were harder hit by COVID-19. 28 Table 7: Summary of EGRA mean scores by subtest (1) Baseline (Grade 2) (2) Midline (Grade 3) (3) Endline (Grade 4) (4) DID Treatment (Nacarôa and Muecate) Control (Murrupula) Treatment (Muecate and Nacarôa) Dif (T-C) Control (Murrupula) Treatment (Muecate and Nacarôa) Dif (T-C) N Mean SE N Mean SE N Mean SE N Mean SE N Mean SE (1) Total correct on Letter Name Identification and Reading (nlpm) 707 4.62 0.32 61 19.6 2.19 401 17.04 1 -2.6 147 27.6 2.72 1523 14.8 0.59 -12.8** -10** Student scored zero on Letter Name Identification and Reading (%) 707 49 1.9 61 22.9 5.42 401 39 2.4 16.1pp* 147 12.2 2.7 1523 29.7 1.1 17.5pp** 1.3pp (2) Total correct on Familiar Words Reading (nwpm) 61 4.2 0.83 401 4.98 0.4 0.72 147 6.77 0.77 1523 2.4 0.12 -4.37** -5.1** Student scored zero on Familiar Words Reading (%) 61 32.7 6 401 39.1 2.4 6.4pp 147 14.2 2.8 1523 41.8 1.2 27.6pp** 21.2pp** (3) Total correct on answers Listening Comprehension (max. 4) 707 0.28 0.025 147 1.02 0.10 1523 0.56 0.02 -0.46** Student scored zero on Listening Comprehension 1 (%) 707 82 1.5 147 44.9 4.1 1523 65 1.2 20.1** (4) Total correct on Oral Reading Fluency (nwpm) 707 1.33 0.16 61 5.6 1.05 401 7.47 0.6 1.9 147 33.2 4.2 1523 7.52 0.62 -25.8** -27.6** 29 (1) Baseline (Grade 2) (2) Midline (Grade 3) (3) Endline (Grade 4) (4) DID Treatment (Nacarôa and Muecate) Control (Murrupula) Treatment (Muecate and Nacarôa) Dif (T-C) Control (Murrupula) Treatment (Muecate and Nacarôa) Dif (T-C) N Mean SE N Mean SE N Mean SE N Mean SE N Mean SE Student scored zero on Oral Reading Fluency (%) 707 62 1.8 61 19.6 5.1 401 25.1 2.1 5.5pp 147 18.4 3.2 1507 45.1 1.2 26.7pp** 21.2pp** (5) Total correct answers on Reading Comprehension (max. 4) 706 0.03 0.008 61 0.15 0.04 401 0.19 0.02 0.04 147 0.43 0.07 1523 0.09 0.009 -0.34** -0.38** Student score zero on Reading Comprehension 2 (%) 706 98 0.5 61 85 4.57 401 83.5 1.9 -1.5pp 147 74.8 3.5 1523 92.8 0.6 18pp** 19.5** Notes: *statistically significant at 5% (between treatment and comparison), **statistically significant at 1% (between treatment and comparison) 30 Table 7 also shows that the proportion of zero scores on all five subtasks ranges from 12.2% to 92.8% across both the treatment and control group. The highest proportion of zero scores was on reading comprehension, the end goal of the reading activity (74.8% in the control and 92.8% in the treatment group). Both groups are struggling to read with comprehension and the results confirm the positive relationship between oral reading fluency and reading comprehension; the speed and accuracy at which a person reads correlates closely with reading comprehension (Daane et al., 2005; Abadzi, 2011). As these and other authors have demonstrated, only when learners can devote more of their attention from mechanical reading (letter recognition, decoding) can they begin to focus more on making meaning out of what they read. Building on this evidence, Figure 2 shows the percentage of children in both the treatment and control districts who score at each subtask at baseline, midline and endline. It is evident that between midline and endline there is a substantial reversion in reading levels in the treatment and control districts, reflecting the effects of school closures without adequate support for continued learning. In the cases where the control district shows some increases, this could be attributed to the introduction of a larger similar interventions (Vamos Ler) in that district. Vamos Ler was a USAID funded project targeted at improving early grade reading outcomes and is serving 2,800 schools in Nampula and Zambézia through teaching young children to read in local languages while simultaneously building Portuguese language acquisition. Figure 2: Correct answers on Each EGRA subtask 31 When we analyze EGRA scores separately by gender, we observe that overall boys performed better than girls in all EGRA subtasks and fewer boys had zero scores compared to girls (Table 8). However, while in treatment schools the difference is statistically significant in most subtasks (at p<1% or p<5%), in control schools the differences were only statistically significant in letter identification and reading subtask (p<5%). These results suggest that while there is gender balance in literacy performance in control schools, there is imbalance in treatment schools, with boys performing relatively better than girls. This pattern is consistent with Murrupula being a better off district; areas with higher socio￾economic status and education tend to have more gender parity in education. Table 8: Summary of EGRA mean scores by subtest by gender Subtasks Gender Control (Murrupula) Treatment (Muecate and Nacarôa) N dif (M-F) dif (M-F) (1) Total correct on Letter Name Identification and Reading (nlpm) male 72 35.04 4.06 15.61* 703 19.71 1.00 female 70 19.43 3.52 793 10.51 0.68 9.2** Student scored zero on Letter Name Identification and Reading (%) male 72 12.50 3.90 -0.3 703 24.80 1.60 female 70 12.80 4.00 793 34.60 1.60 -9.8** (2) Total correct on Familiar Words Reading (nwpm) male 72 7.81 1.14 2 703 3.29 0.22 female 70 5.81 1.10 793 1.49 0.12 1.8* Student scored zero on Familiar Words Reading (%) male 72 13.80 4.10 -1.9 703 37.50 1.80 female 70 15.70 4.30 793 46.40 1.70 -8.9** (3) Total correct on answers Listening Comprehension (max. 4) male 72 1.06 0.15 0.1 703 0.59 0.04 female 70 0.96 0.15 793 0.54 0.03 0.05 Student scored zero on Listening Comprehension 1 (%) male 72 42.00 6.00 -8 703 64.10 1.80 female 70 50.00 6.00 793 66.90 1.60 -2.8 (4) Total correct on Oral Reading Fluency (nwpm) male 72 42.44 6.66 16.55 702 10.90 1.10 female 70 25.89 5.52 793 4.43 0.65 6.47* Student scored zero on Oral Reading Fluency (%) male 72 16.6 0.05 -2 702 44.10 1.80 female 70 18.5 0.05 793 46.60 1.70 -2.5 (5) Total correct answers on Reading Comprehension (max. 4) male 72 0.54 0.12 0.2 703 0.13 0.02 female 70 0.34 0.08 793 0.05 0.01 0.08* Student score zero on Reading Comprehension 2 (%) male 72 72.10 0.05 -3.1 703 89.60 1.10 female 70 75.20 0.05 793 96.20 0.60 -6.6* Notes: *statistically significant at 5%, **statistically significant at 1% The results on letter knowledge, word reading and listening comprehension subtasks are crucial in EGRA analysis as they predict the performance of students on reading comprehension. If students cannot read letters and words and comprehend texts read for them, then they cannot or can hardly read with fluency and comprehend texts they read themselves, as these tasks require, among other things, strong decoding skills. Therefore, the overall EGRA results analyzed indicate that the students both from treatment and control schools are still encountering challenges related to foundational skills (decoding and listening comprehension skills) which do not allow them to read and comprehend basic texts. There is reason to believe this may substantially reflect the immense schooling disruptions experienced in 2020 due to the pandemic. 32 Trend between baseline and endline (Panels 1 and 3) Finally, Table 7 shows that there is a positive trend in reading scores in the treatment group between the baseline and endline. While many reading score components decreased between the midline and endline, students at the endline were still performing better than those at baseline. In all sub-tasks, students scored a higher proportion correct and there were fewer students who scored zero on a particular sub-task. The tables in the remainder of this chapter all follow the same format. The first column lists the result that is targeted by WV. The second column specifies the definition of the indicator used to measure the result. The third column reports the mean value of the indicator at baseline, and the fourth reports the mean of the indicator at midline both only among schools in Nacarôa and Muecate (the treatment group – denoted T)3. The fifth column reports the mean of the indicator in the treatment group and the sixth column in the control group, at endline. The seventh column reports the effect size of the difference between the treatment and control groups at endline. The eighth column reports the effect size and confidence interval comparing the value of the indicator at baseline and at endline (if available). Next, the table lists the target set by WV, the level of achievement (the endline value of the treatment indicator divided by the target), and the status. The status is recorded as “achieved” if the level of achievement is 100% or greater and is recorded as “in progress” if it is less than 100%. Note that effect sizes are only reported if differences are statistically significant. 3 Midline data from the control group is not available. 33 Table 9: Percentage of students who by the end of two grades of primary schooling demonstrate that they can read and understand the meaning of grade level text – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline - baseline (T) Target Level of Achievement Status T T T C ES 95% CI Dif ES Improved Literacy of School￾Aged Children Percentage of students who by the end of two grades of primary schooling demonstrate that they can read and understand the meaning of grade level text Total 25.1 (337) 32 13.2 (1523) 41.8 (147) 0.66** [-12 ± 4.9] (0.66) ▼ 45 29% In progress M N/A 34 18.5 (703) 49.2 (72) 0.66** 45 41% In progress F N/A 31 8.1 (793) 34.3 (70) 0.67** 45 18% In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ▼ statistically significant decrease; ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 34 Classifications by reading ability In the baseline and midline evaluations, a classification criterion was used to rank the students under four main categories: non-reader (cannot read at least 6 wpm), emergent reader (reads between 6 and 44 wpm), established reader (reads between 45 and 80 wpm) and proficient reader (reads more than 80 wpm). For comparison purposes, the same classification criterion was used at endline. Based on this criterion, the results confirm that students in control schools outperformed their peers in treatment schools. As shown in Table 9, 13.2% of the students in treatment group read between 6 and 44 wpm, against 41.8% in the control group. These are considered emergent readers, those who know how to read, despite presenting some difficulties. The distribution of students based on this classification is shown in Figure 2. Figure 3: Students’ reading ability in treatment and control schools at baseline, midline and endline The disaggregation of the results by gender in the endline confirms that overall boys performed better than girls in literacy tasks both in treatment and control schools. However, while the difference between boys and girls is statistically significant in treatment schools (p<1%) no significant difference was detected in control schools. More specifically 18.5% of boys and 8.1% of girls in treatment group read between 6 and 44 wpm, while in control schools the proportion was 49.2% for boys and 34.3% for girls. We did not have access to baseline data disaggregated by gender to compare with these endline results. Finally, Table 9 shows that COVID-19 disruptions rendered it challenging to reach the target for the project. 41.8 13.2 32 25.1 58.2 86.8 68 74.9 0 10 20 30 40 50 60 70 80 90 100 Controle Treatment Treatment Treatment Endline Midline Baseline Emergent Readers Not Emergent Readers 35 3.3 Literacy Instruction (1.1) The midline study found that all grade 1-3 teachers in treatment schools received training in LB by midline. 773 members of educational staff in the target districts received training as a result of USDA assistance. All teachers also demonstrated improved literacy instruction techniques as reported by supervisors and mentors. By endline, 68.8 percent of the teachers in treatment schools demonstrate improved literacy instruction, against 100 percent in control schools (see Table 10). The high rates of improved literacy instruction in control schools presumably reflects in part the implementation of other literacy-related programming in those schools. The result obtained in treatment schools in endline corresponds to difference of 31.2 percentage points. Note that, during the endline survey, we asked about the practices of grade 4 teachers only since we were not able to survey all teachers in the school. Consequently, the measure is different, but can be considered a proxy of the overall teaching quality in a school assuming that there are no substantial differences between the quality of teaching across grades 1-3 and grade 4. Additionally, the definition of “improved literacy instruction” at endline was whether the teacher grouped students either according to ability or to work together with the textbook or other learning materials rather than using a lecture or repetition method. This indicator is more specific compared to asking supervisors about improved literacy instruction in general and is thought to be one of the most effective modes of teaching. Note that following the school closures, there are likely many teachers who were trained in LB but who either did not return to school or began teaching in other areas. Additionally, some of the improved practices may have been difficult to implement due to COVID-19 restrictions. Finally, a substantial USAID￾funded literacy program (Vamos Ler!) was launched in Murrupula following its selection as the control district for this evaluation, and this may have led to a separate set of pedagogical innovations or improvements in Murrupula driven by this enhanced programmatic access/funding. Learning from the Vamos Ler project; World Vision is currently in collaboration with Creative Associates International which is also implementing another USAID funded literacy program in the province to enhance the USDA funded literacy interventions. The collaboration involves the sourcing of their bilingual literacy materials for reproduction and distribution to ECT3 target schools. The ECT3 project has already purchased 10,000 of same books to distribute within the schools in Muecate and Nacaroa districts. 36 Table 10: Percentage of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors or coaches – Baseline, Midline and Endline vs. Target Result Indicator Baselin e Midline Endline Baseline - endline (T) Target Level of Achievement Status T T T C ES ES T Improved Quality of Literacy Instruction Percentage of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors or coaches 74.8 100 100 Teaching practices Percentage of teachers who split students into groups (by ability or to work together with a textbook or other learning materials) 68.8 (160) 100 (15) 0.98** Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 37 3.4 Teacher Attendance (1.1.1) Teacher absenteeism is one of the main challenges in education management in Mozambique and is one of the key factors hindering students’ learning. The ECT2 project included methodologies and activities aiming at enhancing teacher attendance, including teacher performance awards and school council monitoring of teacher attendance. Stakeholders indicated that both methodologies were effective in reducing teacher absenteeism. Indeed, ECT2 monitoring data indicate that at midline, 96% of the teachers in treatment schools attended and taught school at least 80% of scheduled days per school year. However, by the time of the endline survey, the percentage of teachers in target schools who attended and taught school at least 80% of scheduled days per year dropped to 75.6%, against 93.3% in control schools (see Table 11). Note again that this indicator is only reported for grade 4 teachers; it is calculated as the number of days in the past 5 school days that the deputy school director reported that the grade 4 teacher was present in school. We did ask the deputy directors the proportion of all teachers who are present 80 percent of school days, but unfortunately deputy directors did not report this statistic consistently across schools, and thus it cannot be used. The decrease in teacher attendance can be attributed to the disruptions in teaching and learning processes associated with school closures in 2020 and irregular functioning in 2021 due to the COVID-19 pandemic. Indeed, even when schools were open, teachers and students were recommended and allowed to stay home if they felt any potential symptoms of COVID-19 (cough, headache, fever, etc.) Moreover, students were advised to attend school only two to three days per week in order to reduce class sizes and ensure compliance with COVID-19 protocols, further reducing reported attendance. Given the government protocols to minimize exposure risk, the movements of school council members to schools have also reduced to the minimum necessary, and this does not allow them to perform common roles including verification of teachers’ attendance. We must also re-emphasize that our data suggests that Nacarôa and Muecate districts were harder hit by COVID-19 compared to Murrupula. However, over the life of the ECT2 project, there have been many positive gains. All 160 schools did monitor teacher attendance, surpassing the target of 150 schools. Additionally, 1,058 teachers in schools attended school for more than 80 percent of school days. 385 performance awards were distributed to teachers (relative to the target of 600), and 755 performance awards were distributed to students (relative to a target of 600). 38 Table 11: Percentage of teachers in target schools who attend and teach school at least 80% of scheduled days per school year – Baseline, Midline and Endline vs. Target Result Indicator Baselin e Midline Endline Baseline - endline (T) Target Level of Achievement Status T T T C ES ES More Consistent Teacher Attendance Percentage of teachers in target schools who attend and teach school at least 80% of scheduled days per school year 90% 96% 90% Percentage of Grade 4 teachers who attend and teach school at least 80% of scheduled days per school year 75.6 (160) 93.3 (15) 0.51* Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 39 3.5 Access to School Supplies and Materials (1.1.2) ECT2 has contributed to improving access to school supplies and materials in treatment schools. During the project implementation, 18,565 books were procured or produced and stocked in book banks; a total of 141 libraries were established and 16,735 students per year were reported to have checked out library books. 90,906 textbooks and teaching materials were provided as a result of USDA assistance, and all 160 target schools received such materials. For contexts such as Nacarôa and Muecate, characterized by scarcity of reading materials, the establishment of libraries and book banks and provision of books is essential to boost literacy habits among students and community members, in conjunction with training teachers in the creation of reading material as a sustainable method of enhancing materials access (one page story books and stories). The collaboration between project implementers and government education authorities enabled most students to have access to basic textbooks. At endline, 96.3% of the teachers surveyed in treatment schools reported that their students had Portuguese textbooks, against 80% of teachers in control schools. However, as shown in Table 12, in the endline survey only 46.8% of the teachers surveyed in treatment schools reported that had received textbooks and other teaching and learning materials provided either by World Vision or Save the Children. Due to the COVID-19 disruptions in supply chains, we interpret this result as reflecting challenges in ensuring that books and other educational materials remained available in schools. It is possible that some books were lost or damaged during the school closures and/or that teachers moved between schools or across grades within the same school and thus do not remember receiving materials. Unexpectedly, 7.8% of the teachers surveyed in control schools also reported that had received textbooks and other teaching and learning materials provided because of USDA assistance; however, this is likely attributable to recall error or movement of teachers across schools. Note that no deputy school directors said that their school had received LB programming. We also report whether a teacher reported that they received textbooks or other learning materials from any source. Here, 69.4% of teachers in the treatment group and 86.7% of teachers in the control group report having received teaching and learning materials from any source. While it is possible that textbooks and other learning materials may not have been widely distributed by the beginning of the 2021 school year, Table 12 suggests that literacy instruction materials for students were in fact available to schools in the treatment group. Indeed, according to World Vision monitoring data, by the midline, 90 percent of classrooms in treatment schools had instruction materials (textbooks, workbooks) sufficient for effective instruction. The availability of literacy materials for students was also evident in the endline survey, as 82.2% of grade 4 teachers surveyed reported that their classes had literacy instruction materials (textbooks, workbooks) sufficient for effective instruction, against 95% in control schools. There is no statistically significant difference between control and treatment. This can be regarded as a positive indicator, considering the disruptions in the supply of instruction materials due to COVID-19 restrictions measures, which affected the flow of goods and people. However, enumerators noted that not all students present on the day of visit had their own literacy materials, and some shared materials with other students. Once again, over the life of the ECT2 project, 90,906 textbooks have been distributed and all 160 schools received teaching and learning materials. While COVID-19 may have slowed down progression somewhat, over the life of the project, this is a substantial infusion of materials. 40 Table 12: Percentage of Teachers who received textbooks and other teaching and learning materials provided as a result of USDA assistance – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline - baseline (T) Target Level of Achievement Status T T T C ES 95% CI Trend and (ES) Better Access to School Supplies and Materials Percentage of teachers who received textbooks and other teaching and learning materials provided as a result of USDA assistance N/A 100 46.8* (111) 7.8* (13) 0.92 100 46.8% In progress Percentage of teachers who received teaching and learning material from any source 69.4 (160) 86.7 (15) Percentage of classrooms with literacy instruction materials (textbooks, work books) sufficient for effective instruction 80 (337) 90 (405) 82.2 (151) 95 (10) [2+-7.3] ► 100 In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ► no statistically significant difference. ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 41 Despite the relatively positive environment described above, shortages of textbooks and other materials (e.g. maps, posters, cardboards) was one of the main challenges reported by teachers, school managers, students, and parents. The general observation was that the textbooks supplied by the Ministry of Education are not sufficient for all students, which constrains the teaching and learning process. As a teacher commented in a focus group discussion: “We have difficulties with books. Some students have books and others don't, this ends up creating discontent in students from the moment they see that a colleague has a book, and he doesn't. The books we have are not enough for everyone, if it were clear that all students would want to go to school every day. Portuguese books are not enough for everyone either” (Teacher 1, Nacarôa). Strategies used that were mentioned by teachers in focus group discussions to overcome the scarcity of textbooks include writing texts and exercises on the board and requesting that students copy them into their exercise books and creating different activities for students who do not have textbooks and for students who have them. Despite these creative initiatives by teachers, it should be noted that they cut into actual teaching time and makes it difficult for them to engage in more active and demanding teaching and learning practices. These constraints likely affect students’ performance in literacy and other curricular components. 3.6 Improved Attentiveness (1.2) Improving the attentiveness of students was one of the specific objectives of the ECT2 project. The assumption is that by improving students’ attentiveness the project could contribute to enhancing their opportunities to learn. This was planned to be achieved by providing school meals, ensuring that students were better fed and thus more attentive in classes, and by training teachers in the use of group dynamics and classroom arrangement practices that allowed the engagement of students, especially those less active. Project reports and accounts from different stakeholders interviewed indicate that these strategies were effective in improving the students’ attentiveness during the project implementation. At baseline and midline, teachers were asked if their students were more attentive due to the school meals. At endline, rather than asking teachers to estimate the proportion of students they regard as attentive during their classes, we instead asked the students themselves. As shown in Table 13, the result was that 73.6% students in treatment schools reported that they felt more attentive because of the food they get at school, compared to 59.1% in control school. This difference is statistically significant (p<1%). 42 Table 13: Percentage of students in target schools attentive during class/instruction – Baseline, Midline and Endline vs. Target Result Indicator Baselin e Midline Endline Endeline￾baseline (T) Target Level of Achievement Status T T T C ES 95% CI Trend and (ES) Improved Attentivenes s of Students Percentage of students in target schools identified by their teachers as attentive during class/instruction 98 90 98 Percentage of students who feel more attentive because of the food they get at school 73.6 (1,523) 59.1 (147) 0.30** Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 43 In focus group discussions with students, those in treatment schools repeatedly said that they like to eat at school because they do not have access to three meals per day at home and that when they have meals at school, they feel more attentive in classes and more easily retain the content. The account which follows, taken from a focus group discussion with students in Muecate district, substantiates the impact of school meals on the students’ motivation and predisposition to learn: “The school lunch helps me a lot. I study and then I go to eat and get satisfied. When I return home, I don’t take any meals. I learn fast when I eat, because between someone who is hungry and someone who is healthy, the one who is healthy learns more quickly” (Student 2, Muecate). As this student reports, school meals not only boost the students’ predispositions to learn but also assists their households in conserving food, which can benefit those members that do not have access to (free) food outside their homes, such as younger siblings. There are different factors that may influence students’ attentiveness, but hunger is one of the common reasons associated with poor attentiveness in settings in need such as Muecate and Nacarôa, in particular the more rural and less productive areas. Therefore, when 73.6% of the students surveyed attribute their attentiveness in classes to reception of school meals, that should be regarded a positive effect of the USDA school feeding program. 3.7 Reduced Short-Term Hunger (1.2.1) Data from midline suggest positive effects of the intervention by increasing the proportion of students indicating that they are not hungry during school day, although there is a substantial decrease between midline and endline. Indeed, Table 14 shows that at endline 58.8% of the students in treatment schools and 47.3% in control schools indicated that they were not hungry during school day. The difference between the treatment and control groups in endline is statistically significant (p<1%) and the effect size 0.27. We can reasonably assume that the lower proportion of students who are hungry in control schools is due to the fact that school meals are not provided in Murrupula. Consequently, even though school meals may not have been provided as consistently as before and even though many students report that they do feel hungry, the meals have reduced short term hunger successfully among the treatment group suggesting that the school meals are successful in reducing short term hunger. 44 Table 14: Percentage of students in target schools who indicate that they are not hungry during school day – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline￾baseline Target Level of Achievement Status T T T C ES 95%CI Trend (ES) Reduced Short￾Term Hunger Percentage of students in target schools who indicate that they are not hungry during school day 52.6 100 58.8 (1523) 47.3 (147) 0.27** [6.2+-0.3] ▲ (0.12) 100 58.8% In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ▲statistically significant increase; ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 45 A positive trend can also be observed between the baseline and endline in treatment group. There was a significant increase of 6.6 percentage points from baseline to endline (from 52.6% to 58.8 % in endline, p<5%). However, the effect size is small (0.12). The increase in self-reported hunger since the midline survey most likely reflects the impact of the COVID-19 pandemic on families’ well-being. As noted above, 63% of the students interviewed reported household hunger due to COVID-19, and 45% had to sell goods to buy food, against 51% and 29% in control schools, respectively. In addition, as noted above, at the point of the endline survey schools were not yet operating normally. This scenario may explain why the school feeding component was not fully operational. 3.8 Increased Access to Food (School Feeding) (1.2 & 1.3) All students interviewed in treatment schools reported that their schools serve school meals as a result of USDA assistance. Endline data show that a total of 81,589 children (43,634 boys and 37,955 girls) received daily school meals (breakfast, snack, or lunch), which represents 124% of the planned target (see Table 15). Comparing the endline with the baseline data it is found that the total number of school-aged children receiving daily school meals increased by 42% (from 57,501 to 81,589 beneficiary students) reflecting a level of achievement of 124%. According to the census, Mozambique is growing at a rate of 2.93% every year. The fertility rate is high at 4.89 births per woman, caused by early first pregnancies and low use of contraceptive methods. In Nampula the population growth rate is 3.3%, and the fertility rate is one of the highest in the country4. This trend is consistent with large increases in school enrolment. This explains why the level of achievement for this indicator is higher than 100%; a lot more students likely enrolled between midline and endline and World Vision was able to provide meals for more students than planned. Over the life of the ECT2 project, approximately 7.5 million meals have been distributed in schools, serving over 42,000 children, through almost 4,000 tons of commodities for school meals. In addition, the survey collected data on take-home rations offered by schools during COVID-19 related closures. In treatment schools, between 3% and 10% of students and teachers reported that take-home rations had been offered during the period of school closures. During the outbreak of COVID-19, schools were closed and school-aged children were expected to be studying remotely from their homes. With USDA approval, the project distributed take-home rations during the school closures in collaboration with volunteer teachers/cooks, District Administration and Ministry of Education Officials to ensure that school-aged children received their food rations whilst at home. The engagement of volunteer teachers/cooks and relevant government officers made the transition from school meals to THRs very smooth. Parents were assigned to specific food distribution points (FDPs) to receive the rations for their school children according to a well-publicized distribution plan designed by the commodity team and 4 http://www.ine.gov.mz/estatisticas/publicacoes/mulheres-e-homens/mulheres-e-homens-em-mocambique-2017 46 education officials. COVID prevention protocols were fully observed during the distributions. The main objectives for the take home rations were: (i) To make sure that students continue to receive the required nutrition while learning from home. (ii) To reduce the economic burden on parents that was caused by the pandemic by including PLW/CU5 as beneficiaries of the take home ration. This meant that school going children, their under 5 siblings and their pregnant or lactating mothers also received a ration of up to 9 kg. Data from Post Distribution Monitoring indicated that the Take Home Rations helped in making sure that children continued to learn while at home. The take home ration for PLW/CU5 also encouraged a number of women to register themselves and their under 5 children at health centers in order to continue benefiting from the project as well as post-natal and ante-natal services provided at the health facilities. Further, take-home rations have been widely distributed over the life of the project. 2,295 individuals received take home rations, with over 900,000 take home rations having been provided, through 136 tons of commodities provided for the take home rations. 47 Table 15: Number of school aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Target Level of Achievement Status T T T T Increased Access to Food (School Feeding) Number of school aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance Total 57,501 65,806 82,195 65,806 124% Achieved Male 32,775 35,084 43,700 35,084 124% Achieved Female 24,726 30,722 38,495 30,722 119% Achieved Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 48 The school feeding component also addresses one of the basic needs of students in Nacarôa and Muecate – access to food. The program enables students to remain enrolled in schools. The following account from a student in Muecate is revealing: “At school we eat and we like the food. We don't take school lunch’s home. I like the school lunch initiative, because even the children who had dropped out of school have come back and have had a snack. The initiative is commendable and I encourage you to bring other snacks” (Student 1, Muecate). This account illustrates the general perception of students and other stakeholders in relation to the impact of school feeding on the students’ well-being and predisposition to stay in school. This substantiates the general observation by different stakeholders that the school feeding component of the USDA program contributes to an environment conducive to learning in settings in need such as Nacarôa and Muecate. 3.9 Improved Student Attendance (1.3) Improving student attendance was one of the key goals of the ECT2 project. This was set to be achieved through initiatives such as provision of school meals, introduction of teaching techniques conducive to students’ motivation to attend school, improvement of teacher attendance, involvement of school councils in monitoring the attendance of teachers and students, and sensitization of parents and caregivers about the importance of their children’s education. Data gathered from interviews with various stakeholders indicates that these initiatives contributed to improve students’ attendance and retention. However, as shown in Table 16, when the endline was conducted only 43.8% of the students reported that had attended school at least 80% of the school days, which is a decrease from baseline and midline, but is much higher compared to 18.1% in control schools. This difference is statistically significant (p< 1%), and the effect size is considerable (0.55). This substantial shift primarily reflects schools’ new policy of alternating attendance days implemented in response to the COVID-19 pandemic, mandating that students attend class on an alternating schedule characterized by only 2-3 days of school per week per student in order to minimize class sizes. As previously noted in relation to teachers, the reduced proportion of students attending school at least 80% of the school days can be attributed to COVID-related disruptions and associated impacts. Health￾related fears on the part of students and parents prevented students from attending school even when the government authorities considered it safe to do so. Many schools set up schedules requiring students to alternate days or weeks of attendance, both reducing attendance and generating confusion about appropriate attendance policies that renders it challenging for students and parents to manage. Given these challenges and particularly the alternating attendance school schedules implemented post-pandemic, the proportion of students who did attend school 80% of time is actually quite high. 49 Table 16: Percentage of students that attend school at least 80% of the school days – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Beseline /endline (T) Target Level of Achievement Status T T T C ES 95% CI Trend and (ES) Improved Student Attendanc e Percentage of students that attend school at least 80% of the school days 90 80 43.8 (1,523) 18.1 (147) 0.55** [46.+-4.7] 1.06 ▼ 95 46% In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ▼statistically significant decrease; ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 50 Students’ school attendance has been of great concern in Mozambique, even before the COVID-19 pandemic. In fact, as also discussed in the midline report and in other studies conducted in Mozambique, although hunger is one of the factors that limits students school attendance and retention, there are also other socio-cultural factors to take into consideration (e.g. Bagnol et al., 2015; Lauchande, 2015). These include the importance communities give to education (including the differential value accorded to education of boys and girls), the distance between home and school, and household resources. The province of Nampula is one where these factors are particularly relevant, especially early marriage, initiation acts, and deployment of children in farming and informal market activities. 3.10 Reduced Health Related Absences (1.3.2) To reduce health related absences, the ECT2 project included health education activities targeting students and communities aiming at improving health and dietary practices. The school feeding component of the project was one of the ways to improve the students’ nutrition levels. Interviews with different stakeholders suggest that within the scope of the project intervention, the proportion of students not attending school because of health-related issues had substantially reduced. However, Table 17 shows that at endline, 29.4% of the students surveyed in treatment schools had missed more than 10 school days per year due to illness, compared to 20.4% in control schools (p<5%). This pattern likely reflects the differences in socio-economic status and the severity of COVID-19 in Murrupula compared to Nacarôa and Muecate. In both groups, girls (33% in treatment and 24.3% in control schools) tend to miss more school due to health-related issues than boys (24.6% in treatment and 13.9% in control schools (p<1%). 51 Table 17: Percent of students in target schools who miss more than 10 school days/year due to illness – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline - baseline Target Level of Achievement Status T T T C ES 95% CI Trend and (ES) Reduce d Health Related Absenc es Percent of students in target schools who miss more than 10 school days/year due to illness Total 2.2 2.0 29.4 (1,523) 20.4 (147) 0.11* [-27+-2.7] 0.85 ▲ 1.5 19.6% In progress Male 2.1 1.9 24.6 (703) 13.9 (72) 0.25* 1.5 16.4% In progress Female 2.3 2.0 33 (793) 24.3 (70) 1.5 22.0% In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ▲statistically significant increase; ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 52 3.11 Increased Knowledge and Use of Health, Hygiene, and Dietary Practices (2.1) We next turn to Strategic Objective 2 in the McGovern-Dole Results Framework, which concerns school feeding. ECT2 trained 1,537key stakeholders in health and hygiene knowledge and practices. Overall, the initiatives carried out in the project have contributed to improve health and hygiene knowledge and practices in treatment schools and surrounding communities. However, there are still challenges to be addressed. Despite the positive changes introduced in treatment schools and in surrounding communities as a result of the ECT2 project by midline, the endline study found that only 31.6% of the students surveyed could identify at least 3 important health and hygiene practices, against 42.9% in control schools (see Table 18). The difference accounts for an effect size of 0.21. This is an unexpected result, particularly considering that by the midline 86% of the students surveyed in treatment schools could name three or more good health and hygiene practices. When endline and baseline results are compared, we find a considerable decrease of 30.4 percentage points (p<5%), which represents a high effect size (0.63). One of the consequences of the COVID-19 and associated school closures has been loss of part of the knowledge that children acquired before these events. When schools reopened, the focus was on academic basics to try to catch children up in terms of curriculum content, which did not allow much time to impart other lessons around health and hygiene. These facts may help to explain the decrease in the proportion of students that could identify at least 3 good health and hygiene practices by the time of the survey. Table 19 examines handwashing knowledge and practices, starting with the appropriate use of handwashing practices. At midline, 94% of male and 93% of female students used appropriate handwashing practices. Turning to knowledge about good handwashing practices at endline, the results across gender are roughly balanced, although in treatment schools the percentage of girls (90.3%) can name at least one appropriate hand washing practice is slightly higher than that of boys (88.8%), while the opposite case is true for the control schools (94.4 for boys and 92.8 for girls). There are no statistically significant differences between boys and girls either within or across the treatment and control groups. 53 Table 18: Percentage of children in target communities who can identify at least 3 important health/hygiene practices – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline - baseline Target Level of Achievement Status T C ES 95% CI Trend and (ES) Increased Knowledge of Health and Hygiene Practices Percentage of children in target communities who can identify at least 3 important health/hygiene practices 62 86 31.6 (1,523 ) 42.9 (147) 0.21* [31.4+-5.7] (0.63) ▼ 90 35% In progress Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ▼statistically significant decrease; ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 54 Table 19: Percent of target beneficiaries who use appropriate hand washing practices – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Endline - baselie Target Level of Achievement Status T T T C ES 95% CI Trend and (ES) Increased Use of Health and Dietary Practices Percent of target beneficiaries who use appropriate hand washing practices Total NA 94 95 Achieved Male NA 94 95 Achieved Female NA 93 95 Achieved Percent of target beneficiaries who know about recommended hand washing practices Total 89.3 (1523) 93 (147) Male 88.7 (703) 94.4 (72) Female 90.3 (793) 92.8 (70) Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 55 3.12 Increased Knowledge of Safe Food Preparation and Storage Practices (2.2) School meals are prepared by volunteer cooks, mainly parents, but also other community members. These usually work in shifts under the coordination of school councils and school feeding supervisors, as institutionalized in each school. The majority of the volunteers interviewed (88%) among the treatment schools reported that they received training in food preparation. As shown in Table 20, as a result of both training in advance and on the job training, as well as from practical experience, virtually all food preparers surveyed (99.4%) were able to identify 3 or more safe food preparation practices even if they did not report receiving any formal training in these practices. When the volunteer cooks were asked to name some safe food preparation and storage practices, the most frequent responses were: washing the pot before cooking (97%), washing the pot after cooking (88%), washing hands before cooking (96%), washing hands before serving (79%), storing the food in a clean place before cooking (82%), using a clean spoon for stirring (88%), using a clean spoon for serving (81%) and covering food with a lid (86%). Note that we only report data from treatment schools as control schools did not have school cooks. These results are consistent with the cumulative numbers over the life of the ECT2 project. Cooks in all 160 schools have been trained on good food preparation practices, with a total of 818 school cooks trained. Additionally, 95% of the school directors interviewed reported that their schools have adequate facilities (own room, clean, ventilation) to store food (2.6). The report on the midline evaluation indicated that volunteers claimed that they should receive some symbolic compensation for their work, including for satisfying their basic needs. According to this report, volunteers felt that eating the school meals, taking corn soy blend (CSB) rations home on working days, and having a health card that allowed them to have medical examinations and free treatment in the health unit was not enough. However, the endline study found that this perception changed as parents and other community members started to understand that preparing food for the students was a valuable contribution to their children’s health and education. This is a positive result of the project initiatives aiming at improving school-community ties through increased involvement of parents and other community members in the children’s education. 56 Table 20: Percentage of food preparers at target schools who can identify at least 3 key practices aimed at safe food preparation – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Edline Target Level of Achievement Status T T T Increased Knowledge of Safe Food Preparation and Storage Practices Percentage of food preparers at target schools who can identify at least 3 key practices aimed at safe food preparation N/A 100 99.4 100 100 Achieved Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 57 3.13 Increased Access to Clean Water and Sanitation Services (2.4) The endline study found that 129 of the 160 schools in the World Vision project have access to improved water sources and therefore access to clean water (see Table 21). This number represents 80.6% of the treatment schools, which is comparable with the proportion captured in control schools surveyed (83%). The access to improved sanitation services in treatment schools is far better than the access to clean water. Indeed, all schools covered by the project have improved latrines. In all schools, the latrines are separate for students and teachers and for males and females. Comparing endline to baseline results, the number of schools with access to clean water decreased from 150 to 129 schools (6.6%), which correspond to 80.6% of the planned target result. This decrease may be attributable to COVID-19 related water disruptions or degradation of water sources during the school closures. (School closures presumably led to suspension or reduction of maintenance, which may have had meaningful implications for the functioning of this form of infrastructure.) In contrast, access to improved sanitary facilities increased from 40 to 159 schools (297%), representing 109.6% of the target achievement. This increase likely reflects latrines that were constructed between midline and endline, but in the 2019 school year, before COVID-19 school closures. However, over the life of the ECT2 project, all schools were provided with clean water and sanitation services (for example, the installation of tippy￾taps outside canteens and boreholes). Despite improvements, access to clean water and improved sanitary facilities continue to be two challenges faced by schools in Nacarôa and Muecate, as expressed in focus groups discussions: “Some schools already have buildings. We have water (but the water is a little far from the school), you could think of small water holes so that the school has water in the toilets” (Director 1, Nacarôa). The findings suggest that more investment could be made to increase and improve access to clean water and sanitation services in schools in Muecate and Nacarôa and beyond, especially in light of the ongoing pandemic. 58 Table 21: Number of schools using an improved water source and number of schools with improved sanitary facilities – Baseline, Midline and Endline vs. Target Result Indicator Baseline Midline Endline Target Level of Achievement Status T T T C ES Increased Access to Clean Water and Sanitation Services Number of schools using an improved water source 150 160 129 (159) 10 (12) 160 80.6% In progress Number of schools with improved sanitary facilities 40 137 159 (159) 11 (12) 145 109.6% Achieved Notes: Number of observations reported in parentheses. T = treatment group (Nacarôa and Muecate districts) and C = control group (Murrupula district). ES: Effect size computed using Cohen’s d; CI: Confidence Interval; * statistically significant at 5% ** statistically significant at 1%. Target denotes the target value of the indicator set by World Vision. Level of achievement is calculated by dividing the treatment group endline mean by the target. 59 4 COVID-19 Experiences In this chapter we explore the effect of the COVID-19 pandemic on students, educators, and schools. The COVID-19 pandemic has devastated societies and the education of children all over the world. Mozambique has been no exception to this devastation and the children of Nampula province have unfortunately suffered from similar school closures to those seen elsewhere. As of this writing, Mozambique has seen over 148,000 positive COVID-19 cases and 1,881 COVID-19 related deaths, and there is currently mandatory 30-day closure of preschools. While primary and secondary schools are now open and operating, they were closed for approximately a year between the declaration of national emergency on March 31, 2020, and February 2021. The purpose of this section is to describe the impact of COVID-19 and school closures on the education of children and the operation of schools in Nampula province. Using data from the endline survey of the ETC2 project, we begin by summarizing the coping strategies households engaged in during lockdowns, and the hardships that created. We then move to describe the educational mitigation strategies students and teachers undertook both before and during lockdowns. Finally, we explore the reopening plans of schools and the current shortcomings that exist in terms of COVID-19 protection. This chapter does not report on differences between the treatment and control groups since these indicators were not part of the ECT2 programming. Rather, all figures are disaggregated at the district level. 4.1 Lockdown Hardship and Coping Strategies Almost ubiquitously across developed and developing nations, restrictive COVID-19 policies have created stories of food insecurity and financial hardship among families. The issue is likely two-fold in developing nations where the social safety nets that many have relied on in rich nations are often overburdened, mistargeted, or non-existent. Without these protective factors, families may have to turn to other coping mechanisms to meet their needs. These coping mechanisms may involve the sale of assets, reducing future earnings potential; dowry fees, often incurring the negative externalities associated with child marriage; or child labor, at the cost of the child’s education. 60 Figure 4: COVID-19 Hardships and Coping Strategies Notes: The unit of analysis is the pupil. Data is retrieved from Grade 4 pupils who were selected to partake in the EGRA portion of the survey. Bars represent the proportion of children who responded with the respective option to the question “What were the impacts of the coronavirus in your household?” (translated to Portuguese). The five most prevalent answers are displayed. In Figure 4 we explore the hardships families faced due to COVID-19 and the coping mechanisms that families in our sample undertook to help them navigate the COVID-19 related lockdowns throughout Nampula Province. The figure paints a grim story of the effects of COVID-19 on children’s nutrition, with over 60 percent of children reporting that they missed meals due to COVID-19 in the districts of Nacarôa and Muecate. In comparison, the lockdowns seem to have been relatively less detrimental, although still severe, in Murrupula, which may explain some of the disparities we observe in EGRA performance across districts. In terms of household coping strategies, the sale of assets seems to be the first line of defense for many families by a large margin over child labor. One possible reason is that the source of the hardship was an economic downturn due to COVID-19, thus reducing the marginal product of child labor in addition to that of adult labor. Additionally, lockdowns may have prevented both children and adults from working. 61 Figure 5: Main Reasons Children did not Return to School Notes: The unit of analysis is the teacher. Teachers were asked the question “What are the top three reasons pupils did not return to school?”. Bars represent the proportion of teachers who listed a given option as their top reason for the top four options. To explore this possibility, Figure 5 presents the main reasons children did not return to school post￾lockdown, as reported by the grade 4 teacher. Within Muecate and Nacarôa, child labor and child marriage are the leading reasons that teachers believe students did not return to school. This also matches Figure 2, which reports high levels of child marriage across the districts. Curiously, fear of COVID-19 seems to be a much larger reason in Murrupula than in the other two districts. A large number of students, particularly in the treatment villages, also report “Other” as a reason; when asked to specify, these households overwhelmingly specified challenges related to migration. We do not have the data to explore the dynamics underlying these fears, as there is no publicly available data on variation in COVID-19 cases at the district level. However, one may suppose that the direct proximity of Nacarôa and Muecate to the conflict in Cabo Delgado and the more salient danger of violent conflict may have taken attention away from COVID-19. It is also possible that given that households in Murrupula report higher usage of radios and television during the pandemic, they were more exposed to information that led them to be more fearful. In addition, ECT2 focused on provision of COVID-19 education at the community level during school closures, and this programming may have reduced fear of COVID-19 in these districts, which most likely contributed to less fear in Nacarôa and Muecate. Overall, these figures tell a grim story that COVID-19 created hardships for families that were passed down to children. It is likely that the hunger created by economic restrictions will have long lasting consequences for children’s education. It should also be noted that this is the story for those that can return to school. For many other students, it is likely that their education was permanently halted in March 2020. 62 4.2 Education in the Age of COVID-19 The closure of schools has become a contentious topic in many political discussions. However, the reality of the situation is that in developed nations, education was able to proceed with the aid of technology, albeit at a lower quality. The same cannot be said of developing nations, and particularly rural areas, where technology is not as readily available, and electricity can be unreliable. Here, the closure of schools may mean the full delay (or termination) of education. In this section, we explore the techniques that teachers undertook to prepare for and mitigate the impact of their children’s education before the school lockdowns occurred. Figure 6: Pre- COVID-19 Educational Measures Notes: The unit of analysis is the teacher. Bars represent actions taken by teachers to ensure their students continue learning during lockdown. “Radio” represents teachers informing students about educational radio shows, while “Television” represents the same but for television. The top four responses are represented. We begin by exploring actions that teachers took prior to the lockdown to ensure that learning would continue. Figure 6 reports the measures that teachers took prior to lockdowns to ensure learning continued. We observe that take home materials were by far the most widely used strategy, whereas educational television and radio programming were not used much. We also observe that a small number of teachers called their pupils’ parents to discuss the impeding lockdowns and to encourage them to continue their child’s education. 63 Figure 7: Educational Measures during Lockdowns Notes: The unit of analysis is the teacher. Bars represent actions taken by teachers to ensure their students continue learning during lockdown. “Radio” represents teachers informing students about educational radio shows, while “Television” represents the same but for television. The top six responses are represented. Once the lockdowns began, teachers were forced to continue children’s education without having them in a classroom. We describe the actions they took in Figure 7. We observe a similar pattern to that observed during pre-lockdown. We once again see that take home materials were the most popular measure. However, unlike prior to the lockdowns, we see far greater communication between parents and teachers and a small number of teachers who directly taught students, either by making home visits or using small group sessions. In contrast to the pre-lockdown measures, educational radio programs seem to have become particularly popular in Murrupula during the lockdowns. In Murrupula district, more households’ own radios, and Murrupula has a community ratio station (not present in the other districts). It is possible that these radio programs, and the greater teacher-parent communication in Murrupula throughout the lockdowns bolstered education enough to create the disparities in EGRA scores that we observe during the ECT2 endline. 4.3 School Reopening In March 2021, schools were allowed to reopen to students. However, still amid a pandemic, with new variants emerging frequently, schools were opening to tense communities worried about the safety of their children. Additionally, the government specified guidelines under which schools could reopen. As such, schools were forced to open with COVID-19 protections in place. In this section, we explore the mitigation strategies that were enacted and the shortcoming of the infrastructure in place to protect children in Nampula Province. 64 In Figure 8, we describe the most popular COVID-19 mitigation strategies put in place by schools. We see that the installation of handwashing stations and the requirement that staff and students wear masks are by far the most popular options and almost ubiquitous across schools. Meanwhile, slightly under half of schools enforced distancing within classrooms and a small group of schools only allowed older students to return to class or reached out to parents with information about COVID-19. Altogether, it seems as if a large majority of schools are imposing some sort of COVID-19 protection policy that is in-line with public health guidance. Figure 8: COVID-19 Mitigation during School Reopening Notes: The unit of analysis is the teacher. Bars represent actions taken by the school to ensure student’s safety when reopening. The “hand washing option” represents the installation of a handwashing station in the school. The “Public Information Campaign” Option signifies that the school reached out to parents to inform them of proper COVID protections. The most popular five options are reported. Unfortunately, despite the educational system’s best efforts to enforce COVID-19 mitigation during school reopening, there are several supply chain and infrastructure problems that plague Nampula province and prevent effective mitigation. In Figure 9, we present the largest perceived shortcomings among teachers that are preventing proper COVID-19 mitigation. We observe that across districts, the supply of masks for students is an issue, potentially putting children at risk to the new Delta variant that is proving to be particularly dangerous to children, a variant that is already widespread in Mozambique. Additionally, despite the attempts by schools to impose handwashing, teachers seem to be concerned with the WASH infrastructure in schools. Meanwhile, while within Murrupula and Nacarôa there are some concerns about the size of classrooms, this concern is even more salient in Muecate. Here, teachers seem to be much more worried about the 65 size and ventilation of their classrooms than teachers in the other two districts. This may indicate that classrooms within the district are simply too small and closed off to maintain proper COVID-19 protections during in-class instruction. Figure 9: COVID Mitigation Shortcomings Notes: The unit of analysis is the teacher. Bars represent actions taken by the school to ensure student’s safety when reopening. The “hand washing option” represents the installation of a handwashing station in the school. The “Public Information Campaign” Option signifies that the school reached out to parents to inform them of proper COVID protections. The most popular five options are reported. Overall, the weight of the evidence presented in this chapter tells a story that is intuitive; that COVID-19 and the subsequent policies enacted to curb the spread of the virus, have been major disruptive factors towards the education of children in Mozambique. For those that have returned to school, the nutritional deficit created by lockdowns will hamper their ability to learn and the infrastructural shortcomings of PPE supply chains means that they will be increasingly at risk of more dangerous and more virulent strains of the virus. For those that did not return to school, COVID-19 may have spelled the end of their formal education, reducing their earnings potential over the long term, and increasing the likelihood they will find themselves in poverty later in life. 66 5 Conclusions and Recommendations The evaluation of the ECT2 programme is being undertaken in a context of COVID-19 disruptions and fears, rendering it challenging to present the achievements and impacts of the project. Project reports and stakeholders’ accounts indicate that during the project implementation period prior to 2020, significant strides had been made and several target indicators had been achieved or were on the verge of being achieved. However, by the time of the survey, the level of achievement of the targets in many indicators was low, reflecting the disruptions linked to the COVID-19 pandemic. Overall, the schools in the control group performed better than those in the treatment group, a pattern also captured in the midline evaluation, reflecting that those schools were already better off. When socioeconomic factors are also considered, it is evident that the two groups are not comparable. Murrupula district and the students in the sample are in a better position compared to Nacarôa and Muecate districts and respective students in the sample; in fact, the treatment districts were chosen for inclusion in ECT2 specifically because it was characterized by more adverse student outcomes. Therefore, even if improvements have been made in education in Nacarôa and Muecate because of USDA assistance, that may still be less visible compared to Murrupula, as those districts started from a lower base. In addition, Murrupula has been exposed to other, intensive interventions rolled out by USAID through the Vamos Ler! program. This program also targeted early grade literacy and rolled out substantial support in the form of pedagogical interventions as well as school materials. It was implemented only after Murrupula was already chosen as the control district for this evaluation. Our data also suggest that Nacarôa and Muecate were more affected by COVID-19 than was Murrupula (see chapter 4). All these factors contribute to the interpretation of the results in this report. The conclusion should not be that the ECT2 program did not work. Rather, many of the impact estimates look worse than they should because Murrupula was already better off, and many levels of achievement that look low are because of COVID-19. Below, we summarize the results and provide some recommendations within each of the broad targets of the ECT2 program. The recommendations can serve to inform the next phase of the project, ECT3. Improved Literacy of School-Aged Children Although ECT2 may have contributed to improved literacy levels of early grade students in Nacarôa and Muecate, the difference-in-difference estimate of the EGRA results shows a decrease in treatment relative to control schools. This can be attributed to the shocks linked to COVID-19. These findings indicate that although students in control schools performed better than their peers in treatment schools, both groups are still struggling to read with comprehension. The introduction of bilingual education in Nacarôa and Muecate could be a way to enhance the opportunities of local children to develop early grade literacy skills. The introduction of bilingual education could start with the training of Education Officers and Teachers in that area. Some of these target officers and teachers can be trained as Trainers of Trainers (TOTs) to be responsible for downstream training of others to help ensure sustainability The language of teaching and learning may also have a bearing on the overall poor performance in early grade reading. Literacy instruction is conducted in Portuguese in a context where only 2.8% of the students surveyed reported to speak this language at home (1.4% in Murrupula, 3.5% in Nacarôa and 2.5% in Muecate). As is typical in Mozambican rural settings, most of these students encounter Portuguese for the first time when they start schooling and have scarce access to this language outside their schools. Therefore, at the same time they are struggling to learn the language, they are acquiring basic reading skills. Taking into account national and international recommendations and practices (e.g. UNESCO, 1953, 67 1990), the sociolinguistic situation in the setting in this study and the results reported, it is plausible to suggest that the students in these settings would develop their early literacy abilities better in Macua than in Portuguese. In fact, in the focus groups different stakeholders suggest that the introduction of mother tongue based bilingual education in Muecate and Nacarôa may prove to be a very good way to enhance children’s opportunities to learning. In addition to bilingual reading materials supply, more teachers and school officials should also be trained in bilingual literacy techniques. More Consistent Teacher Attendance Although stakeholders and ECT2 monitoring data indicate that teacher attendance had substantially improved, this study found some decreases when comparing endline results with the midline results. The decrease can be attributed to the disruptions in teaching and learning processes associated with school closures in 2020 and to the irregular functioning of schools in 2021 due to the ongoing COVID-19 pandemic. Measures that were in place before the school closures, such as school council monitoring of teacher attendance and public displays of teacher attendance should be continued. In order to ensure sustainability, the ECT3 created District Platforms of key stakeholder groupings can be made responsible for providing support to the school councils in the discharge of their duties regarding the monitoring of teachers school attendance. Better Access to School Supplies and Materials By the time of the midline survey, almost all classrooms had access to textbooks and other learning materials. However, by the endline survey far fewer teachers reported having adequate textbooks and learning materials. This result can likely be explained by books getting damaged or lost during the year￾long period of school closures. Otherwise, students, teachers, and community members noted the contribution the materials have made to student learning and students are particularly motivated by them. This observation, combined with those above, suggest that providing more books and teaching and learning materials, particularly in Emakhuwa, would benefit children’s learning. In fact, the District Education Directorate was planning to pilot bilingual teaching in the target districts in 2020, but this pilot was cancelled following the onset of the COVID-19 pandemic. WV should pick this initiative up and support the District Education Directorates to procure bilingual reading and teaching materials for supply to the target schools. Improved Attentiveness Project reports and accounts from different stakeholders indicate that ECT2 contributed to improving students’ attentiveness in Nacarôa and Muecate districts. However, the endline study found that the impact of school feeding on students’ attentiveness has dropped. Since many households needed to cut back on food due to the pandemic, there may have been further reductions in attentiveness even if school meals were present. Students and teachers do report increased attentiveness due to the school meals and one of the main recommendations of this report is to continue the school meals. Additionally, to continue school meals in a self-sustaining manner, building up community infrastructure to provide the meals is highly recommended. ECT3 can expand the number of farmer groups supported from the current 90 to about 150 to ensure that all schools have community-based structures like the farmer groups to support in the supply of locally produced food to supplement the USDA provided CSB+ in school meals. Reduced Short-Term Hunger The intervention increased the proportion of students who reported that they were not hungry during school day, as illustrated by the positive trend between the baseline and endline (from 52.6% to 58.8%). However, there was a substantial decrease between midline and endline (from 100% to 58.8%). These 68 patterns may reflect the effects of the COVID-19 pandemic on families’ well-being, as many households decreased the number of meals consumed per day. Increased Access to Food (School Feeding) The number of school aged children receiving daily school meals in Nacarôa and Muecate increased by 42% when comparing the baseline and endline results (from 57,501 to 81,589 students). This increase means that the USDA program boosted the well-being and predisposition to learn for a growing number of children in need, thus enhancing their opportunities to learn and remain in school longer. Improved Student Attendance ECT2 contributed to improving students’ attendance and retention. However, by the time of the endline survey the percentage of students who attended school at least 80% of the school days had dropped relative to the baseline. This decline can be attributed to COVID-19 related school closures and health fears, which prevented students from attending school in a regular basis as well as irregular school schedules once schools reopened. The reduction in recommended days of school attendance per pupil in order to reduce class size and comply with COVID-19 protocols is particularly important. Reduced Health-Related Absences The proportion of students not attending school because of health-related issues at midline had substantially reduced, but at endline, this proportion was again substantial. The COVID-19 outbreak, and associated health concerns may explain this setback, as students were advised to stay home if they were experiencing any COVID-19 symptoms. Increased Knowledge and Use of Health and Hygiene Practices The intervention contributed to improve health and hygiene knowledge and practices in treatment schools despite some challenges still faced in the target schools. The percentage of students surveyed that could identify at least 3 important health and hygiene practices increased from baseline to midline but decreased at endline. The decrease is likely due to some lessons being lost while schools reopened and focused mostly on the basic curriculum. In contrast, the use of appropriate hand washing practices may have only slightly decreased thanks to the fact that hand washing has been widely promoted as a COVID-19 prevention practice. The program should strengthen community based educational campaigns on health and hygiene to ensure parents and other household members also have access to the information being provided to students in schools. Increased Knowledge of Safe Food Preparation and Storage Practices The ECT2 project contributed to increased knowledge of safe food preparation and storage practices in Nacarôa and Muecate treatment schools. Almost all food preparers surveyed (99.4%) were able to identify 3 or more safe food preparation practices, which can be regarded as the outcome of training offered within the scope of the project intervention. Increased Access to Clean Water and Sanitation Services The intervention contributed to increased access to clean water sources and the number of schools with improved sanitary facilities rose. The number of schools with access to clean water increased from 150 in baseline to 160 in midline but decreased to 129 in endline. In contrast, access to improved sanitary facilities increased from 40 in baseline to 137 in midline and then to 159 in endline. Nevertheless, access to clean water and improved sanitary facilities continue to be challenges in many schools and communities in Nacarôa and Muecate and should be supported, especially considering the ongoing pandemic. The follow on ECT3 program should continue in the provision of water and sanitation facilities in schools and possibly 69 find some adaptive ways of extending the water from the boreholes near schools to the communities by pumping to elevated water tanks and linking them to stand-pipes within the communities. 70 References Abadzi, H. (2011) Reading Fluency Measurements in EFA Partner Countries: Outcomes and Improvement Prospects, EFA, FTI Secretariat Update September 2011. In: https://documents.worldbank.org/curated/en/925221468179361979/pdf/797780WP0readi0Box03797 89B00PUBLIC0.pdf (accessed on November 11, 2019) Abeberese, A. B., Kumler, T. J., and Linden, L. L. (2014). Improving Reading Skills by Encouraging Children to Read in School: A Randomized Evaluation of the Sa Aklat Sisikat Reading Program in the Philippines. Journal of Human Resources, 49(3), 611–633. https://doi.org/10.3368/jhr.49.3.611 Ahmed, Akhter U. (2004). Impact of Feeding Children in School: Evidence from Bangladesh. IFPRI Working Paper 2033: 20006–1002. Washington, DC: International Food Policy Research Institute. http://www.lcgbangladesh.org/FSN/reports/IFPRI Final Report_School Feeding in Bangladesh.pdf. Alderman, Harold, and Donald Bundy. (2012). School Feeding Programs and Development: Are We Framing the Question Correctly?, World Bank Research Observer 27 (2): 204–21. https://doi.org/10.1093/wbro/lkr005. Alderman, Harold, Daniel O. Gilligan, and Kim Lehrer. (2012). The Impact of Food for Education Programs on School Participation in Northern Uganda. Economic Development and Cultural Change 61 (1): 187–218. https://doi.org/10.1086/666949. Andrabi, T., Das, J., and Khwaja, A. I. (2012). What Did You Do All Day? Maternal Education and Child Outcomes. Journal of Human Resources, 47(4), 873–912. https://doi.org/10.3368/jhr.47.4.873 Aurino, E., Gelli, A., Adamba, C., Osei-Akoto, I., and Alderman, H. (2020). Food for thought? Experimental Evidence on the Learning Impacts of a Large-Scale School Feeding Program. Journal of Human Resources, 1019. https://doi.org/10.3368/jhr.58.3.1019-10515R1 Bagnol, B., De Sousa, L., Fátima, F. & Cabral, Z. (2015) As Barreiras à Educação da Rapariga no Ensino Primário na Zambézia. Maputo: IBIS. Bando, R., Näslund-Hadley, E., and Gertler, P. (2019). Effect of Inquiry and Problem Based Pedagogy on Learning: Evidence from 10 Field Experiments in Four Countries (Working Paper No. 26280; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w26280 Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukherji, S., Shotland, M., and Walton, M. (2016). Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India (Working Paper No. 22746; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w22746 Banerjee, A. V., Cole, S., Duflo, E., and Linden, L. (2007). Remedying Education: Evidence from Two Randomized Experiments in India*. The Quarterly Journal of Economics, 122(3), 1235–1264. https://doi.org/10.1162/qjec.122.3.1235 Banerji, R., Berry, J., and Shotland, M. (2017). The Impact of Maternal Literacy and Participation Programs: Evidence from a Randomized Evaluation in India. American Economic Journal: Applied Economics, 9(4), 303–337. https://doi.org/10.1257/app.20150390 71 Banerji, R., and Chavan, M. (2016). Improving literacy and math instruction at scale in India’s primary schools: The case of Pratham’s Read India program. Journal of Educational Change, 4(17), 453–475. https://doi.org/10.1007/s10833-016-9285-5 Barrera-Osorio, F., Gertler, P., Nakajima, N., and Patrinos, H. (2020). Promoting Parental Involvement in Schools: Evidence From Two Randomized Experiments (Working Paper No. 28040; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w28040 Beg, S., Fitzpatrick, A., Lucas, A., Tsinigo, E., and Atimone, H. (2019). Strengthening teacher accountability to reach all students (STARS). World Bank SIEF). Milestone, 3. Berry, J., Mehta, S., Mukherjee, P., Ruebeck, H., and Gauri Kartini Shastry. (2018). Implementation and Effects of India’s National School-Based Iron Supplementation Program. http://academics.wellesley.edu/Economics/gshastry/berry%20et%20al%202018%20IFA.pdf Berry, J., Saurabh Mehta, Priya Mukherjee, Hannah Ruebeck, and Gauri Kartini Shastry. (2019). Inputs, monitoring and crowd-out in school-based health interventions: Evidence from India's midday meals program. Working Paper. http://academics.wellesley.edu/Economics/gshastry/berry%20et%20al%202019.pdf Björkman Nyqvist, M., and Guariso, A. (2021). Supporting Learning In and Out of School: Experimental Evidence from India. Working Paper. Daane, M.C., Campbell, J.R., Grigg, W.S., Goodman, M. J. & Oranje, A. (2005) Fourth-grade Students Reading Aloud: NAEP 2002 Special Study of Oral Reading (NCES 2006-469). U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Washington, DC: Government Printing Office. In: https://nces.ed.gov/nationsreportcard/pdf/studies/2006469.pdf. (accessed on November 11, 2019). Drake, Lesley, Meena Fernandes, Elisabetta Aurino, Josephine Kiamba, Boitshepo Giyose, Carmen Burbano, Harold Alderman, Lu Mai, Arlene Mitchell, and Aulo Gelli. 2017. “School Feeding Programs in Middle Childhood and Adolescence.” In Disease Control Priorities 3, edited by D. Bundy, N. De Silva, S. Horton, D. Jamison, and G.C. Patton. Washington, D.C.: The World Bank. http://dcp￾3.org/sites/default/files/chapters/DCP3 CAHD_Ch 12.pdf. Ernst & Young (2018) Mid-Term Evaluation of the Mozambique McGovern-Dole International Food for Education and Child Nutrition Educating Children Together - Phase II (ECT2). Maputo: Ernst & Young. Foster, A. D., and Gehrke, E. (2017). Start What You Finish! Ex-Ante Risk and Schooling Investments in the Presence of Dynamic Complementarities (Working Paper No. 24041; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w24041 Glewwe, P., and Muralidharan, K. (2016). Chapter 10 - Improving Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications. In E. A. Hanushek, S. Machin, and L. Woessmann (Eds.), Handbook of the Economics of Education (Vol. 5, pp. 653–743). Elsevier. https://doi.org/10.1016/B978-0-444-63459-7.00010-5 Grantham-McGregor, S M, S Chang, and S P Walker. (1998). Evaluation of School Feeding Programs: Some Jamaican Examples. The American Journal of Clinical Nutrition 67 (4): 785S-789S. http://www.ncbi.nlm.nih.gov/pubmed/9537629. 72 Hulett, Judie L., Robert E. Weiss, Nimrod O. Bwibo, Osman M. Galal, Natalie Drorbaugh, and Charlotte G. Neumann. (2014). Animal Source Foods Have a Positive Impact on the Primary School Test Scores of Kenyan Schoolchildren in a Cluster-Randomised, Controlled Feeding Intervention Trial. British Journal of Nutrition 111 (05): 875–86. https://doi.org/10.1017/S0007114513003310. Jacoby, Enrique, Santiago Cueto, and Ernesto Pollitt. (1996). Benefits of a School Breakfast Programme among Andean Children in Huaraz, Peru. Food and Nutrition Bulletin 17 (1): 1–11. https://doi.org/10.1177/156482659601700111. Krämer, Marion; Kumar, Santosh; Vollmer, Sebastian. (2018). Improving Children Health and Cognition: Evidence from School-Based Nutrition Intervention in India. GLO Discussion Paper No. 203. Maastricht: Global Labor Organization. Lakshminarayana, R., Eble, A., Bhakta, P., Frost, C., Boone, P., Elbourne, D., and Mann, V. (2013). The Support to Rural India’s Public Education System (STRIPES) Trial: A Cluster Randomised Controlled Trial of Supplementary Teaching, Learning Material and Material Support. PLOS ONE, 8(7), e65775. https://doi.org/10.1371/journal.pone.0065775 Lauchande, C. (2015) Contextual Factors Influencing School Effectiveness in Mozambique. https://www.up.ac.mz/2015-08-21-07-21-15/.../44-revista-udziwi. Mendez, M. A., and Adair, L. S. (1999). Severity and Timing of Stunting in the First Two Years of Life Affect Performance on Cognitive Tests in Late Childhood. The Journal of Nutrition, 129(8), 1555– 1562. https://doi.org/10.1093/jn/129.8.1555 Muralidharan, K. (2017). Chapter 3—Field Experiments in Education in Developing Countries. In A. V. Banerjee and E. Duflo (Eds.), Handbook of Economic Field Experiments (Vol. 2, pp. 323–385). North￾Holland. https://doi.org/10.1016/bs.hefe.2016.09.004 Neumann, Charlotte G, Suzanne P Murphy, Connie Gewa, Monika Grillenberger, and Nimrod O Bwibo. 2007. “Meat Supplementation Improves Growth, Cognitive, and Behavioral Outcomes in Kenyan Children.” The Journal of Nutrition 137 (4): 1119–23. http://www.ncbi.nlm.nih.gov/pubmed/17374691. Popova, A., Evans, D. K., Breeding, M. E., and Arancibia, V. (2021). Teacher Professional Development around the World: The Gap between Evidence and Practice. The World Bank Research Observer, lkab006. https://doi.org/10.1093/wbro/lkab006 Powell, C A, S P Walker, S M Chang, and S M Grantham-McGregor. (1998). Nutrition and Education: A Randomized Trial of the Effects of Breakfast in Rural Primary School Children. The American Journal of Clinical Nutrition 68 (4): 873–79. http://www.ncbi.nlm.nih.gov/pubmed/9771865. Raupp, M., Newman, B. & Revés, L. (2013) Impact Evaluation for the USAID/Aprender a Ler Project in Mozambique: Baseline Report, Revised in June. Raupp, M., Newman, B., Revés, L., Lauchande, C., Alan, E. & Jordan, M. (2016). Impact Evaluation for the USAID / Aprender a Ler Project in Mozambique: Final Report, May. Romero, M., Chen, L., and Magari, N. (2021). Cross-Age Tutoring: Experimental Evidence from Kenya. Economic Development and Cultural Change. https://doi.org/10.1086/713940 RTI International (2018) McGovern–Dole Mozambique Impact Evaluation and Literacy Assessment: Final Baseline Report. Maputo: RTI. 73 Soemantri, A. G., Pollitt, E., and Kim, I. (1985). Iron deficiency anemia and educational achievement. The American Journal of Clinical Nutrition, 42(6), 1221–1228. https://doi.org/10.1093/ajcn/42.6.1221 Stark, K., Chimbutane, F., Peart, D. & Lauchande, C. (2020) Early Grade Reading Asessment and Early Grade Mathematics Assessment in Targeted Schools in Angonia District, Tete Province, Mozambique: EGRA & EGMA Report. Maputo: COWI. Taylor, S., Cilliers, J., Prinsloo, C., Fleisch, B., and Reddy, V. (2019). Improving early grade reading in South Africa. International Institution for Impact Evaluation. https://www.3ieimpact.org/sites/default/files/2019-03/GFR-pw2.10-Early-grade-reading￾southafrica.pdf Turney, A., Noyes, D., Dhliwayo, T. M., Machkasov, Y., Van de Waal, W. & Hua, H. (2018) Effectiveness Evaluation. Early Grade Reading Assessment & Supplementary Tools. Baseline Report – Mozambique (Draft). Maputo: World Education. UNESCO. (1953). The use of vernacular languages in education. Monographs on Fundamental Education 8. Paris: UNESCO. UNESCO. (1990). Education for All by Year 2000. Paris: UNESCO. Vermeersch, Christel, and Michael Kremer. (2005). School Meals, Educational Achievement, and School Competition: Evidence from a Randomized Evaluation. Policy Research Working Paper 3523. Washington, DC: The World Bank. https://doi.org/10.2139/ssrn.667881. Whaley, Shannon E., Marian Sigman, Charlotte Neumann, Nimrod Bwibo, Donald Guthrie, Robert E. Weiss, Susan Alber, and Suzanne P. Murphy. (2003). The Impact of Dietary Intervention on the Cognitive Development of Kenyan School Children. The Journal of Nutrition 133(11): 3965S–3971S. https://doi.org/10.1093/jn/133.11.3965s. 74 Appendix 1: Performance Indicators Table A1. ECT2 Summary of Performance Indicators at Final Report/Evaluation (FE) MGD Result Indicator Gender/Type Unit Baseline MTE FE Final Targets Level of Achieve ment (%) Status SO1 - Improved Literacy of School-Aged Children 1 Improved Literacy of School Aged Children Percentage of students who by the end of two grades of primary schooling demonstrate that they can read and understand the meaning of grade level text (male/female) Male % 0.0 34% 18.5 45.0 41.1 Below target Female % 0.0 31% 8.1 45.0 18.0 Below target Total % 25.1 32.0 13.2 45.0 29.3 Below target Number of individuals benefiting directly from USDA – funded interventions Male Nr. 34088.0 36544.0 43700 36544.0 119.6 Above target Female Nr. 29628.0 31900.0 38495 31900.0 120.7 Above target Continuing Nr 63716.0 68444.0 82195 68444.0 120.1 Above target 1.1 Improved Quality of Literacy Instruction Percentage of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors or coaches % 74.8 100.0 68.8 100.0 68.8 Below target 1.1.1 More Consistent Teacher Attendance Percentage of teachers in target schools who attend and teach school at least 80% of scheduled days per school year % 90.0 96.0 75.6 90.0 84.0 Below target 1.1.2 Better Access to School Supplies and Materials Percentage of Teachers who received textbooks and other teaching and learning materials provided as a result of USDA assistance % 0.0 100 46.8 100.0 46.8 Below target Number of textbooks and other teaching and learning materials provided as a result of USDA assistance Nr. N/A 1634.0 90906 2400 3787.8 Above target Percentage of classrooms with literacy instruction materials (text books, work books) sufficient for effective instruction % 80.0 90.0 82.2 100.0 82.2 Below target 75 MGD Result Indicator Gender/Type Unit Baseline MTE FE Final Targets Level of Achieve ment (%) Status 1.1.3 Improved Literacy Instructional Materials Number of classrooms with literacy instructional materials (textbooks, workbooks,) sufficient for effective instruction * Nr. 337.0 406 450.0 1.1.4 Increased Skills and Knowledge of Teachers Number of teachers/educators/teaching assistants in targeted schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance † Nr. 0 566.0 566.0 Number of teachers/educators/teaching assistants trained or certified as a result of USDA assistance Nr. 0 708.0 773 708.0 109.2 Above target 1.1.5 Increased Skills and Knowledge of School Administrators Number of school administrators and officials in target schools who demonstrate use of new techniques or tools as a result of USDA assistance Nr. 0 106.0 146 240.0 60.8 Below target Number of school administrators and officials trained or certified as a result of USDA assistance Nr. 300.0 282.0 282 300.0 94.0 Below target 1.2 Improved Attentiveness Percentage of students in target schools identified by their teachers as attentive during class/instruction * % 98.0 90.0 98.0 Percentage of students who feel more attentive because of the food they get at school † % 73.6 1.2.1 Reduced Short -Term Hunger Percentage of students in target schools who indicate that they are not hungry during school day % 52.6 100.0 58.8 100.0 58.8 Below target 1.2.1.1 Increased Access to Food (School Feeding) Number of individuals receiving take-home rations as a result of USDA assistant Nr. 0.0 728.0 765 728.0 105.1 Above target Number of take -home rations provided as a result of USDA assistant Nr. 0.0 527,289.0 0 7520000 9726666.7 77.3 Below target Number of school aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance Male Nr. 32775.0 35084.0 39,850 35084.0 113.6 Above target Female Nr. 24726.0 30722.0 34,884 30722.0 113.5 Above target Continuing Nr. 57501.0 65806.0 74,734 65806.0 113.6 Above target 76 MGD Result Indicator Gender/Type Unit Baseline MTE FE Final Targets Level of Achieve ment (%) Status Number of daily school meals (breakfast, snacks, lunch) provided to school aged children as a result of USDA assistance Nr. 12 ,615 ,812 22,174,98 0 39,644,345 51,093,460 77.6 Below target Number of social assistance beneficiaries participating in productive safety nets as result of USDA assistance Male Nr. N/A 36246.0 43,391 36246.0 119.7 Above target Female Nr. N/A 31499.0 38,066 31499.0 120.8 Above target Continuing Nr. N/A 67745.0 81,457 67745.0 120.2 Above target 1.3 Improved Student Attendance Percentage of students that attend school at least 80% of the school days Male % 96.7 85.2 44.7 98.0 45.6 Below target Female % 84.8 74.8 43.2 92.0 46.9 Below target Continuing % 90.8 80.0 43.95 95.0 46.26 Below target 1.3.1 Increased Economic and Cultural Incentives (Or Decreased Disincentives) Number of schools benefiting from saving groups social funds as result of USDA assistance Nr. N/A 146.0 160 120.0 133.3 Above target 1.3.2 Reduced Health Related Absences Percent of students in target schools who miss more than 10 school days/year due to illness Male % N/A N/A 24.6 10 246 Above target Percent of students in target schools who miss more than 10 school days/year due to illness Female % N/A N/A 33 10 330 Above target Percent of students in target schools who miss more than 10 school days/year due to illness Continuing % 2.1 2.0 29.4 10 294 Above target 1.3.3 Improved School Infrastructure Number of educational facilities (i.e., school buildings, classroom, and latrines) rehabilitated/const ructed as a result Schools Nr. 11.0 17.0 48 15.0 320 Above target Storerooms Nr. 150.0 48.0 95 90.0 105.6 Above target Latrines Nr. 440.0 1430.0 1595 1595.0 100 On target Canteens Nr. 150.0 123.0 250 150.0 166.7 Above target 77 MGD Result Indicator Gender/Type Unit Baseline MTE FE Final Targets Level of Achieve ment (%) Status of USDA assistance Wells Nr. N/A 36.0 43 48.0 89.6 Below target Small Water Nr. N/A 12 12 12 100 On target 1.3.4 Increased Student Enrollment Number of students enrolled in school receiving USDA assistance Male Nr. 32775.0 35084.0 42891 35084.0 122.3 Above target Female Nr. 24726.0 30722.0 37623 30722.0 122.5 Above target Continuing Nr. 57501.0 65806.0 80514 65806.0 122.4 Above target 1.4.1 Increased Capacity of Government Institutions Number of district and Provincial MINED officials who know standard operating procedures and tools for management and oversight of school feeding programs and Literacy Boost Nr. 0 48.0 48 48.0 100 On target 1.4.3 Increased Government Support Percentage of Education coordination meetings where school feeding is part of the agenda % 0 31.0 100 100.0 100 On target 1.4.4 Increased Engagement of Local Organizations and Community Groups Number of community groups that report having access to meaningful opportunities for feedback about project implementation Nr. 0.0 37.0 41 37.0 110.8 Above target Number of Parent-Teacher Association (PTAs) or similar “school” governance structures supported as a result of USDA assistance Nr. 0.0 160.0 160 160.0 100 On target Number of public - private partnership formed as result of USDA assistance (nutrition) Nr. 0.0 80.0 80 80.0 100 On target SO2-Increased Use of Health & Dietary Practices 2 Increased Use of Health and Dietary Practices Percent of target beneficiaries who use appropriate hand washing practices Male % 0.0 94.0 88.7 95.0 93.4 Below target Female % 0.0 95.0 90.3 95.0 95.1 Below target Continuing % 0.0 94.0 89.3 95.0 94.0 Below target Percent of school￾aged children receiving a minimum acceptable diet Male % 0.0 53.0 38.0 53.0 71.69 Below target Female % 0.0 47.0 40.6 47.0 86.38 Below target Continuing % 0.0 100 39.4 100 39.4 Below target 78 MGD Result Indicator Gender/Type Unit Baseline MTE FE Final Targets Level of Achieve ment (%) Status 2.1 Increased Knowledge of Health and Hygiene Practices Percentage of children in target communities who can identify at least 3 important health/hygiene practices % 61.7 86.0 31.6 90.0 35.1 Below target 2.2 Increased Knowledge of Safe food prep and Storage Practices Percentage of food preparers at target schools who can identify at least 3 key practices aimed at safe food preparation % 0 100.0 99.4 100.0 99.4 Below target 2.3 Increased Knowledge of Nutrition Number of individuals trained in child health and nutrition as result of USDA assistance Male Nr. N/A 482.0 520 818.0 63.6 Below target Female Nr. N/A 1005.0 1017 546.0 186.3 Above target Continuing Nr. N/A 1487.0 1537 1364.0 112.7 Above target 2.4 Increased Access to Clean Water and Sanitation Services Number of schools using an improved water source Nr. 150.0 160.0 129 160.0 80.6 Below target Number of schools with improved sanitary facilities Nr. 40.0 130.0 159 145.0 109.7 Above target 2.5 Increased Access to Preventive Health Interventions Number of schools who receive at least 2 visits per year from health facility staff Nr. N/A 147.0 160 160.0 100 On target Number of students receiving de￾worming medication(s) Nr. N/A 55949.0 71258 59411.0 119.9 Above target 2.6 Increased Access to requisite Food prep and storage tools and equipment Number of schools with appropriate food prep and storage equipment Nr. 150.0 160.0 160 150.0 106.7 Above target Note: Indicators denoted with an asterisk (*) were not measured during the endline survey. Indicators denoted by † are the indicators used as proxies for those that were not measured. 79 Appendix 2: Evaluation Terms of Reference (TOR) World Vision Mozambique Nampula Province (Muecate and Nacaroa Districts) USDA Food for Education FFE-656-2015/011-00 Educating Children Together Phase 2 (ECT2) Project Final Program Evaluation Terms of Reference (ToRs) / Scope of Work Attachment A 80 1. Introduction World Vision Mozambique is seeking the services of a consulting firm to perform a Final Evaluation of its USDA-funded, Food for Education project in Nampula, Mozambique. World Vision Mozambique, through funding from USDA's McGovern-Dole International Food for Education Program, is currently implementing a five-year school feeding project, called Educating Children Together Phase 2 (ECT2), launched in 2016 in coordination with the Mozambique Ministry of Education, specifically PRONAE (National Program for School Feeding). ECT2 is expected to build upon the project that WV implemented under ECT1 from 2013 to 2015 and inform the project that WV is implementing under ECT3 (2020-2024). Several key activities are undertaken through this project to improve literacy, health and nutrition for school children. This includes the provision of daily school meals, the provision of school materials as well as educational capacity building within an enabling environment intended to improve literacy, attendance and enrollment rates of school children. While ECT2 project activities come to a close and ECT3 activities begin, it is important to measure progress toward the ECT2 goal and objectives and/or identify constraints occurring during implementation through a Final Evaluation. This evaluation will create feedback for the decision￾making process related to long-term planning and recommend any adjustments necessary to the implementation strategy and results framework for ECT3. Since ECT III starts in Q4 2019, while the current ECT II ends Q4 2020, there will be an overlap period during which the ECT II endline evaluation will be conducted. Since this is a continuation project working in the same target areas, with a similar target population, WV will seek to reduce over-burdening communities and project participants and increase efficiency by utilizing the ECT II endline survey as the ECT III baseline (referred to in this document as the “Final Evaluation”). This is suitable since the scope of activities and indicators for ECT II and ECT III are largely similar. This document describes the objectives and goals of this Final Evaluation. It also explains the logistical details to be considered during the data collection process and steps, technical procedures and tools to be used. An independent consulting firm will be contracted to conduct the evaluation. 2. Project Background World Vision (WV) has 30 years of experience in Mozambique and 15 cumulative years of experience implementing USDA's Food for Education (FFE) programs in Afghanistan, Mongolia, Mozambique, and Nicaragua. In addition, WV is currently implementing USDA FFE programs in Nicaragua, Rwanda, Cambodia and Haiti, partnering with the World Food Program. The overall goals of this five-year Mozambique ECT2 project are to improve literacy skills of primary school aged children and to improve health and dietary practices. The primary interventions are aimed at improving the quality of literacy instruction, student attentiveness, and student and teacher attendance. World Vision is implementing the project with Save the Children International (SC), who is responsible for the literacy component. Baseline Study: In 2016, during ECT2’s first year of implementation, the final evaluation results of ECT1 were used as baseline results for ECT2. The results from this ECT1 study were used to set the project’s overall targets and measure performance during routine monitoring. The study 81 was conducted by an external consultant firm - COSDER Consultants. Both quantitative and qualitative methods were used to interview key informants and direct beneficiaries, obtaining information on needs, challenges, attitudes and behaviors of the target groups. Mid-Term Evaluation: The mid-term evaluation for ECT2 was conducted by an external consultant firm- Ernst and Young (E&Y). The purpose of the mid-term evaluation was to examine evidence of early changes in the target communities – both positive and negative – and compare them to the changes anticipated in the Results Framework. It identified factors in the project implementation and context that appeared to promote or obstruct those early changes that had been identified. The specific objectives were: (a) Assess implementation progress and constraints to determine the likelihood of achieving target results; (b) Assess relevance of interventions, provide evidence of effectiveness of the interventions and identify, explain and learn from successful strategies as well as challenges; (c) Assess early signs of sustainability, review service and input delivery mechanisms and the quality of services (e.g. trainings, sensitization sessions, school activities, distribution of food and non-food items, etc.) and highlight stakeholder’s views on or perceptions of project interventions, i.e. what is working and what adjustments need to be made; (d) Measure the extent of implementation of ECT Phase 1 evaluation findings and lessons learned from Phase 1; (e) Identify and document new or continuing lessons learned, challenges, good practices and recommendations. The evaluation results showed that the implementation of ECT2 had been successful particularly when the baseline outcome indicators results were compared with the midterm outcome indicators results. The project vision, strategies and standards are in line with Government of Mozambique National Policies and Strategies for Education and Food Security. The preliminary results of the project implementation bring positive lessons to PRONAE (School Feeding National Program) to support the school feeding programs’ policy direction. In addition, a clear alignment of the project with the beneficiaries’ needs was perceived. ECT3 Additional Components: Furthermore, to improve the quality of literacy instruction, the new five-year funded project (ECT3) is bringing innovations to the same targeted districts including: (a) the Unlock Literacy (UL) Model which is expected to build on the community action module to promote literacy outside the classroom and engage parents, local leaders and the broader community to value and support child education; (b) Take-home rations to benefit 24,032 Pregnant, lactating women (PLW) and 16,696 children under 2 (CU2) in coordination with the Ministry of Health (MoH) and Community Health Committees (CHC). These Health and Nutrition activities are part of the thousand (1000) days approach aiming to reduce risk factors for stunting, which can negatively impact development outcomes for infants and young children and affect their ability to learn during schooling years, and; (c) WV’s sub-recipient, the Center for the Learning and Training of Civil Society (CESC), a non-profit Mozambican civil society organization, will engage local organizations and community groups, especially regarding their roles in promoting school enrollment and literacy education. These additional project activities will need to be taken under consideration during data collection. Audience: The participation of a wide cross-section of key stakeholders will be essential to the study and should include: 1) school children, cooks, teachers, school administrators, community health volunteers, parents and farmer’s groups, 2) food monitors and field facilitators, 3) WV M&E 82 team and technical specialists, 4) Ministry of Education, Health, Agriculture officers, and 5) local government entities. Research Design & Methods: The final evaluation report of ECT1 should be utilized during the Final Evaluation of ECT2 to review the achieved targets for ECT2, assess the quality of the implemented activities and measure the development results. Data collection methods for this study should include: 1) review of project information and analysis of relevant documents, tools and prior assessments done, including the 2017 RTI Impact Evaluation Baseline Data for literacy assessment for students as well as the Mid-Term Evaluation Report for ECT2, 2) questionnaires for teachers, deputy school directors, cooks, and students to evaluate minimum acceptable hygiene and health practices captured directly by enumerators into electronic tablets, and 3) key informant interviews/focus group discussions. 3. Project Description The project theory of change posits that improved literacy, health, and hygiene instruction paired with regular school meals will lead to improved literacy, health, and dietary practices of students via various pathways, including increased student and teacher attendance, improved student attentiveness, and increased knowledge of health and hygiene. The intended beneficiaries of the McGovern-Dole program are primary school students in targeted districts in Nampula Provinces, their families, and communities. World Vision, Inc. (WV Inc.) is using USDA donated commodities and any funds provided by FAS to implement a school feeding project in Mozambique over a period of approximately five years, focusing on achieving the following objectives: • Improve the literacy of school-age children through higher quality literacy instruction and increased student attentiveness and attendance; • Improve the quality of literacy instruction through better-trained, more-available instructors and administrators and better access to improved literacy materials; • Improve attentiveness by reducing short-term hunger with daily school feeding; • Improve student attendance through promoting the benefits of education and enrollment, improving school infrastructure, providing furniture and equipment to schools, and raising awareness of the importance of education and the barriers that affect school attendance; • Improve health and dietary practices by training health workers in good health and hygiene practices; • Improve knowledge of health and hygiene practices through training of communities in good health and nutrition practices; • Increase knowledge of safe food preparation and storage practices through training in commodity management and safe food preparation and storage practices; • Increase access to requisite food preparation and storage tools and equipment by building kitchens and storage facilities and providing energy-saving stoves; • Increase access to clean water and sanitation services by building and rehabilitating latrines and water access points; and • Increase access to preventive health interventions through distribution of essential medical supplies, vitamin and mineral supplements, and de-worming medication. 83 Approximately 64,000 students are benefitting at 160 schools in Muecate and Nacaroa districts of Nampula. Districts were selected based on USDA guidance, GoM school feeding priorities, and overlap with ongoing programming areas. ECT2 covers 100% of primary schools in the two target districts to work within existing government structures to avoid creating conflict in communities and prevent students from moving within districts to schools where feeding is available. The project results framework supports the MGD Program Results Frameworks by aligning each ECT2 result with a result of the MGD Strategic Objectives 1 and 2. See the project results framework at the end of this document. 4. Purpose of the Final Evaluation The purpose of the final evaluation is to assess whether the project has achieved the expected results as outlined in the results framework, or in other words, to determine effectiveness. The final evaluation will also assess areas of project design, implementation, management, lessons learned and replicability. Considering that the final evaluation will provide lessons learned and recommendations for USDA, program participants, and other key stakeholders for future food assistance and capacity building programs, understanding the impact and outcomes of school feeding will be extremely important to inform future policy and strategy. The results of the final evaluation will also serve as the baseline for ECT3. In this regard, the evaluation team will be expected to review the ECT3 PMP in order to identify any relevant additional outcome and impact indicators to cover during the evaluation. 5. Objectives of the Final Evaluation In general, the final evaluation should assess: Relevance The extent to which the project interventions met the needs of the project beneficiaries and is aligned with the country’s agriculture and/or development investment strategy and with USDA and US Government’s development goals, objectives, and strategies. Relevance should also address the extent to which the project was designed taking into account the economic, cultural and political context and existing relevant program activities. • Did the project address the most critical problems or constraints to improve educational outcomes? • Are the constraints faced by the target beneficiaries as outlined in the original document still relevant? Effectiveness The extent to which the project has achieved its objectives. Effectiveness should also assess the extent to which the interventions contributed to the expected results or objectives. • To what extent did the project theory of change contribute to the achievements in terms of project results and outcomes? This evidence will be descriptive rather than causal. Efficiency The extent to which the project resources (inputs) have led to the achieved results. An assessment of efficiency should also consider whether the same results could have been achieved with fewer resources or whether alternative approaches could have been adopted to achieve the same results. • To what extent did the project resources lead to achieved results? Sustainability 84 Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. • Which were the factors that had an impact in the achievement or non-achievement of the project? Impact Assessment of the medium and long-term effects, both intended and unintended, of a project intervention. Effects can be both direct or indirect and positive or negative. To the extent possible, the evaluation should assess the extent to which the effects are due to the project intervention and not other factors. • To what extent have the outcomes contributed to capacity development of beneficiaries, government of Mozambique, and community-based organizations? • What has been the lessons learned from this project and how might best practices be adapted for future projects? Project Design Improvement In addition to the focus on relevance, effectiveness, efficiency, sustainability and impact as described above the evaluation may focus on other areas of particular interest to USDA. • In what ways were the foundational results well designed to contribute to the project outcome? 6. Approach & Methodology Research Design: The ECTII endline/ECTIII baseline study will include a quasi-experimental design (QED) in which the baseline will be comprised of the intervention districts and a control district, for greater external validity and attribution of project outcomes. The control district was carefully selected under ECT II baseline to ensure it is comparable or similar at baseline in the following areas: individuals and community characteristics, culture, presence of other NGOs, education outcomes, and agricultural and health systems. WV will select the same control group to extend the longevity of comparison between the intervention and control groups. Research Methods: The endline/baseline study will combine both qualitative and quantitative data collection methods to assess the status of each indicator in the PMP highlighted in red below and a plan for collecting and analyzing data. The following are the data collection methods: 1) review of literature and analysis of relevant documents, 2) literacy assessment for school pupils, utilizing the Early Grade Reading Assessment tool (EGRA), 3) school-based survey of teachers, deputy school directors, cooks, and pupils, and 4) key informant interview/focus group discussions. Quantitative data: The quantitative data collection will use school-based surveys. In the ECT3 districts of Nacaroa and Muecate, schools will be randomly allocated to one of three groups: school meals only; school meals and Unlock Literacy’s teacher component; and school meals, UL teacher component, and enhanced community component. The randomization technique will be stratified randomization within ZIPs to ensure equal representation of the treatment groups within ZIPs. A detailed survey methodology similar to the ECTII methodology (including the specific sampling calculations and sizes) will be developed by the external consultant. Qualitative data: Qualitative data collection will be conducted through focus group discussions. 85 Subgroups within each community will be selected to better understand the effectiveness of key interventions, including students, volunteer cooks, community health volunteers, community leaders, farmer group members, school councils and teachers. Key informant interviews will be conducted with implementing partners, government officials, and school administrators. In addition, on-site observations will be conducted at the school to gain insights into classroom activities and school feeding at the midline of ECT3. Secondary data: The evaluators will also access and review all relevant internal and external secondary data, including data sets from the recent RTI Impact Evaluation Baseline Survey, that directly and indirectly inform the project. An illustrative, but non-exhaustive list of sources of this data that are internal to the project include: proposals, budgets, baseline reports, strategies, plans, reports, studies, assessments, monitoring forms, implementation guidelines and policies, training manuals, and others. An illustrative, but non-exhaustive list of data sources that are external to the project include Mozambique Ministry of Education statistics, list of key informants, contacts, PTA general plans, and food distribution reports. It is also understood that these documents should be used to develop topical outlines and tools for the qualitative portion of the evaluation. Follow-up data collection: Project indicators highlighted in red in the WVUS-IFPRI Sub-grant Agreement, FFE-626-3019/018-00 (Attachment D) will be measured to establish benchmarks against progress during the final evaluation. The collected data will help project staff understand challenges and successes and adjust ECT3 implementation methodologies to better help the project succeed and attain its goals before after final evaluation. The final evaluation data collected and analyzed will also create an atmosphere of learning and adaptation of good practices for future programs. It will be used as a dissemination tool at the municipal and government level to assist the government and the community in strengthening its policies on literacy education and health. Data (quantitative and qualitative) will be provided to WV in full in anonymized form (removing identifiers such as names, locations, etc. Only IDs will be provided), per rules of IRB and privacy laws. IFPRI will own all the data collected, both qualitative and quantitative to publish in peer￾reviewed journals and other outlets. 7. Evaluation Team Criteria used for selection of independent consultants will include: 1) financially and legally separate from implementing partners, 2) have staff with demonstrated knowledge, analytical capability, language skills (Portuguese and English) and experience in conducting evaluations of development programs involving agriculture, education, and nutrition in Mozambique, 3) use acceptable analytical frameworks such as comparison with non-project areas surveys, involvement of stakeholders in the evaluation, and statistical analyses, 4) use local consultants, as appropriate to conduct portions of the evaluation, and 5) provide a detailed outline of the evaluation, major tasks, and specific schedules prior to initiating the evaluation. 8. Audience and key stakeholders There is a broad group of stakeholders who are interested in the achievements of this school feeding project, given that an important component of the Final Program Evaluation is to determine whether the project has achieved the intended results and outcomes or not. Direct beneficiaries of the program, project staff, implementing partners (SC in ECT2 and CESC and IFPRI in ECT3), the Ministry of Education and Human Development (MINEDH), Ministry of Agriculture and Food 86 Security (MASA), Ministry of Health (MoH) and USDA will be considered as primary stakeholders. Direct beneficiaries include: school children and their families/households who receive program services, school councils, farmer associations, health committees and other community groups who participate in the ECT2 project. The findings from the Final Evaluation will be particularly important to program staff and partners, as it will help identify best practices and critical areas of the program that need to be fully addressed, as well as developing practical ways to enhance implementation. The donor, USDA, will be critically interested in the Final Evaluation to determine whether the project has reached the expected goals and objectives, as well as to inform similar interventions in Mozambique and other target countries. 9. Roles and responsibilities Evaluator/ Contractor Responsibilities The contractor will be responsible for logistics and support of the evaluation, including hiring of the evaluation staff, vehicle hire and transportation, translation services, printing, etc. The ECT2/ECT3 program will provide office space as requested in evaluation target areas. ECT2/ECT3 program vehicles will NOT be available for use in data collection or transport of evaluation personnel. The ECT2/ECT3 program will provide the venue and associated costs for briefing and debriefing meetings and the presentation of evaluation results. Final Evaluation Team Composition and Qualification The Final Evaluation team should consist of a team leader plus technical specialists in food security, child health, nutrition and education. No member of the Final Evaluation team will have had any responsibility in the design or implementation of the program under evaluation. The team leader must be external to the ECT2/ECT3 program and all agencies involved in program implementation. To ensure independence as a third-party and avoid disruption in program implementation that could affect the evaluation results, the Final Evaluation team must not use ECT2/ECT3 staff as translators, enumerators or supervisors. During data collection and analysis, the primary role of ECT2/ECT3 staff members are as informants and observers. They may review and provide comments on data collection tools and instruments before they are finalized. They may observe some of the Final Evaluation process, but they will not collect primary data, or participate in translation, analysis or interpretation of this data. Team leader qualifications: • Must possess a post-graduate degree (program evaluation, statistics, anthropology, applied research, organizational development, sociology and/or organizational change) • Must possess extensive evaluation experience using mixed methods in developing countries • Must be knowledgeable in conceptual frameworks • Must be experienced in evaluation of food security programs, with strong preference toward USDA FFE programs. • Must be bilingual in Portuguese and English, with high writing proficiency in English. Team Leader responsibilities: • Organize and lead the overall evaluation • Ensure a thorough review and analysis of project and secondary data • Lead the sample selection and outputs for primary data collection 87 • Ensure adequate triangulation and validation of evidence collected • Assess information about the project’s M&E processes and the integration of project sectors and interventions through qualitative interviews. • Ensure that 1) final report presentation is logical, well-written, and presented in a way that clearly separated the evidence collected, conclusions, and recommendations in different sections of the report, and 2) all evidence, conclusions and recommendations are based on the evidence presented in the report; • Liaise with World Vision and USDA at the inception • It would be preferable and advantageous for the Team Leader to also serve as one of the technical sector team members. Team member qualifications: • Must possess substantial application of quantitative and qualitative research skills and analysis in one of the following areas (with all areas covered by the collective team) food security, child health, nutrition, gender and education in developing countries. • Must have extensive practical experience in one of the following areas (with all areas needing to be covered by the collective team) food security, child health, nutrition, gender and education. • A postgraduate degree related to one or more of the project’s technical sectors is preferable. Team member responsibilities: • Lead the collection and analysis of primary and secondary data related to his/her field(s) of expertise • Document findings, draw conclusions and form recommendations for the sector(s) • Evaluate the general aspects of the implementation of all interventions related to his/her sector(s) World Vision responsibilities: • Conduct a review of and provide timely feedback and approval of all draft deliverables listed above under contractor responsibilities. • Provide an illustrative list of secondary data, made available to the evaluators at least one month before the start of the qualitative data collection activity. • Logistical and Administrative Guidance and Support: o Arrange meetings between the evaluation team and USDA – at the beginning and end of the evaluation process. o Advise about local protocols and permissions to gain entry to operational areas o Provide advice related to travel (international travel, local vehicles and drivers for hire) o Identify local firms with potential to provide technical expertise – including translation o Provide office space in the ECT2/ECT3 program areas as needed for meetings, desk work, and presentations. o World Vision will provide a liaison/contact person who will be in close communication with the consulting team leader to coordinate the development and implementation of the evaluation process. *Note: World Vision will NOT arrange enumerators and logistics (travel documents, health insurance, laptops, flights, and ground transport) for the evaluation team. Furthermore, World 88 Vision vehicles are not permitted for use in Final Evaluation activities. This is to ensure the highest level of independence for the consultant in the evaluation. 10. Required Deliverables • Inception Report aimed to be submitted 6 weeks from the contract signing date. This will cover the final evaluation schedule, Evaluation matrix demonstrating the linkages between evaluation questions, type of data to be collected, data sources, methods of data collection, data analysis and expected reports; evaluation design and methodology; draft data collection tools; sampling and quality assurance plan. A final due date will be determined alongside WV to take into account Covid-19 restrictions. • Draft Evaluation Report • Final Evaluation report including executive summary, data analysis, findings, and recommendations/conclusions as well as the following: o Data collection instruments (English and all translations) o Lists of sites visited with types and numbers of informants at each site o Limitations to the study o Quantitative and qualitative datasets. o Raw (anonymized) qualitative data including transcripts of interviews and focus groups discussions; and notes or products of observations or other qualitative methods o Raw (anonymized) quantitative data including data sets and related materials o Presentation of the evaluation findings, conclusions, recommendations, lessons learnt and good practices to the project stakeholders. 11. Illustrative Timeline World Vision plans to conduct this evaluation May 2020 to September 2021. Below is an illustrative timeline of the activities to be completed. Key Final Evaluation activities Month/Year Develop TORs for consultancy February/March 2020 Recruit consultant March/April 2020 Evaluation design with partners October 2020 Enumerator training TBD but hopefully February 2021 Data collection, entry and analysis February/March 2021 Draft report writing, submission and presentation July 2021 Refining and submission of final report September 2021 Dissemination of results September 2021 Due to Covid-19 restrictions, this timeline will be updated as needed. 12. Evaluation Management Overall Roles and responsibilities: Chief of Party: Will ensure the coordination and technical oversight of all evaluation processes from the baseline up to the final evaluation. S/he will provide information about the communities, partners, and community groups, share key documents & lessons learned, obtain approval for 89 evaluation activities, provide feedback to the evaluation report and prepare for dissemination with stakeholders, including relevant government ministries and the donor. Project Staff: Will work alongside the M&E team and consultants to develop appropriate methodologies and questionnaires. They will also be the conduit between the M&E team and the community, informing and requesting consent for interviews, assisting in the data collection process, sharing results findings, gathering lessons learned, and working with the community to make informed decisions on the direction of project activities. Monitoring and Evaluation Manager: Will hire and prepare consultancy TORs for all evaluation phases in consultation with WVUS, USDA and project stakeholders. S/he will provide technical guidance in the methodologies selected and lead the WV and consultancy M&E team in the data collection, analysis and reporting process. Monitoring and Evaluation Team: Will perform the groundwork for the project, providing information on communities, partners, community groups, and associations as well as report on progress of project activities. The team will also coordinate all logistics and field activities, recruit enumerators and data collection team members, mobilizing beneficiaries for participation in evaluation tasks. They will provide technical oversight to consultancy field sight ensuring accountability and quality data is collected from the field. Consulting Firm: The firm will conduct a desk review from project records, secondary data and literature; consult with the project team and stakeholders to prepare evaluation tools, sampling methodologies, and review data collection; provide leadership and oversight during field data collection; train enumerators, data entry clerks and field supervisors; organize and conduct focus group discussions, key informant interviews, observations and site visits; conduct data analysis and present preliminary findings to the project team. The firm will also prepare and share the draft report with stakeholders for feedback, make a presentation to disseminate findings and incorporate feedback to finalize the evaluation report. Roles and responsibilities of key stakeholders throughout the evaluation process: Project Partners: (USDA, Provincial officials, government of Mozambique): Project partners will provide input and feedback on the TOR and the evaluation reports. SC: Implementing partner will assist with Literacy Assessment Data Collection, as well as provide feedback on the evaluation reports. WVUS: Program Management Officer, M&E Specialist and Senior Technical Advisors will provide input and feedback on the TORs, as well as provide feedback to the evaluation reports. Community: Will include beneficiaries, community leaders and organizations, parents, and teachers. The community will work with WV as key informants, participate in focus group discussions, key informant interviews, and provide feedback in the preliminary analysis and shared results of baseline, midterm, and final evaluation. The government will use the data results to make informed decisions that affect WV target catchment areas. 90 Appendix 3: Survey Instruments IFPRI PRIMARY SCHOOL SURVEY: CLASS NAMES AND ATTENDANCE REGISTER, GRADE 4 PUPILS QUESTIONNAIRE DASHBOARD: A01 School ID Enter school ID from school sample list A02 School Name Prefilled A03a District Prefilled A03b Administrative Post Prefilled A04 Locality Prefilled A05 Village Prefilled A06 ZIP number to which school belongs Prefilled A07 Enumerator ID Enter enumerator ID A08 Date and time stamp Automatic A08a Is there a Grade 2 class register from 2020? 1 – Yes 2 – No >> A09a A08b From which term is this register? Enter number (1-3) A08c Screen capture of Grade 4 class register Photo A09a Is there a Grade 4 class register for 2021? 1 – Yes 2 – No >> Section B A09b From which term is this register? Enter number (1-3) A09c Screen capture of Grade 4 class register Photo Enumerator: You will build a complete Grade 4 register for four types of pupils: (1) pupils who were enrolled in grade 2 in 2020 before schools closed due to the pandemic, (2) all pupils on the current (2021) school register for grade 4, (3) pupils who are present in the class but not on the school register from grade 4 and attend and do the work of the grade 4 class regularly, (4) pupils who are absent in the class (but should not be depending on the week they usually attend school) but not on the school register for grade 4 but attend and do the work of the grade 4 class regularly, Type 1: Pupils enrolled in grade 2 in 2020 before schools closed. Enter the names of all the pupils listed on the school’s register from 2020 for grade 2. Type 2: Pupils on the current (2021) school register for grade 4. Enter the names of all the pupils listed on the school’s register from all grade 4 classes. Type 3: Pupils who are present in class but are not on the school register for grade 4 who regularly attend and do the work of the grade 4 class. Ask if there are any such pupils. If so, enter their name onto our register. Type 4: Pupils who are absent in class (but should not be depending on the week they usually attend school) but were not on the school register for grade 4 who regularly attend and do the work of the grade 4 class. Ask if there are any such pupils. If so, enter their name onto our register. 91 Remember not to enter duplicate names. Verify with the pupils in the classroom whether there are actually two pupils with the same name. If not, only enter once. If there are two pupils with the same name, include their father’s name or other identifying name to differentiate the two. Once all the pupil names have been listed for all four types, the tablet will go to the first pupil whose name was entered and you will ask a series of questions about each student. Questions should be directed to the entire class. SECTION B: NAMES OF PUPILS IN THE GRADE 4 POPULATION Pupil Number Pupil Name Name of the class (if more than one stream) (use A, B, C, etc.) Is the pupil on the school’s paper register? 1 – Yes 2 – No 3 – No paper register (if B02=3 for the first pupil, the tablet will skip this question for all pupils) What is the gender of this pupil? 1 – Male 2 – Female What is the age of this pupil? Number Was this pupil enrolled in school before schools closed due to Covid-19 (before March 2020)? 1 – Yes 2 – No >>B08 Has this pupil returned to school since schools re￾opened? 1 – Yes >>B09 2 – No >>B10 Did this pupil enrol after the reopening of schools (new pupil)? 1 – Yes 2 – No Has this pupil attended this class at least once this term? 1 – Yes >>B13 2 – No Is this pupil coming back to this class this term? 1 – Yes >>B13 2 – No 3 – Don’t know >>B13 Why is this pupil not coming back to this class this term? 1 – Promoted to Grade 5 2 – Demoted to Grade 3 3 – Transferred schools 4 – Dropped out 5 – Other 6 – Don’t know 7 – This is a fake pupil >>next pupil if B11 is not 4 Why did this pupil drop out? 1 – parents couldn’t afford school 2 – had to work for money 3 – had to work on the farm 4 – had to help take care of siblings 5 – got sick 6 – passed away 7 – other, specify >> next pupil Is the pupil present in class today? 1 – Yes 2 – No >> next pupil Did the pupil’s parent / guardian consent to the pupil participa ting in the study? 1 = Yes 2 = No >> next pupil Can the pupil take the exam (not sick or disabled ?) 1 = Yes 2 = No B00 B01 B02 B03 B04 B05 B06 B07 B08 B09 B10 B11 B12 B13 B14 B15 Automatically generated Automatically generated Automatically generated B16. How many children did you have to ask to leave because you thought they did not belong in the Grade 4 class? ___________________________ [NUMBER] The CAPI should only select students to whom the EGRA and questionnaire should be administered if B13, B14, and B15 are all Yes. 92 IFPRI PRIMARY SCHOOL SURVEY: DEPUTY SCHOOL DIRECTOR QUESTIONNAIRE DASHBOARD: A01 School ID Enter school ID from school sample list A02 School Name Prefilled A03a District Prefilled A03b Administrative Post Prefilled A04 Locality Prefilled A05 Village Prefilled A06 ZIP number to which school belongs Prefilled A07 Enumerator ID Enter enumerator ID A08 Date and time stamp Automatic A09 Deputy school director’s ID Automatically generated A10 Deputy school director name Enumerator: attempt to interview the Deputy School Director Informed Consent Good Morning/Good Afternoon. My name is (NAME), I work for a team of researchers in the United States and in Mozambique. The research team is collecting data for an education research project and we would like to invite you to participate in this survey. To help you decide if you want to accept to participate, I will give you more explanation about what we are trying to do. If in doubt, you can ask for clarification at any time. If you need to, you can ask for time to reflect or consult someone you trust. Why is this research being carried out? This research is being carried out to gain an understanding of how education works in Mozambique. Many other schools will be participating in the same research study. The research will investigate how to improve education programming in Mozambique. Research Objectives The objective is to analyze the effectiveness of education programming implemented by World Vision in some districts in Nampula province. Type of Research/ Intervention The data collected during this research will be used in a statistical model that will allow measuring the effect that the educational programs implemented by World Vision have on students’ literacy performance. Selection of participants This survey will include many schools from three districts in Nampula province. They have been selected from a group of schools just like yours. Voluntary Participation Your participation is voluntary. You are not required to participate in this research. If you decide not to participate, there will be no harm to you. If you decide to participate, you can interrupt at any time without prejudice. There are no sanctions or consequences if you decide you don't want to participate. You can also choose not to answer any specific question in the interview, and you can also ask us not to use your information at any time. Risks, Discomfort, and Inconvenience There will be very little risk to you from the study. Your studies will not be impacted. The interview will take approximately 30 minutes. As you can see, we are taking considerable precautions regarding Covid-19. All interviews will take place outside, the enumerator will be 6 feet away from the respondent or will have a clear plastic barrier in front of them, the enumerator will always wear a mask and will sanitize his/her hands and equipment before and after each interview. Enumerators have their temperatures checked every morning and nobody who is ill is allowed to come to the school. 93 Benefits We cannot promise any benefits to you or others who participate in this survey. Your participation in the study is very important, we hope that your participation in the study can help us to improve education in Mozambique. Cost of participation/ Compensation You will receive a token of our appreciation for participating in this study and there will be no cost to you for participating in this study. Privacy Your name, as well as any other information that can be used to identify you, will not be shared with anyone including the school or the government. No one, except one researcher, will be able to access the information and see any answers. All information will be stored in an encrypted, password-protected folder that only the lead researcher will have access to. Confidentiality All information you give to the interviewer will be kept confidential. We will never report on individual information, only describe general standards and the conclusions resulting from the analyses of the information provided by all participants. No data identified by individual participants will be published for third parties who are not project personnel, and no data collected during the study is considered to be sensitive in nature. All information that could be used to identify the participant will be treated, protected, and accessed only by the team authorized for research. Sharing Results The results of this study will result in recommendations and the elaboration of education policies, which will be shared with the Ministry of Education and Human Development of Mozambique, international institutions, and education practitioners in Mozambique and outside Mozambique. The results of the research will also be published in conferences, seminars, workshops, and scientific publications. Whom to Contact (Researchers and Committee of Ethics) Researchers: In case you would like to speak with someone about the research, please contact: Dr. Feliciano Salvador Chimbutane Universidade Eduardo Mondlane, Maputo Tel: +258828173490 Email: felicianosal@yahoo.com.au OR Dr. Carlos Lauchande Universidade Pedagógica, Maputo Tel: +258 828487629 Email: lauchand59@gmail.com Bioethics Committee: Ministério da Saúde - Comissão Nacional de Bioética para Saúde Maputo Tel.: +258 824066350 A11. Consent given Yes…1 No…..2 >> end interview SECTION B: Respondent details Question Code Response 94 B01 Is the person you are interviewing the deputy school director? 1- Yes >> B04 2- No B02a Who is the person you are interviewing? 1 – School director >> B04 2 – Grade 4 teacher >> B04 43 – Grade 7 teacher >> B04 54 – Other teacher 65 – Other person at the school B02b What is the position of the person you are interviewing? B03 Name of respondent Name B04 Respondent’s primary telephone number Number B05 Respondent’s secondary telephone number Number SECTION C: Characteristics of deputy school directors No. Question Response code Response C01 Gender 1- M 2- F C02 What is your age? (completed years) YEARS C03a What is your highest level of education completed? 1- Primary (Grade 5) 2- Upper Primary (Grade 7) 3- Secondary (Grade 10) 4- Upper secondary (Grade 12) 5- Undergraduate degree 6- Graduation (Completion of Undergraduate degree final thesis) 7- Graduate degree 8- Ph.D 9- Other qualification (outside education) C03b What level of teacher training have you completed? 1 – Teacher Training Center 2 – Teacher Training Institute 3 – Higher Education Teacher Training 4 – None 5 – Other, specify C04a Have you ever received training on early grade literacy? 1- Yes 2- No C04b Are you the main teacher for a class in this school? 1- Yes 2- No >>C05a C04c Which grade(s) do you teach? Check all that apply: 1_ 2_ 3_ 4_ 5_ 6_ 7_ C05a How long have you been working full time at this school? YEARS (enter 0 if less than one year) C05b How long have you been working as a teacher at any school? YEARS (enter 0 if less than one year) C06a Does the deputy school director [or you if B01=1] attend classes to observe teachers’ teaching? 1 – Yes 2 – No >> Section D C06b How many classes does the deputy school director [or you if B01=1] observe per week? Number C06c Does the deputy school director [or you if B01=1] provide feedback to teachers on what they are doing well and not so well? 1 – Yes 2 – No C06d What does the deputy school director [or you if B01=1] provide advice about? 1- Where pupils sit 2- What they should write on the blackboard 3- How they should manage pupils learning at different levels 4- What teaching methodologies they should apply 95 No. Question Response code Response 5- Supporting teachers in teaching early grade literacy SECTION D: School facilities Question Code Response D01a Does the school have a functioning latrine for the pupils within the school premises? 1 – Yes 2 – No >> D02 D01b Is there a physical separation/demarcation for the girls’ latrine? 1 – Yes 2 – No D02 What is the school’s main source of water? 1- Piped 2- Tubewell 3- Well 4- Rainwater 5- River 6- Other, specify D03 Does the school have electricity? 1- Yes 2- No D04a Does the school have reading materials in Portuguese that grade 4 pupils can use? 1- Yes 2- No D04b Does the school have reading materials in local language that grade 4 pupils can use? 1- Yes 2- No D04c Who provided the reading materials? [if D04a or D04b = 1] a. Parents b. The school c. School director(s) or teachers d. World Vision/USDA e. Save the Children f. Farmer groups g. A different NGO h. School council i. Community leaders j. Church k. Don’t know D05 Which grades have reading materials they can use? [if D04a or D04b = 1] Check all that apply: 1_ 2_ 3_ 4_ 5_ 6_ 7_ None __ D06a Does this school operate in shifts? 1- Yes 2- No >>D07a D06b How many shifts? 1 – One 2 – Two 3 – Three D06c What is the start and end time of the first shift? Start ________ End _______ D06d What is the start and end time of the second shift? [skip to D07a if D06b = 1] Start ________ End _______ D06e What is the start and end time of the third shift? [skip to C07a if C06b = 2] Start ________ End _______ D07a Does the school have a school council? 1- Yes 2- No >>D08a D07b How often does the school council meet? 1- Daily 2- Weekly 3- Monthly 4- Less than once a month 5- Never D07c When did the last school council meeting take place? 1 – last week 2 – last month 3 – last year D07d How many members are there on the school council? Number D07e Are parents of children in this school part of the school council? 1- Yes 2- No D08a How often does your ZIP meet? 1- Once a week 2- Once a month 3- Once a term 4- Not structured 5- Other, specify 96 Question Code Response D08b How often does the deputy school director attend the ZIP meetings? 1- Once a week 2- Once a month 3- Once a term 4- Not structured 5- Other, specify D08c How often do teachers attend the ZIP meetings? 6- Once a week 7- Once a month 8- Once a term 9- Not structured 10- Other, specify 11- Don’t know D08d What topics are discussed in ZIP meetings? (check all that apply) 1 – Teaching methods 2 – Trainings people have participated in 3 – Upcoming trainings 4 – Sharing of teaching and learning materials 5 – Discussing problems at school 6 – Finding support for problems at school (financial or other resources) 7 – Other, specify D09 Has this school received two or more visits from a health facility staff member this or the previous school year? 1- Yes 2- No D10 How many pupils in this school received a deworming treatment this school year or last school year? Number D11a This school year or the previous school year did the school collect any fees/money from grade 4 pupils for any purpose? 1- Yes 2- No>> D12a D11b For what purpose(s)? Check all that apply. [Enumerator: read list] a. Annual fee b. Monthly fee c. Exam fee d. PTA fee e. Development fee f. Textbook fee g. Uniform fee h. Salary of school cleaners or school guards i. Other fee, specify D11c What is the total amount per term? Meticals D12a Does the school have soap (or other detergent) available to staff? 1 – yes 2 – no D12b Does the school have soap (or other detergent) available to students? 1 – yes 2 – no SECTION E: Previous programs (school meals and Literacy Boost) Question Code Response E01a Are school meals provided at the school? 1 – Never >> E05 2 – Occasionally 3 – Weekly 4 – Daily E01b Who pays for the school meals? (check all that apply) a. Children/parents b. The school c. School director(s) or teachers d. World Vision/USDA e. Save the Children f. Farmer groups g. A different NGO 97 Question Code Response h. School council i. Community leaders j. Church k. Other, specify E01c For how long has this school been receiving school meals? Years E02a How many KGs of food have farmer groups provided to the school during this school year? KGs. Skip if E01b is not e E02b What foods have the farmer groups provided? Skip if E01b is not e E03 What grades receive the school meals? Check all that apply: KG_ 1_ 2_ 3_ 4_ 5_ 6_ 7_ E04 Are the school meals fortified with CSB+? 1- Yes 2- No E05 Is there a hand washing station or tippy tap available to pupils to wash their hands before and/or after eating or using the latrine? 1- Yes 2- No E06 Does the school have adequate facilities (own room, clean, ventilation) to store food? 1- Yes 2- No E07 What is the name of the main cook at this school? Enumerator: this is the person you should interview for the school cook survey, unless they are not available, in which case survey any cook / server at the school. Name E08a Do pupils get take home rations? 1- Yes 2- No>> E09a E08b In which grades do pupils get take home rations? Check all that apply: KG_ 1_ 2_ 3_ 4_ 5_ 6_ 7_ E08c Who pays for these take home rations? Check all that apply. a. Children/parents b. The school c. School director(s) or teachers d. World Vision/USDA e. Save the Children f. Farmer groups g. A different NGO h. School council i. Community leaders j. Church E09a Did your school participate in Literacy Boost, implemented by Save the Children? 1- Yes 2- No >> E10 E09b What was provided through the Literacy Boost program? Check all that apply. 1- Training on early grade literacy teaching 2- Training on other topics 3- Reading materials in Portuguese 4- Reading materials in local language 5- Teaching materials for teachers 6- Learning materials for pupils 7- Establishment of reading clubs in the community 8- Other, specify E09c For how many years did Literacy Boost operate in your school? Years 98 Question Code Response E10 Have students in this school ever participated in reading clubs? 1 – No 2 – Rarely 3 – Sometimes 4 – Often 5 - Always E11 What other interventions is the school receiving or what other programs does the school participate in? 1- None 2- WASH training 3- WASH clubs 4- Involvement in community report cards 5- School report cards (World Vision Program) 6- Deworming (World Vision Program) 7- Other, specify Please ask the respondent to get the enrollment logs for 2020 and 2021. SECTION F: Enrolment, repetition, and dropout No. Question Response code Response F01 How many pupils are currently enrolled in this school (in all classes)? Male _________ Female _______ F02 How many grade 4 pupils are currently enrolled in this school? Male _________ Female _______ F03 What proportion of pupils in this school miss school for more than 10 days in a year due to illness? Percent F04 How many pupils were enrolled in school at the beginning of the school year in 2020 before the Covid-19 school closures? KG Male __ Female __ Grade 1 Male __ Female __ Grade 2 Male __ Female __ Grade 3 Male __ Female __ Grade 4 Male __ Female __ Grade 5 Male __ Female __ Grade 6 Male __ Female __ Grade 7 Male __ Female __ F05 How many pupils returned to school after the reopening of schools? KG Male __ Female __ Grade 1 Male __ Female __ Grade 2 Male __ Female __ Grade 3 Male __ Female __ Grade 4 Male __ Female __ Grade 5 Male __ Female __ Grade 6 Male __ Female __ Grade 7 Male __ Female __ F06 What were the top 3 reasons that pupils did not return to school? Check in order 1 – Had to work 2 – Family could not afford 3 – Marriage 4 – Pregnancy 5 – Parent passed away 6 – Pupil passed way 7 – Pupil sick with COVID-19 8 – Other, specify F07a Were any measures taken before schools closed to ensure continued learning of pupils during the closures? 1- Yes 2- No >> F8a F07b What were those measures? Check all that apply. 1- Gave pupils materials to take home 2- Encouraged pupils to listen to radio education programming 3- Encouraged pupils to watch TV education programming 4- Phoned parents to reassure them 99 No. Question Response code Response 5- Other, specify F08a Were any measures taken during the school closures to ensure continued learning of pupils during this period? 1- Yes 2- No >> F9 F08b What were those measures? Check all that apply. 1- Gave pupils materials to take home 2- Encouraged pupils to listen to radio education programming 3- Encouraged pupils to watch TV education programming 4- Phoned parents to encourage them to help their child learn at home 5- Visited pupils at home 6- Gathered small groups of pupils for lessons 7- Other, specify F09 What measures were in place in your school to open slowly and safely? Check all that apply. 1- Only allow grade 7 pupils at first 2- Only allow grade 3 pupils at first 3- Space desks/seating further apart 4- Ensure pupils and staff wash their hands frequently 5- Require staff to wear masks or other PPE 6- Hold classes outside when possible 7- Continue to keep libraries and other common spaces closed 8- Provide information to pupils and parents 9- Opened some classes before others 10- Had different students attend on different days 11- Other, specify F10a Now that schools have reopened, are you having students attend school in shifts? 1- Yes 2 - No F10b How are the shifts organized? 1 – alternating days 2 - Alternating weeks 3 - Other, specify F10c How are students divided into shifts? 1- Alphabetical order 2- Order listed on the school register 3- By ability 4- Other, specify SECTION G: TEACHERS Enumerator: ask for the teacher register and record the grade 4 teacher’s attendance for the last 5 days (only school days – exclude weekends and public holidays). If there is no record of teacher attendance, ask the school director or deputy school director to respond and explain why. G00. Is there more than one grade 4 teacher in this school? 1- Yes 2- No. >>G02a Response: ______ G01. What is the name of the main grade 4 teacher for reading? Name: _____________________________ Enumerator: if there is more than one grade 4 teacher but none are the main grade 4 teacher for reading, choose one grade 4 teacher at random. G02a. Is there a record of attendance for teachers in the school? 1- Yes 2- No. Response: _____ 100 G02b. Is there a record of attendance for the grade 4 teacher listed in G01? 1- Yes 2- No. Response: _____ Attendance of teacher listed in G01: No. Day Date Response G03a Today CODE 2 G03b 1 school day ago CODE 2 G03c 2 school days ago CODE 2 G03d 3 school days ago CODE 2 G03e 4 school days ago CODE 2 No. Question Response code Response G04 How many permanent teachers are there in the school? Number G05 How many temporary / volunteer teachers are there in the school? Number G06 How many permanent teachers have attended school 80 per cent or more of the time in the past 30 days? Days G07 How many temporary / volunteer teachers have attended school 80 per cent or more of the time in the past 30 days? Days G08 How many teachers have received training on early grade literacy? Number CODE 2 (Section G): 1 – Present 2 – Absent (sanctioned – teacher had permission from school director or other) 3 – Absent (unsanctioned – teacher did not have permission) 4 – Don’t know 101 SECTION H: Depression (PHQ-A) Enumerator: Ask the respondent: “Over the last 2 weeks, how often have you been bothered by any of the following problems?” and read all the response options. Then they should answer the question. At the end of this section, please provide the contact information of the resource for mental health in Nampula/Mozambique to all respondents. No. Question Response code Response H01 Little interest or pleasure in doing things 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H02 Feeling down, depressed, or hopeless 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H03 Trouble falling or staying asleep, or sleeping too much 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H04 Feeling tired or having little energy 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H05 Poor appetite or overeating 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H06 Feeling bad about yourself—or that you are a failure or have let yourself or your family down 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H07 Trouble concentrating on things, such as schoolwork, reading, or watching television 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H08 Moving or speaking so slowly that other people could have noticed? Or the opposite—being so fidgety or restless that you have been moving around a lot more than usual 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H09 Thoughts that you would be better off dead or of hurting yourself in some way 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day H10 If you checked off any problems, how difficult have these problems made it for you to do your work, take care of things at home, or get along with other people? 1 – Not difficult at all 2 – Somewhat difficult 3 – Very difficult 4 – Extremely difficult G11 Do you think these feelings and experiences were worse during the school closures compared to before the closures? 1 – Much worse 2 – Somewhat worse 3 – Same as before 4 – Somewhat better 5 – A lot better SECTION I: Anxiety (GAD-7) Over the last two weeks, how often have you been bothered by the following problems? No. Question Response code Response I01 Feeling nervous, anxious, or on edge 1 – Not at all 2 – Several days 3 – More than half the days 102 4 – Nearly every day I02 Not being able to stop or control worrying 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day I03 Worrying too much about different things 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day I04 Trouble relaxing 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day I05 Being so restless that it's hard to sit still 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day I06 Becoming easily annoyed or irritable 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day I07 Feeling afraid as if something awful might happen 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day SECTION J: Fear of Covid-19 and Trust No. Question Response code Response J01 I am afraid of becoming infected with Covid-19. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree J02 I am worried that friends or family will be infected. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree J03 I am taking precautions to prevent infection (e.g., washing hands, avoiding contact with people, avoiding door handles). 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree J04 I feel that schools are not doing enough to deal with the virus. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree Enumerator, thank the respondent! 103 IFPRI PRIMARY SCHOOL SURVEY: GRADE 4 TEACHER QUESTIONNAIRE DASHBOARD: A01 School ID Enter school ID from school sample list A02 School Name Prefilled A03a District Prefilled A03b Administrative Post Prefilled A04 Locality Prefilled A05 Village Prefilled A06 ZIP number to which school belongs Prefilled A07 Enumerator ID Enter enumerator ID A08 Date and time stamp Automatically generated A09 Grade 4 teacher’s ID Automatically generated A10 Teacher name A11 Primary phone number A12 Secondary phone number Informed Consent Good Morning/Good Afternoon. My name is (NAME), I work for a team of researchers in the United States and in Mozambique. The research team is collecting data for an education research and we would like to invite you to participate in this survey. To help you decide if you want to accept to participate, I will give you more explanation about what we are trying to do. If in doubt, you can ask for clarification at any time. If you need to, you can ask for time to reflect or consult someone you trust. Why is this research being carried out? This research is being carried out to gain an understanding of how education works in Mozambique. Many other schools will be participating in the same research study. The research will investigate how to improve education programming in Mozambique. Research Objectives The objective is to analyze the effectiveness of education programming implemented by World Vision in some districts in Nampula province. Type of Research/ Intervention The data collected during this research will be used in a statistical model that will allow measuring the effect that the educational programs implemented by World Vision have on students’ literacy performance. Selection of participants This survey will include many schools from three districts in Nampula province. They have been selected from a group of schools just like yours. Voluntary Participation Your participation is voluntary. You are not required to participate in this research. If you decide not to participate, there will be no harm to you. If you decide to participate, you can interrupt at any time without prejudice. There are no sanctions or consequences if you decide you don't want to participate. You can also choose not to answer any specific question in the interview, and you can also ask us not to use your information at any time. Risks, Discomfort, and Inconvenience There will be very little risk to you from the study. Your studies will not be impacted. The interview will take approximately 30 minutes. As you can see, we are taking considerable precautions regarding Covid-19. All interviews will take place outside, the enumerator will be 6 feet away from the respondent or will have a clear plastic barrier in front of them, the 104 enumerator will always wear a mask and will sanitize his/her hands and equipment before and after each interview. Enumerators have their temperatures checked every morning and nobody who is ill is allowed to come to the school. Benefits We cannot promise any benefits to you or others who participate in this survey. Your participation in the study is very important, we hope that your participation in the study can help us to improve education in Mozambique. Cost of participation/ Compensation You will receive a token of our appreciation for participating in this study and there will be no cost to you for participating in this study. Privacy Your name, as well as any other information that can be used to identify you, will not be shared with anyone including the school or the government. No one, except one researcher, will be able to access the information and see any answers. All information will be stored in an encrypted, password-protected folder that only the lead researcher will have access to. Confidentiality All information you give to the interviewer will be kept confidential. We will never report on individual information, only describe general standards and the conclusions resulting from the analyses of the information provided by all participants. No data identified by individual participants will be published for third parties who are not project personnel, and no data collected during the study is considered to be sensitive in nature. All information that could be used to identify the participant will be treated, protected, and accessed only by the team authorized for research. Sharing Results The results of this study will result in recommendations and the elaboration of education policies, which will be shared with the Ministry of Education and Human Development of Mozambique, international institutions, and education practitioners in Mozambique and outside Mozambique. The results of the research will also be published in conferences, seminars, workshops, and scientific publications. Whom to Contact (Researchers and Committee of Ethics) Researchers: In case you would like to speak with someone about the research, please contact: Dr. Feliciano Salvador Chimbutane Universidade Eduardo Mondlane, Maputo Tel: +258828173490 Email: felicianosal@yahoo.com.au OR Dr. Carlos Lauchande Universidade Pedagógica, Maputo Tel: +258 828487629 Email: lauchand59@gmail.com Bioethics Committee: Ministério da Saúde - Comissão Nacional de Bioética para Saúde Maputo Tel.: +258 824066350 A13. Consent given Yes…1 No…..2 >> end interview SECTION B: Teacher and classroom characteristics 105 No. Question Response code Response B01 Are you a permanent or contract / volunteer teacher? 1- Permanent 2- Temporary / volunteer B02 Gender 1- Male 2- Female B03 What is your age? (completed years) Number B04 What language do you speak at home? (check all that apply) 1 – Emakhuwa 2 – Portuguese 3 – English 4 – Other, specify B05a What is your highest level of education completed? 10- Primary (Grade 5) 11- Upper Primary (Grade 7) 12- Secondary (Grade 10) 13- Upper secondary (Grade 12) 14- Undergraduate degree 15- Graduation (Completion of Undergraduate degree final thesis) 16- Masters 17- Ph.D 18- Other qualification (outside education), specify B05b What level of teacher training have you completed? (check all that apply) 1 – Teacher Training Center 2 – Teacher Training Institute 3 – Higher Education Teacher Training 4 – None 5 – Other, specify B06a How long have you been working full time at this school? YEARS (enter 0 if less than one year) B06c How long have you been working as a teacher at any school? YEARS (enter 0 if less than one year) B07 Which classes and subjects do you teach this school year on a regular basis at this school? Check all that apply. Subject Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 a b c d e f g i Math ii Portuguese Iii Emakhuwa Iv Social studies v Natural Science vi Visual Education & Crafts vii Physical Education viii Other subject(s), specify B08a What is the primary language of instruction in your grade 4 reading class? 1- Portuguese 2- Emakhuwa 3- English 4- Other, specify B08b What is the secondary language of instruction in your grade 4 reading class? 1- Portuguese 2- Emakhuwa 3- English 4- Other, specify 5- No secondary language B09a Do your grade 4 pupils have access to reading materials in Portuguese? 1- Yes, one per pupil 2- Yes, some pupils share 3- None have textbooks B09b Do your grade 4 pupils have access to reading materials in Emakhuwa? 1- Yes, one per pupil 2- Yes, some pupils share 3- None have reading materials B10 What teaching practices do you use in the grade 4 classroom to teach reading? Check all that apply. 1- Splitting pupils into groups to study with the textbook 2- Splitting pupils into groups to work with other learning materials 106 No. Question Response code Response 3- Split pupils into groups according to their ability levels 4- Lecture 5- Call and response 6- Other: Specify B11 Do you have access to a copy of the book(s) that outline the reading curriculum for grade 4? 1- Yes 2- No B12a Have you received teaching and learning materials or reading materials? 1- Yes 2- No >>B13 B12b Who provided these materials? Check all that apply. 1- Save the Children 2- World Vision 3- Other NGO, specify 4- Government 5- Private sector firm 6- Other: Specify B12c In what language are these materials? Check all that apply. 1- Portuguese 2- Emakhuwa 3- English 4- Other, specify B13 How often do you attend ZIP meetings? 1- Never 2- Rarely 3- Half the time 4- Often 5- All the time B14 What topics are discussed in ZIP meetings? (check all that apply) 1 – Teaching methods 2 – Trainings people have participated in 3 – Upcoming trainings 4 – Sharing of teaching and learning materials 5 – Discussing problems at school 6 – Finding support for problems at school (financial or other resources) 7 - Other, specify SECTION C: Programs No. Question Response code Response C01 Have you heard of Literacy Boost implemented by Save the Children? 1- Yes 2- No >>D01a C02 Did you receive training through this program? 1- Yes 2- No>>D01a C03 When did you participate in the training? Month ____ Year ____ C04 What did you learn at the training? Check all that apply. 1- How to teach early grade reading 2- Good pedagogical practices 3- How to use teaching and learning materials 4- How to produce teaching and learning materials using low-cost resources available locally 5- How to organize a literacy reach classroom environment 6- Other, specify SECTION D: Covid-19 No. Question Response code Response D01a How many pupils were enrolled in the grade 2 class in 2020 before the Covid-19 school closures? Male __ Female __ 107 No. Question Response code Response D01b How many of those pupils returned to school after the reopening of schools this year? Male __ Female __ D01c What were the top 3 reasons that pupils did not return to school? Check in order 1 – Had to work 2 – Family could not afford 3 – Marriage 4 – Pregnancy 5 – Family feared about their health 6 – Parent/family member died due to COVID-19 7 – Parent/family member was sick due to COVID-19 8 – Pupil was sick due to COVID-19 9 – Pupil died of COVID-19 10 - Other: Specify D02a Were any measures taken before schools closed to ensure continued learning of pupils during the closures? 3- Yes 4- No >> D03a D02b What were those measures? Check all that apply. 6- Gave pupils materials to take home 7- Encouraged pupils to listen to radio education programming 8- Encouraged pupils to watch TV education programming 9- Phoned parents to reassure them 10- Other, specify D03a Were any measures taken during the school closures to ensure continued learning of pupils during this period? 3- Yes 4- No >> D04a D03b What were those measures? Check all that apply. 8- Gave pupils materials to take home 9- Encouraged pupils to listen to radio education programming 10- Encouraged pupils to watch TV education programming 11- Phoned parents to encourage them to help their child learn at home 12- Visited pupils at home 13- Gathered small groups of pupils for lessons 14- Other, specify D04a What measures were in place in your school to open slowly and safely? Check all that apply. 12- Only allow grade 7 pupils at first 13- Only allow grade 3 pupils at first 14- Space desks/seating further apart 15- Ensure pupils and staff wash their hands frequently 16- Require staff to wear masks or other PPE 17- Hold classes outside when possible 18- Continue to keep libraries and other common spaces closed 19- Provide information to pupils and parents 20- None 21- Other, specify D04b Does your school have the infrastructure and supplies it needs to practice COVID-19 and other disease prevention protocols? 1 – Yes >>D05a 2 - No D04c What is missing? 1 – Masks for students 2 – Masks for educators 3 – Handwashing station 4 – Ventilated classrooms 5 – Large enough spaces for students to physically distance when learning 108 No. Question Response code Response 6 – Other, specify D05a Did you receive any support regarding how to manage Covid-19 in your school? 1- Yes 2- No >> D06a D05b Who provided that support? Check all that apply. 1- School director 2- District education officer 3- Provincial education officer 4- MINEDH education officer 5- ZIP Coordinator 6- Other, specify D05c What support was provided? Check all that apply. 1- How to help children continue to learn during school closures 2- How to help bring children back into school after reopening 3- Other: Specify D06a Has the pandemic affected you personally in any way? 1- Yes 2- No>>D07 D06b In what way(s) has the pandemic affected you? Check all that apply. 1- Family member passed away 2- Family member caught Covid 3- Family member lost their job 4- Some of my income was lost because I could not do my second job 5- It has been difficult to manage the children since schools have reopened. 6- Other: Specify D07 What were pupils primarily doing while schools were closed? 1- Studying 2- Nothing in particular 3- Working 4- Other: Specify 5- Don’t know SECTION E: Depression (PHQ-A) Enumerator: Ask the respondent: “Over the last 2 weeks, how often have you been bothered by any of the following problems?” and read all the response options. Then they should answer the question. At the end of this section, please provide the contact information of the resource for mental health in Nampula/Mozambique to all respondents. No. Question Response code Response E01 Little interest or pleasure in doing things 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E02 Feeling down, depressed, or hopeless 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E03 Trouble falling or staying asleep, or sleeping too much 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E04 Feeling tired or having little energy 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day 109 No. Question Response code Response E05 Poor appetite or overeating 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E06 Feeling bad about yourself—or that you are a failure or have let yourself or your family down 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E07 Trouble concentrating on things, such as schoolwork, reading, or watching television 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E08 Moving or speaking so slowly that other people could have noticed? Or the opposite—being so fidgety or restless that you have been moving around a lot more than usual 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E09 Thoughts that you would be better off dead or of hurting yourself in some way 0 – Not at all 1 – Several days 2 – More than half the days 3 – Nearly every day E10 If you checked off any problems, how difficult have these problems made it for you to do your work, take care of things at home, or get along with other people? 1 – Not difficult at all 2 – Somewhat difficult 3 – Very difficult 4 – Extremely difficult E11 Do you think these feelings and experiences were worse during the school closures compared to before the closures? 1 – Much worse 2 – Somewhat worse 3 – Same as before 4 – Somewhat better 5 – A lot better SECTION F: Anxiety (GAD-7) Over the last two weeks, how often have you been bothered by the following problems? No. Question Response code Response F01 Feeling nervous, anxious, or on edge 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day F02 Not being able to stop or control worrying 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day F03 Worrying too much about different things 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day F04 Trouble relaxing 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day F05 Being so restless that it's hard to sit still 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day F06 Becoming easily annoyed or irritable 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day 110 No. Question Response code Response F07 Feeling afraid as if something awful might happen 1 – Not at all 2 – Several days 3 – More than half the days 4 – Nearly every day SECTION G: Fear of Covid-19 and Trust No. Question Response code Response G01 I am afraid of becoming infected with Covid-19. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree G02 I am worried that friends or family will be infected. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree G03 I am taking precautions to prevent infection (e.g., washing hands, avoiding contact with people, avoiding door handles). 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree G04 I feel that schools are not doing enough to deal with the virus. 1 – Strongly agree 2 – Agree 3 – Neither agree nor disagree 4 – Disagree 5 – Strongly disagree Enumerator, thank the respondent! 111 IFPRI PRIMARY SCHOOL SURVEY: SCHOOL COOK QUESTIONNAIRE DASHBOARD: A01 School ID Enter school ID from school sample list A02 School Name Prefilled A03a District Prefilled A03b Administrative Post Prefilled A04 Locality Prefilled A05 Village Prefilled A06 ZIP number to which school belongs Prefilled A07 Enumerator ID Enter enumerator ID A08 Date and time stamp Automatic A09 School cook ID Automatically generated A10 School cook’s name A11 Primary phone number A12 Secondary phone number Informed Consent Good Morning/Good Afternoon. My name is (NAME), I work for a team of researchers in the United States and in Mozambique. The research team is collecting data for education research and we would like to invite you to participate in this survey. To help you decide if you want to accept to participate, I will give you more explanation about what we are trying to do. If in doubt, you can ask for clarification at any time. If you need to, you can ask for time to reflect or consult someone you trust. Why is this research being carried out? This research is being carried out to gain an understanding of how education works in Mozambique. Many other schools will be participating in the same research study. The research will investigate how to improve education programming in Mozambique. Research Objectives The objective is to analyze the effectiveness of education programming implemented by World Vision in some districts in Nampula province. Type of Research/ Intervention The data collected during this research will be used in a statistical model that will allow measuring the effect that the educational programs implemented by World Vision have on the student’s literacy performance. Selection of participants This survey will include many schools from three districts in Nampula province. They have been selected from a group of schools just like yours. Voluntary Participation Your participation is voluntary. You are not required to participate in this research. If you decide not to participate, there will be no harm to you. If you decide to participate, you can interrupt at any time without prejudice. There are no sanctions or consequences if you decide you don't want to participate. You can also choose not to answer any specific question in the interview, and you can also ask us not to use your information at any time. Risks, Discomfort, and Inconvenience There will be very little risk to you from the study. Your studies will not be impacted. The interview will take approximately 30 minutes. As you can see, we are taking considerable precautions regarding Covid-19. All interviews will take place outside, the enumerator will be 6 feet away from the respondent or will have a clear plastic barrier in front of them, the 112 enumerator will always wear a mask and will sanitize his/her hands and equipment before and after each interview. Enumerators have their temperatures checked every morning and nobody who is ill is allowed to come to the school. Benefits We cannot promise any benefits to you or others who participate in this survey. Your participation in the study is very important, we hope that your participation in the study can help us to improve education in Mozambique. Cost of participation/ Compensation You will receive a token of our appreciation for participating in this study and there will be no cost to you for participating in this study. Privacy Your name, as well as any other information that can be used to identify you, will not be shared with anyone including the school or the government. No one, except one researcher, will be able to access the information and see any answers. All information will be stored in an encrypted, password-protected folder that only the lead researcher will have access to. Confidentiality All information you give to the interviewer will be kept confidential. We will never report on individual information, only describe general standards and the conclusions resulting from the analyses of the information provided by all participants. No data identified by individual participants will be published for third parties who are not project personnel, and no data collected during the study is considered to be sensitive in nature. All information that could be used to identify the participant will be treated, protected, and accessed only by the team authorized for research. Sharing Results The results of this study will result in recommendations and the elaboration of education policies, which will be shared with the Ministry of Education and Human Development of Mozambique, international institutions, and education practitioners in Mozambique and outside Mozambique. The results of the research will also be published in conferences, seminars, workshops, and scientific publications. Whom to Contact (Researchers and Committee of Ethics) Researchers: In case you would like to speak with someone about the research, please contact: Dr. Feliciano Salvador Chimbutane Universidade Eduardo Mondlane, Maputo Tel: +258828173490 Email: felicianosal@yahoo.com.au OR Dr. Carlos Lauchande Universidade Pedagógica, Maputo Tel: +258 828487629 Email: lauchand59@gmail.com Bioethics Committee: Ministério da Saúde - Comissão Nacional de Bioética para Saúde Maputo Tel.: +258 824066350 A13. Consent given Yes…1 No…..2 >> end interview SECTION B: School lunch program 113 No. Question Response code Response B00 How long have you been the cook at this school? years B01 Gender 1- M 2- F B02 What is your age? (completed years) Number B03 How many children eat the school meals you provide on an average school day? Number B04 How many hours do you devote to the preparation of school meals on an average school day? B05 In your opinion, how important are the school meals in encouraging children to attend school and participate actively on a scale of 1 to 4, where 1 is relatively unimportant, and 4 is highly important? 1 ____ 2 ____ 3 ____ 4 ____ B06 Compared to what pupils usually eat at home, is the food provided at school: 1 – More nutritious 2 – Same level of nutritious 3 – less nutritious B07 Do you have any guidelines for providing school meals? 1 – Yes 2- No B08 Do you generally have enough ingredients to prepare the school meals? 1 – Always 2 – Most of the time 3 – Some of the time 4 – Not very often 5 – Never B09 On a typical day, what does the school lunch consist of? 1- Porridge 2- Nutrient fortified porridge 3- Other: Specify B10 Have the following foods ever been included in school meals? Check all that apply. Refer to annex for a list of foods that belong to each group 1- Cereals 2- Vegetables and tubers rich in Vitamin A 3- White tubers and roots 4- Dark green leafy vegetables 5- Other vegetables 6- Fruits rich in Vitamin A 7- Other Fruits 8- Organ meat 9- Flesh meat 10- Eggs 11- Fish 12- Others B11 Please list some things that enable the safe preparation of food? Check all that apply 1- Washing the pot before cooking 2- Washing the pot after cooking 3- Washing hands before cooking 4- Washing hands before serving 5- Storing the food in a clean place before cooking 6- Using a clean spoon for stirring 7- Using a clean spoon for serving 8- Covering food with a lid B12 How did COVID-19 affect school meals now that schools have reopened? Check all that apply 1 – It did not affect meals 2 – Had to stop providing school meals 3 – There is now less availability of food 4 – There is now less funding for school meals 5 – Other, specify 114 Annex FOOD GROUP FOODS CEREALS Corn, rice, wheat, soy, maize or any other grains or foods made from these (i.e. bread, rice, spaghetti, porridge or other grain products) + (a) Potatoes, china (african food), bread, rice, spaghetti, crackers, or other foods made out of cereals VEGETABLES AND TUBERS RICH IN VITAMIN A Pumpkin, carrots, pumpkin or orange/yellow sweet potato, other foods available locally with vitamin A rich in vegetables (i.e. red pepper) WHITE TUBERS AND ROOTS Rhine potatoes, white yams, white cassava, or other foods made from tubers DARK GREEN LEAFY VEGETABLES Dark green leafy vegetables, including wild berries + leaves rich in Vitamin A, available locally, cassava leaves, kale, gren beans, spinach, tseke (nhewe) (african food) OTHER VEGETABLES Other vegetables (i.e. tomatoes, onions, eggplant), including other wild vegetables FRUITS RICH IN VITAMIN A Ripe mangoes, ripe papaya, red-flesh guava, peach + other fruits available locally, rich in vitamin A OTHER FRUITS Other fruits, including wild fruits (banana, apple, mandarin orange, grapefruit, grapes, massanica (similar to cherries), rava (african fruit)) ORGAN MEAT (RICH IN IRON) Liver, kidney, heart or other organs or blood-based foods FLESH MEAT Pork, sheep, cow, goat, rabbit, chicken, duck or other birds EGGS Chicken, duck, or from any other bird FISH Fresh or dried fish, and seafoods Others Some food made with a type of beans, peas, lentils, almonds or seeds Enumerator, thank the respondent! 115 Note that the Grade 4 pupil questionnaire is only available in Portuguese. INQUÉRITO DO IFPRI SOBRE A ESCOLA PRIMÁRIA: QUESTIONÁRIO AOS ALUNOS DA 4a CLASSE DADOS GERAIS: A01 Número de Identificação da Escola Introduza o Número de Identificação da Escola, conforme a lista de amostras da escola A02 Nome da Escola Pré-preenchido A03a Distrito Pré-preenchido A03b Posto administrativo Pré-preenchido A04 Localidade Pré-preenchido A05 Cidade ou Vila Pré-preenchido A06 ZIP da escola Pré-preenchido A07 Número de Identificação do(a) Inquiridor(a) Introduza o Número de Identificação do(a) Inquiridor(a) A08 Indicação da Data e Hora Gerado automaticamente A09 Número de Identificação do(a) Aluno(a) Selecione o número de identificação do(a) aluno(a) na caixa de selecção A10 Nome do(a) Aluno(a) Pré-preenchido Inquiridor(a): Leia esta declaração ao(à) aluno(a) e pergunte se ele concorda em ser entrevistado(a). Registe a resposta e a data / hora. Depois, faça as perguntas que se seguem ao(à) aluno(a), se ele / ela consentir. Consentimento informado Bom dia/ Boa tarde. O meu nome é (NOME), eu trabalho com uma equipe de pesquisadores dos Estados Unidos e de Moçambique. A equipe de pesquisa está a recolher dados para um estudo sobre literacia. Pelo que, gostaríamos de convidar-te a participar desta pesquisa. Para te ajudar a decidir se aceitas participar do estudo, darei mais explicações sobre o que se pretende fazer. Em caso de dúvida, podes solicitar esclarecimentos a qualquer momento. Se precisares, podes pedir algum tempo para reflectires ou consultares alguém da tua confiança antes de tomares a tua decisão. Conversamos com [o(a) teu/tua pai/mãe / o(a) teu/tua encarregado(a) de educação]. Ele(a) concordou que tu participasses do estudo, se assim o desejares. Contudo, só tu podes decidir se aceitas ou não participar do estudo. Por que é que este estudo está a ser feito? Com esta pesquisa pretende-se entender como é que a educação funciona em Moçambique. Muitas outras escolas participarão deste mesmo estudo. A pesquisa investigará como melhorar a programação e implementação da educação em Moçambique. Objectivos do estudo 116 O objectivo do estudo é analisar a eficácia de programas de educação implementados pela Visão Mundial em alguns distritos da província de Nampula. Tipo de Pesquisa / Intervenção Os dados a recolher ao longo desta pesquisa serão analisados a partir de um programa estatístico, que permitirá medir o efeito dos programas educacionais implementados pela Visão Mundial no desempenho dos alunos em literacia. Seleção dos participantes Esta pesquisa incluirá muitas escolas de três distritos da província de Nampula. Estas escolas foram seleccionados de um grupo de escolas como a tua. Participação Voluntária A tua participação no estudo é voluntária. Tu não és obrigado(a) a participar da pesquisa. Se tu decidires não participar do estudo, não haverá qualquer tipo de consequências. Mesmo depois de decidires participar do estudo, podes interromper a tua participação a qualquer momento sem qualquer tipo de prejuízo. Não há sanções ou consequências se tu decidires não participar do estudo. Tu podes optar por não responder a alguma pergunta específica na entrevista, e também, a qualquer momento, podes pedir-nos para não usarmos as informações que nos forneceres. Riscos, Desconfortos e Inconvenientes Não correrás riscos importantes pela participação no estudo. Os teus estudos não serão afectados. A entrevista levará aproximadamente 30 minutos. Como podes ver, estamos a tomar precauções consideráveis em relação à COVID-19. Todas as entrevistas ocorrerão ao ar livre, o(a) inquiridor(a) estará a um metro e meio de distância de ti ou haverá uma barreira plástica transparente a separar vocês os dois. O(a) inquiridor(a) usará sempre uma máscara e higienizará as suas mãos e equipamentos antes e depois de cada entrevista. Vai-se verificar a temperatura do corpo dos inquiridores todas as manhãs e ninguém que esteja doente deverá fazer-se à escola. Benefícios Não podemos prometer quaisquer benefícios a ti ou a outras pessoas que participem nesta pesquisa. A tua participação no estudo é muito importante. Esperamos que ela nos possa ajudar a contribuir para melhorar a educação em Moçambique. Custos da participação / Compensações Receberás uma pequena prenda de agradecimento por participares deste estudo. Não haverá nenhuma despesa para ti por participares deste estudo. Privacidade O teu nome e qualquer outra informação que possa ser usada para te identificar não serão partilhados com ninguém, nem com os membros da escola ou entidades do governo. Ninguém, excepto um pesquisador, 117 poderá aceder às tuas informações pessoais e ver as respostas que deres. Todas as informações serão armazenadas numa pasta criptografada e protegida por senha, à qual somente a Investigadora Principal terá acesso. Confidencialidade Toda a informação que prestares ao(à) inquiridor(a) será mantida em sigilo. Nunca reportaremos as informações que nos deres de forma individual, descreveremos apenas os padrões e tendências gerais e as conclusões decorrentes da análise da informação dada por todos os participantes. Nenhum dado identificando participantes individuais será partilhado com terceiros, que não sejam parte do pessoal do projecto. Nenhum dado a recolher no decorrer do estudo é considerado de natureza sensível. Toda a informação que tu prestares será tratada confidencialmente e será apenas acessível às pessoas autorizadas da equipe de pesquisa. Partilha de Resultados Os resultados deste estudo vão ser usados para produzir recomendações a serem consideradas na revisão e elaboração de políticas de educação. Estes resultados serão partilhados com o Ministério da Educação e Desenvolvimento Humano e com instituições internacionais e profissionais de educação em Moçambique e fora de Moçambique. Os resultados da pesquisa serão também disseminados em eventos científicos (conferências, seminários, workshops) e através de publicações científicas. Pessoas de Contacto (Investigadores e Comité de Ética) Investigadores: Caso tu queiras falar com alguém sobre esta pesquisa, favor entrar em contato com: Dr. Feliciano Salvador Chimbutane Universidade Eduardo Mondlane, Maputo Tel: +258828173490 Email: felicianosal@yahoo.com.au Ou Dr. Carlos Lauchande Universidade Pedagógica, Maputo Tel: +258 828487629 Email: lauchand59@gmail.com Comité de Bioética: 118 Ministério da Saúde - Comité Nacional de Bioética para Saúde Maputo Tel: +258 824066350 Inquiridor(a): Assinale com um “x”, se o(a) aluno(a) concorda ou não em participar no estudo. O(a) aluno(a) pode concordar acenando com a cabeça ou dizendo “sim”. Sim:____ Não: ____ A13. Assentimento dado? 1 – Sim 2 – Não >> fim da entrevista 119 SECÇÃO B: Early Grade Reading Assessment (EGRA) Vocabulário Oral A. MATERIAL NECESSÁRIO: uma folha de papel, lápis, livro Eu vou dizer alguns nomes de algumas partes do teu corpo em Português. E tu vais-me mostrar a que parte do teu corpo se refere cada nome. Por exemplo, "nariz" (e tu vais apontar o teu nariz). Outro exemplo, "os teus olhos" (e tu vais apontar os teus olhos). Bravo! Vamos lá começar? O teu braço O teu pé O teu queixo O teu joelho O teu ombro As tuas costas A tua cabeça A tua sobrancelha Leia as instruções na língua materna do(a) aluno(a): (por favor traduza a instrução para a língua materna do(a) aluno(a)) Leia as partes do corpo ao aluno(a) somente em Português. Número total de respostas correctas: /8 B. ORGANIZAR: um lápis e uma folha de papel lado a lado na frente do aluno. Estás a ver este lápis, sim? Vais colocar o lápis onde eu te disser para colocares, está bem? Vamos começar? Leia as instruções na língua materna do(a) aluno(a): (por favor traduza a instrução para a língua materna do(a) aluno(a)) Leia as frases ao aluno(a) somente em Português. Coloca o lápis: No papel Atrás de ti No chão Por baixo do papel Na tua frente Ao lado do papel Número total de respostas correctas: /6 INSTRUÇÃO PARA O(A) INQUIRIDOR(A): Vai pedir ao(a) aluno(a) para que mostre as partes do seu corpo e objectos que estão à sua volta e observar a sua compreensão dos termos espaciais. Observação: ver se a criança dá as respostas correctas: Se der uma Resposta Correcta: dê parabéns ao(a) aluno(a) por cada resposta certa. Erro: risque, usando o sinal em parênteses (/) as respostas erradas que o(a) aluno(a) der. Autocorrecção: se o(a) aluno(a) der uma resposta errada, mas corrigi-la mais tarde (autocorrecção), assinale a resposta antes considerada errada, como certa. TOTAL de respostas correctas: /14 120 Conceitos sobre materiais impressos DIGA OS PASSOS NA LÍNGUA MATERNA DO(A) ALUNO(A) Diga: Vamos fazer um jogo com este livro da escola. (Classifique os passos 3-5 se todos os movimentos são indicados num único gesto) INSTRUÇÕES: entregue o livro ao aluno(a), pegando o livro na posição vertical, com a dobra apontada ao aluno e a parte oposta virada para si. Assinale o resultado por cada passo efectuado com um “X” na caixa. 1. Diga: Mostra-me a frente do livro Correcto Incorrecto Não Responde 2. Diga: Abre o livro na página onde começa a história. Correcto Incorrecto Não Responde 3. Diga: Mostra-me onde devo começar a ler esta história Correcto Incorrecto Não Responde 4. Diga: Em que direcção se lê cada linha do livro? Correcto Incorrecto Não Responde 5. Diga: Quando eu termino de ler uma linha, onde vou para continuar? Incorrecto Não Responde 6. Diga: Vou ler algumas linhas desta história. Quero que tu apontes as palavras enquanto eu leio. (Apenas instruções em língua materna para o(a) aluno(a), mas o texto deve ser lido em Português). Leia algumas linhas completas de forma lenta mas contínua. O(a) aluno(a) deve apontar enquanto você (inquiridor(a)) lê: Tudo Correcto Maioria Correcto Maioria Incorrecto Tudo Incorrecto Não Responde 7. Diga: Mostra-me a parte inicial da história. Agora mostra-me a parte final da história. Ambos Correctos Apenas 1 Correcto AmbosIncorrectos Não Responde 8. Diga: Como tu sabes em que página estás? Agora passa para a página “8” Não Responde 9. Diga: Mostra-me uma letra e diz o nome da letra Correcto Incorrecto Não Responde 10. Diga: Agora mostra-me uma palavra e lê a palavra. Correcto Incorrecto Não Responde Correcto Incorrecto Correcto 121 Nesse cartão, estão todas as letras do abecedário. Por favor, diz-me o NOME do maior número de letras que puderes. Por exemplo, o nome desta letra é [aponte para j] “jota”. Vamos praticar: diz-me o nome da letra [aponte para n]: Se a criança responder correctamente, diga: muito bem, acertaste o nome da letra é: “ene” Se a criança não responder correctamente, diga: A resposta correcta do nome da letra é: “ene” Percebeste o que vamos fazer? Quando eu disser “começar”, por favor, diz-me o nomedas letras damelhor maneira quepuderes. Lêas letras ao longo da página, iniciando pela primeira. [Aponte para a primeira letra na linha depois do exemplo]. Conhecimento sobre o Nome das Letras Entregue à criança o cartão nº 1, de letras (abecedário), e leia as seguintes instruções: Inicie o cronómetro quando a criança ler a primeira letra. Siga as letras com a sua caneta e marque claramente com uma barra (/) a letra que for lida erradamente pelo(a) aluno(a). Quando a criança se corrigir, conte essa letra como correcta. Fique calado(a), excepto em situações como: se a criança hesitar mais de 3 segundos. Neste caso, diga o nome da letra, aponte para a próxima letra e diga “Por favor, continua”. Marque a letra que você deu como resposta incorrecta para a criança. APÓS 60 SEGUNDOS DIGA, “pára”. Marque a última letra lida com uma chaveta, na posição de fechar (]). Regra para interromper o exercício: Se a criança não fornecer nenhuma resposta certa na primeira linha, diga “Muito Obrigado”, pare o exercício, marque no quadro abaixo e passe para o próximo exercício. L I H R S p E O N T (10) I E T D A t a D E N (20) H O E M U r L G R U (30) G R B E I f m T S R (40) S T C N P A F C A E (50) T S Q A M C O T N P (60) E A E S O F h U A T (70) R G H B S i g M I L (80) L I N O E o E R P X (90) N A C D D I O J E N (100) Marque um X se a criança não deu nenhuma resposta certa na primeira linha: 1. Caso a criança leia todas as letras em menos de 60 segundos, por favor anote o número de segundos que o(a) aluno(a) levou a completar o exercício (Número desegundos): 2. Anote o número TOTAL de letras lidas durante o tempo do exercício: 3. Anote o número de letras CERTAS lidas durante o exercício: 4. Anote o número de letras ERRADAS lidas durante o exercício: 122 Leitura das Palavras Entregue à criança, o cartão nº 2, de palavras, e leia as seguintes instruções: Nesse cartão, estão algumas palavras. Por favor, lê em voz alta o maior número de palavras que puderes. O(a) aluno(a) começa a ler as palavras da lista. O(a) inquiridor(a) marca a palavra lida como correcta ou incorrecta ou não respondida com um “X”. Se o(a) aluno(a) hesitar durante a leitura, por mais de cinco segundos, ou fizer um esforço para ler a palavra por cinco segundos, o(a) inquiridor(a) deve pedir ao(à) aluno(a) para ler a próxima palavra na lista. O inquiridor nunca deve corrigir a palavra dita pelo(a) aluno(a). Além disso, o(a) inquiridor(a) nunca deve ler a palavra correctamente para o(a) aluno(a). O(a) inquiridor(a) continua a perguntar ao(a) aluno(a) sobre as palavras na lista. O(a) inquiridor(a) pára com o teste quando o(a) aluno(a) lê incorrectamente ou não responde a cinco palavras seguidas. Contudo, se o(a) aluno(a) erra três palavras e em seguida, por exemplo, lê uma palavra correcta, o(a) inquiridor(a) continua com o exercício até que ele(a) leia cinco palavras seguidas incorrectamente ou conclua o exercício. No final deve somar o número de “X” de cada coluna: Correcto Incorrecto Não Responde 1 E 2 De 3 Ter 4 Dia 5 Ele 6 Segundo 7 Depois 8 Primeiro 9 Lá 10 Anos 11 Também 12 Cada 13 Vir 14 Triste 15 Um 16 Avô 17 Bandeira 18 Saúde 19 Lembrar 20 Ela 21 Classe 22 Descrever 23 Rua 24 Atrás 25 Olhos 26 Pai 27 Nunca 28 Através 29 Entre 30 Três TOTAIS 123 Compreensão Oral do Texto Diga à criança que vai ler um texto para ela e depois vai fazer-lhe algumas perguntas sobre esse texto. Peça à criança para prestar muita atenção, para depois poder lembrar-se da história que vai ler para ela. Leia para a criança o texto que se segue em voz alta. Leia o texto pausadamente e SOMENTE UMA VEZ. CONTO 1: Para o(a) Inquiridor(a) Ler para o(a) Aluno(a) O Paulo e os seus irmãos O Paulo tem oito anos. Ele estuda na terceira classe e gosta de estudar. Ele tem dois irmãos: o Dito e a Rita. A Rita faz cinco anos no Sábado. Ela está muito contente porque a mãe vai fazer-lhe um bolo. A Rita e os irmãos gostam muito de bolos. 06 11 16 20 29 36 40 47 56 Depois da leitura, faça as seguintes perguntas de compreensão. Conceda uns 15 segundos à criança para responder a cada pergunta. # Pergunta Resposta Correcta Incorrecta 1 Quantos anos tem o Paulo? [O Paulo tem oito anos.] 2 Como se chamam os irmãos do Paulo? [Os irmãos do Paulo chamam-se Dito e Rita.] 3 Quem faz anos no Sábado? [Quem faz anos no Sábado é a Rita.] 4 Por que é que a Rita está muito contente? [A Rita está muito contente porque a mãe vai fazer-lhe um bolo.] Indique o total de respostas correctas dadas pela criança. ________/4 124 Leitura e Compreensão do Texto Entregue à criança, o cartão nº 3, Conto 2, e leia as seguintes instruções: 1. Active o cronómetro quando a criança começa a ler a primeira palavra. Se, depois de três segundos, a criança não conseguir pronunciar a primeira palavra da passagem, diga a palavra em voz alta, marque como incorrecta, e nesse instante inicia o cronómetro novamente. 2. Siga a leitura da criança na sua cópia, e marque as palavras incorrectas com uma diagonal (/). Ao fim de um minuto, assinale com uma chaveta vertical, na posição de fechar, logo após a última palavra que a criança tentou ler. ( ] ). 3. Quando a criança terminar a leitura, [Diga: Muito obrigado(a), agora vou-te fazer algumas perguntas sobre o que estiveste a ler, podes referir-te ao conto se quiseres]. 4. Se, ao fim de um minuto, o(a) aluno(a) apenas tiver lido a primeira linha, [Diga: Muito obrigado, e passe para a o texto seguinte ou para a Secção C, se estiverem a tratar o segundo texto]. Não é necessário seguir com as perguntas de compreensão visto que a criança não consegue ler. Faça as perguntas de compreensão. Se o(a) aluno(a) lê atá à linha 15 (Conto 1 e 2), faça a pergunta nº 1. Se o(a) aluno(a) lê até à linha 35 (Conto 1) ou até à linha 60 (Conto 2), faça a pergunta nº 2. Se o(a) aluno(a) lê até à linha 70 (Conto 1) ou até à linha 95 (Conto 2), faça a pergunta 3. Se o(a) aluno(a) lê até à linha 110 (Conto 1) ou até à linha 135 (Conto 2), faça a pergunta 4. 5. Quando a criança terminar com as perguntas, [Diga: Muito obrigado(a). Por favor pede ao(a) teu(tua) professor(a) que mande o(a) próximo(a) aluno para fazer este jogo]. INSTRUÇÕES PARA ANOTAR AS CLASSIFICAÇÕES DOS ALUNOS. 1. Classifique os alunos apenas quando o exame a TODOS os alunos tiver sido terminado. 2. Conte o número total de palavras lidas CORRECTAMENTE em um minuto. 3. Depois conte o número de todas as palavras lidas CORRECTAMENTE na passagem. 4. Anote o número total de respostas correctas às perguntas de compreensão. em voz alta, iniciando pela primeira palavra. Por favor, lê da esquerda para a direita (mostre à criança como fazer). [diga:] Começa por ler cada palavra. Se encontrares uma palavra que não sabes ou não reconheces, eu digo-te qual é. Por favor, lê o melhor que saibas. Percebeste o que quero que tu faças? 125 CONTO 2: PARA O(A) ALUNO(A) LER A vida em comunidade 4 Era uma vez um macaquinho que 10 andava sempre triste. Um dia, 15 o mocho encontrou-o assim triste 20 e perguntou-lhe o motivo da 25 sua tristeza. – Eu gostaria de 30 ter muitos amigos que brincassem 35 comigo – respondeu omacaquinho. O 40 que é que faço para 45 arranjar amigos? O mocho, um 50 animal sábio, deu os seguintes 55 conselhos ao macaquinho: – Temos de 60 trabalhar para mostrar aos outros 65 que temos valor. É desta 70 maneira que conquistamos o coração 75 dos outros e fazemos amigos. 80 Então, o macaquinho decidiu seguir 85 o conselho do mocho. E 90 daí em diante, todos naquela 95 floresta passaram a gostar dele: 100 todas as mamãs macacas o 105 tratavam como filho, os outros 110 macaquinhos tratavam-no como irmão e 115 todos o tratavam como amigo. 120 1. Como se sentia o macaquinho? [O(a) aluno(a) leu até à linha 15] (Resposta: triste) Correcto Incorrecto Não Responde 2. Por que o macaquinho andava triste? [O aluno(a) leu até à linha 35] (Resposta: Queria ter mais amigos) Correcto Incorrecto Não Responde 3. Para fazer amigos, que conselhos deu o mocho ao macaquinho? [O(a) aluno(a) leu até à linha 70] (Resposta: Tem de mostrar aos outros que tem valor) Correcto Incorrecto Não Responde 4. Como as mamãs macacas tratavam o macaquinho? [O(a) aluno(a) leu até à linha 110] (Resposta: como filho) Correcto Incorrecto Não Responde A. Tu já conhecias esta história? SIM NÃO NÃO SABE/NÃO RESPONDE B. Número total de palavras lidas CORRECTAMENTE em umminuto C. Número total de respostas correctas às perguntas decompreensão FIM DO EGRA 126 SECÇÃO C: Escola No. Pergunta Código de resposta Resposta C01 Na semana passada, quantos dias frequentaste a escola? Número de 1-5 C02 Fizeste alguma coisa nos últimos 12 meses para ganhares dinheiro para ti e/ou tua família? 1 – Sim 2 – Não C03 Já faltaste às aulas porque estavas a trabalhar? 1 – Sim 2 – Não C04 Faltaste às aulas por mais de 10 dias neste ano escolar porque estavas doente? 1 – Sim 2 – Não SECÇÃO D: Nutrição e fome No. Pergunta Código de resposta Resposta D01 Quantas vezes por dia costumas comer? Número D02 Quantas dessas vezes comes na escola? Número D03 Nos últimos cinco dias de aulas, quantas vezes recebeste comida na escola? Número D04 Geralmente estás com fome durante o dia na escola? 1- Sim 2- Não D05 Sentes que prestas mais atenção na aula quando recebes comida na escola? 1- Sim 2- Não D06 Recebes comida ou lanche para levares para casa? 1- Sim 2- Não D07 Quais são algumas das boas práticas de higiene que conheces? 1- Lavar as mãos depois de usar a casa de banho ou latrina 2- Lavar as mãos antes de comer 3- Lavar as mãos depois de comer 4- Lavar as frutas antes de comê-las 5- Beber água limpa / potável 6- Lavar as mãos com sabão 7- Usar a casa de banho ou a latrina para urinar ou para defecar 8- Usar sandálias ou chinelos para ir à latrina/casa de balho ou para brincar 9- Outra: especifique D08a Na escola, há infraestruturas e recursos para tu implementares estas boas práticas de higiene? 1 – Sim >> E01 2 - Não D08b O que a escola não tem? 1 – Latrinas ou casas de banho suficientes 127 No. Pergunta Código de resposta Resposta 2 – Pontos para a lavagem das mãos 3 – Sabão 4 – Água (torneira, poço, furo) SECÇÃO E: Actividades de literacia No. Pergunta Código de resposta Resposta E01 Tens livros em casa para ler? (incluindo livros ilustrados) 1- Sim 2- Não E02 Lês livros na escola? (incluindo livros ilustrados) 1- Sim 2- Não E03a Os teus pais ajudam-te a fazer os TPCs? 1- Sim>> E04a 2- Não E03b Por que não? 1- Não sabem ler 2- Não têm tempo 3- Outro: especifique E04a Actualmente, tu fazes parte de algum acampamento de leitura? 1- Sim >> Secção F 2- Não E04b Já fizeste parte de algum acampamento de leitura? 1- Sim 2- Não >> Secção F E04c O que você fazias no acampamento de leitura? 1- Cantar músicas 2- Brincar 3- Ler livros 4- Dançar 5- Competir 6- Aprender o alfabeto 7- Aprender novas palavras 8- Outro: especifique SECÇÃO F: Covid-19 No. Pergunta Código de resposta Resposta F01 O que é que tu fizeste para continuares com a tua aprendizagem durante o tempo em que a escola estava fechada por cauda da Covid-19? 1- Nada 2- Ouvi lições na rádio 3- Assisti lições na TV 4- Tive um tutor 5- Um membro da minha família ajudou-me na aprendizagem 6- Um(a) professor(a) veio à minha casa ensinar￾me 128 No. Pergunta Código de resposta Resposta 7- Estudei usando os materiais que recebi na escola 8- Aprendi no acampamento de leitura 9- Outro: especifique F02 Aconteceram algumas destas coisas em tua casa por causa do coronavírus? 1- Alguém ficou doente 2- Alguém morreu 3- Alguém perdeu o emprego 4- Alguém deixou a cidade ou migrou 5- Outro, especifique F03 Qual foi o impacto do coronavírus em tua casa? 1- Passamos fome 2- Tivemos que vender coisas para comprar comida 3- Uma das minhas irmãs casou-se/ foi lobolada 4- Uma irmã ficou grávida 5- Tive que trabalhar 6- Outro: especifique F04 Que práticas de higiene foste instruído(a) a usar como resultado do coronavírus? 1- Lavar as mãos frequentemente 2- Manter-me a 2 metros separado de outra pessoa 3- Não beijar a bíblia na igreja 4- Não apertar as mãos 5- Ficar dentro de casa o máximo tempo possível 6- Usar uma máscara/cobrir o rosto se estiver fora 7- Certificar que as salas têm ventilação 8- Outro: especifique SECÇÃO G: Dados demográficos No. Pergunta Código de resposta Resposta G01a Vives com a tua mãe? 1- Sim 2- Não G01b Vives com o teu pai? 1- Sim 2- Não G02 Com quantos irmãos vives? Número G03 Que língua falas em casa? 1- Português 2- Emakhuwa 129 No. Pergunta Código de resposta Resposta 3- Outra, especifique G04 Qual dos seguintes itens há na tua casa? a Rádio 1- Sim 2- Não b Gado ou aves domésticas 1- Sim 2- Não c Telemóvel 1- Sim 2- Não d Bicicleta 1- Sim 2- Não Inquiridor(a), agradeça ao(à) entrevistado(a)! 130 Appendix 4: Conflict of Interest Statement The authors of this report declare that they do not have any financial or other conflicts of interest.