Liberia McGovern￾Dole FY 17 Project Final Evaluation June 2022 FY 17 McGovern-Dole Liberia project implemented by Save the Children Final Evaluation Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-669-2017/011-00 Funding Year: Fiscal Year 2017 Implemented by: SAVE the Children Evaluation Authored by: AIR Balakrishnan, Uttara Downes, John Gulemetova, Michaela Heaner, Gwendolyn Majid, Farhan Panagoulias, Philomena Robertson, Laura Zaas, Daniel DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this document can be made available by contacting fas.monitoring.evaluation@usda.gov Liberia Empowerment Through Attendance, Reading, and Nutrition (LEARN) Endline Evaluation Michaela Gulemetova, Farhan Majid, Daniel Zaas, John Downes, Gwendolyn Heaner, Philomena Panagoulias, Lauren Robertson, Uttara Balakrishnan Liberia USDA McGovern-Dole International Food for Education and Child Nutrition JUNE 2022 Presented to: Save the Children 899 North Capitol St. NE, Suite 900 Washington, D.C., 20002 Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-669-2017/032-00 Funding Year: Fiscal Year 2021 Project Duration: 10/1/2017–9/30/2022 Implemented by: Save the Children Acknowledgments American Institutes for Research (AIR) thanks the U.S. Department of Agriculture (USDA) and Save the Children (SC) for their financial support. The evaluation team extends our special thanks to Clay Westrope, Dina Rakotomalala, Timothy Grimes, and Morris Gbessagee and their colleagues at SC for their support and advice. Special thanks go to the Center for Action Research and Training (CART), our data collection partner; their managing director, Kou Johnson; fieldwork managers George Thomas and Prince Williams; and 24 amazing and professional quantitative enumerators and qualitative researchers. Without the support and hard work of the enumerators in the field, the evaluation team would not have been able to collect such high-quality data. The evaluation team also acknowledges the Ministry of Education of Liberia and its County Education offices in Grand Bassa, Grand Gedeh, Rivercess, and River Gee Counties for supporting and facilitating the rollout of LEARN data collection. AIR team also thanks Uttara Balakrishnan, Senior Research Economist at AIR, and Andi Coombes, Senior Researcher at AIR, for their great technical insights and support throughout the endline evaluation. Finally, AIR thanks all the local stakeholders and beneficiaries who freely shared their perspectives and enabled the development of in-depth analysis for the report. The findings, interpretations, and conclusions expressed in this report are entirely those of the authors. They do not necessarily represent the views of AIR, SC, or USDA. i | AIR.ORG Table of Contents Executive Summary.......................................................................................................................i Project Background and Purpose ............................................................................................. i Evaluation Methodology .......................................................................................................... i Findings and Conclusions..........................................................................................................ii Evaluation Limitations............................................................................................................xii Recommendations.................................................................................................................xiii 1. Introduction ............................................................................................................................ 1 1.1 Project Context.................................................................................................................. 1 1.2 Project Description ............................................................................................................ 4 1.3 Results Framework ............................................................................................................ 6 1.4 Purpose of the Evaluation ................................................................................................. 7 2. Evaluation Approach............................................................................................................... 8 2.1 Evaluation Questions......................................................................................................... 8 2.2 Evaluation Design .............................................................................................................. 8 2.3 Sampling Methods............................................................................................................. 9 2.4 Data Collection Methods................................................................................................. 16 2.5 Data Analysis Methods.................................................................................................... 24 2.6 Evaluation Limitations..................................................................................................... 28 3. Project Evaluation Outcomes ............................................................................................... 29 3.1 Evaluation Sample ........................................................................................................... 30 3.2 Project Evaluation Endline Results.................................................................................. 35 4. Impact Evaluation Findings................................................................................................... 56 4.1 Impact Evaluation Samples.............................................................................................. 56 4.2 Impact Evaluation Results ............................................................................................... 59 5. Qualitative Discussion........................................................................................................... 66 5.1 Relevance......................................................................................................................... 67 5.2 Efficiency.......................................................................................................................... 73 5.3 Effectiveness and Impact (Perceived) ............................................................................. 78 5.4 Sustainability.................................................................................................................... 96 ii | AIR.ORG 6. Conclusions......................................................................................................................... 102 6.1 Key Outcomes, by LEARN Interventions........................................................................ 102 6.2 Key Findings with Respect to Evaluation Criteria.......................................................... 104 7. Recommendations.............................................................................................................. 112 Annexes................................................................................................................................... 116 Annex A. References................................................................................................................. A1 Annex B. LEARN Results Framework......................................................................................... B1 Annex C. LEARN Evaluation Questions..................................................................................... C1 Annex D. McGovern-Dole Performance Indicators..................................................................D1 Annex E. Additional Tables ........................................................................................................E1 Annex F. Other Subtests of Reading Assessment......................................................................F1 Annex G. Inter-Rater Reliability ................................................................................................G1 Annex H. Survey Instruments ...................................................................................................H1 Annex I. Qualitative Protocols ................................................................................................... I1 Annex J. Endline ToR.................................................................................................................. J1 iii | AIR.ORG List of Exhibits Exhibit 1. LEARN Geographical Coverage ....................................................................................... 4 Exhibit 2. Program Activity Packages.............................................................................................. 5 Exhibit 3. Sample Sizes for the Project Evaluation at Endline, by County.................................... 10 Exhibit 4. Numbers of Schools and Students in Impact Sample................................................... 13 Exhibit 5. Impact Evaluation Design for Grand Gedeh County..................................................... 14 Exhibit 6. Impact Evaluation Measurement Comparisons ........................................................... 14 Exhibit 7. Number of Respondents for the Project Evaluation, by County .................................. 15 Exhibit 8. Overview of Student Survey Key Topics ....................................................................... 17 Exhibit 9. Overview of Topics Covered in Qualitative Protocols .................................................. 18 Exhibit 10. Endline Analysis Plan................................................................................................... 25 Exhibit 11. DID Effect on Literacy Outcomes................................................................................ 27 Exhibit 12. Baseline and Endline Levels for Key Project Indicators.............................................. 30 Exhibit 13. Distribution of Endline Sample ................................................................................... 31 Exhibit 14. Student Sex Distribution, by Grade ............................................................................ 31 Exhibit 15. Age Distribution, by Grade.......................................................................................... 32 Exhibit 16. Proportion of Students for Whom English Is Their Main Language ........................... 33 Exhibit 17. Proportion of Students with Above-Average Amounts of Important Items at Home, by County .......................................................................................................................... 34 Exhibit 18. Availability of Reading Materials in the Home ........................................................... 34 Exhibit 19. Access to Non-Textbook Reading Materials in School ............................................... 36 Exhibit 20. Frequency With Which Students Borrowed Non-Textbook Reading Materials to Take Home................................................................................................................................ 37 Exhibit 21. Household Literacy Activities in the Past Week ......................................................... 38 Exhibit 22. Teacher Attendance.................................................................................................... 40 iv | AIR.ORG Exhibit 23. Teacher Tardiness....................................................................................................... 41 Exhibit 24. Grade 2 Students’ Literacy Skills................................................................................. 43 Exhibit 25. Letter Knowledge........................................................................................................ 44 Exhibit 26. Proportion of Readers, by County .............................................................................. 44 Exhibit 27. Comprehension Subtests............................................................................................ 46 Exhibit 28. Student Knowledge Versus Practice of Critical Handwashing Moments ................... 47 Exhibit 29. Number of Times a Student Eats per Day................................................................... 48 Exhibit 30. Student Knowledge of Code of Conduct, by County .................................................. 50 Exhibit 31. Student Identification of Rules to Guide Teacher Behavior ....................................... 50 Exhibit 32. Student Perceptions of Gender Norms, by Gender.................................................... 54 Exhibit 33. Students Who Disagreed with Four out of Five Gender Norms, by County............... 55 Exhibit 34. Treatment Interventions............................................................................................. 57 Exhibit 35. Numbers of Schools and Students in the Impact Sample........................................... 57 Exhibit 36. Composition of Students’ Sex in the Impact Sample at Endline................................. 58 Exhibit 37. Grade Repetition......................................................................................................... 59 Exhibit 38. Overall Impact of LEARN on Key Learning Outcomes, by Treatment and Gender ....................................................................................................................................................... 60 Exhibit 39. Overall Impact of LEARN on Key Nutrition and WASH Outcomes, by Treatment Group and Gender ........................................................................................................................ 63 Exhibit 40. Qualitative Intervention Sites, by LEARN Packages Delivered ................................... 66 Exhibit 41. Respondents’ Educational Attainment, by Interview Type and Intervention Stakeholder................................................................................................................................... 67 Exhibit 42. Relevance-Related Indicators and Findings................................................................ 68 Exhibit 43. Efficiency-Related Indicators and Findings................................................................. 73 Exhibit 44. Effectiveness and Perceived Impact-Related Indicators and Findings....................... 78 Exhibit 45. Sustainability-Related Indicators and Findings........................................................... 97 v | AIR.ORG List of Acronyms AIR American Institutes for Research ANOVA Analysis of Variance CART Center for Action Research and Training COVID Corona Virus Disease 19 DAC Development Assistance Committee DEO District Education Officer DID Difference in Differences ECD Early Childhood Development EGRA Early Grade Reading Assessment EMIS Educational Management Information System FFP Food for Peace FGD Focus Group Discussion GPS Global Positioning System ICC Intra-Cluster Correlation IRB Institutional Review Board IRC International Red Cross IRR Inter-Rater Reliability KAP Knowledge, Attitudes, and Practices KII Key Informant Interview LB Literacy Boost LBRA Literacy Boost Reading Assessment LC Literacy Champions LEARN Liberia Empowerment Through Attendance, Reading, and Nutrition MDE Minimum Detectable Effect MGD McGovern-Dole MOE Ministry of Education NGO Non-Governmental Organization OECD Organization for Economic Cooperation and Development vi | AIR.ORG OLS Ordinary Least Squares PIRE University of Liberia Pacific Institute for Research and Evaluation PMP Performance Monitoring Plan PTA Parent Teacher Association RTI Research Triangle Institute SC Save the Children SES Socio-Economic Status SF School Feeding SHC School Health Club SHN School Health and Nutrition SMS Short Message Service SOW Scope of Work SRGBV School-Related Gender-Based Violence TG1 Treatment Group 1 TG2 Treatment Group 2 THR Take Home Ration TOR Terms of Reference UNESCO United Nations Educational, Scientific, and Cultural Organization UNICEF United Nations Children’s Fund USAID United States Agency for International Development USDA United States Department of Agriculture WASH Water and Sanitation WFP World Food Programme WHO World Health Organization i | AIR.ORG LEARN Endline Evaluation Report Executive Summary Project Background and Purpose Liberia Empowerment Through Attendance, Reading, and Nutrition (LEARN) is a 5-year program (2017–2022) funded by the United States Department of Agriculture (USDA) McGovern-Dole International Food for Education and Child Nutrition program (McGovern-Dole). Save the Children (SC) is leading the implementation of LEARN in partnership with SC Liberia, Mercy Corps (MC), and government partners, including the Ministry of Education (MOE), the Ministry of Agriculture, and the Ministry of Health. LEARN aims to reach 132,780 direct beneficiaries, 60,164 of whom (students) are expected to receive meals through school feeding activities in a total of 220 schools recruited in two cohorts of schools across four counties: Grand Bassa, Grand Gedeh, Rivercess, and River Gee. Both cohorts of schools, which started receiving the same program activities in the 2018–2019 school year, aim to achieve the McGovern-Dole objectives in a similar manner. LEARN program activities are designed to achieve USDA’s two strategic objectives: (a) improve the literacy of school-age children by enhancing the quality of instruction and increasing student attentiveness and attendance through provision of school meals and take-home rations and (b) increase the use of health and dietary practices by enhancing the knowledge of health, nutrition, and hygiene best practices; upgrading sanitation facilities; and improving food safety and storage systems. Evaluation Methodology SC selected the American Institutes for Research (AIR) to design and conduct the project and impact evaluations of the LEARN project from 2018 to 2022. The endline evaluation, which is the focus of this report, examines the relevance and effectiveness of the interventions, measures the progress of LEARN implementation across all four targeted counties, determines whether the project met its desired objectives, and provides lessons learned and recommendations for consideration in planning future projects. Below is a snapshot of our evaluation approach for the endline evaluation:  Sampling. A sample of 85 schools in which data were collected from Grade 2 and Grade 6 students for the performance evaluation and a sample of 55 schools in which data were collected from Grade 2 students for the impact evaluation. ii | AIR.ORG LEARN Endline Evaluation Report Performance Evaluation Impact Evaluation 85 Schools 1,655 Grade 2 & Grade 6 Students 55 Schools 721 Grade 2 Students  Methodology. A mixed-methods approach for a performance evaluation to measure LEARN progress over time as well as a difference-in-differences framework for an impact evaluation to assess the causal effect of LEARN interventions on student literacy, health, and nutrition.  Data sources. Primary quantitative data from 2,376 students. Qualitative data from 350 students, caregivers, teachers, principals, cooks, storekeepers, community mobilizers, and district education officers (DEOs) across 12 school communities (48 focus group discussion [FGDs] and 63 key informant interviews [KIIs]); four national SC and MC staff members; four national Government of Liberia staff members; and one county-level Government of Liberia staff member. 2,376 Students 148 Schools 48 FGDs 63 KIIs 350 Community Stakeholders  Analysis. A comparison of baseline (2018) and endline (2022) values across all four counties, triangulating the survey data with qualitative interviews for the project evaluation, as well as an assessment to capture changes in the literacy of school-aged children and their health and nutrition knowledge, attitudes, and practices (KAP) due to the three key packages of LEARN, including school feeding, literacy boost, and/or school health and nutrition interventions in Grand Gedeh County. Findings and Conclusions We highlight below the project and impact evaluation endline results that are most pertinent to the key research question themes from the McGovern-Dole results framework. We first present key outcomes informed exclusively by the quantitative data. Then, to explain and contextualize some of the outcome findings, we present findings around key outputs and intermediate outcomes that are informed primarily, though not exclusively, by the qualitative data. iii | AIR.ORG LEARN Endline Evaluation Report Key Outcomes Literacy Outcomes Reading with comprehension increased overall from baseline to endline, and this result was driven mainly by improvement among boys. At endline, 4% of students could read with comprehension. Interestingly, there was a significant decrease in the proportion of students who could correctly answer an evaluative question related to the text. This finding was common across all counties and may point to a loss in learning due to COVID-19 school closures which in turn may be differentially affecting girls due to prevailing gender norms. Among readers, we observed improvements in reading accuracy and fluency. The percentage of words read accurately improved greatly from 10% at baseline to 51% at endline, and fluency (words read correctly per minute) doubled from 11% to 22%, on average. The combined package (school meals plus literacy boost and school health and nutrition interventions) led to improvement in reading comprehension scores, particularly among boys. Boys in schools that received the combined package schools were 6.3 percentage points more likely than boys in comparison schools to read with comprehension. In contrast, we do not find any similar impact on reading comprehension skills for girls who received the combined package. Effects on any of the intervention schools (base package or combined package), show a 1 percentage point increase in reading comprehension relative to the comparison group for girls but a 4-percentage point increase for boys. This effect was likely driven by the ability of combined packages to increase readers among boys; null effects were found on readership among girls. Several key reading outcomes worsened from baseline to endline, and these outcomes did not always vary uniformly between counties. Like at baseline, at endline students were mostly successful at identifying letters but struggled beyond that to identify full words. While 85% of the Grade 2 students could identify letters at endline, only 23% could identify words, and only 11% were classified as readers. These outcomes generally decreased similarly across counties except for students classified as readers, whose proportion decreased the most in Grand Gedeh and River Gee Counties, by 14 and 10 percentage points, respectively. We did not find any significant effects of either the base package or combined package on knowledge of letters. COVID-19 related school closures may explain the general decline in reading outcomes, but further analysis is needed to understand differential trends in reading outcomes across countries, especially inGrand Gedeh and River Gee where literacy boost interventions were implemented. Overall, the combined package of LEARN interventions may help children to become readers but may not help the average Grade 2 student who does not know letters to become more literate or a reader. iv | AIR.ORG LEARN Endline Evaluation Report Nutrition Outcomes Knowledge of a balanced diet did not change from baseline to endline. Only 3% of students stated that they knew the definition of a balanced diet at endline, and of those, nine students could successfully identify all three components of a healthy diet. The impact evaluation did not show any significant change in knowledge of a balanced diet due to any of the interventions, perhaps due to the small sample sizes. Handwashing Outcomes Handwashing knowledge only slightly increased since baseline, with knowledge levels varying by county. Handwashing knowledge significantly increased in both Grand Bassa and Grand Gedeh and significantly decreased in River Gee and Rivercess between baseline and endline. Overall, self-reported handwashing behavior improved—albeit by 6 percentage points—relative to baseline. The data show statistically significant improvements in Grand Bassa and Grand Gedeh and decreases in River Gee and Rivercess. Grand Gedeh had the largest proportion of students with handwashing knowledge and recommended practices at endline, registering a 21-percentage point improvement in self-reported practices over the low 2% level at baseline. School-Related Gender-Based Violence Outcomes Students were largely aware that rules exist for how teachers should treat students at school, and this high level of awareness remained constant from baseline to endline. The county-level data showed large differences, with nearly all students in Rivercess and River Gee showing awareness of these rules and the proportion decreasing in Grand Bassa County by 7 percentage points. The proportion of students who had improved knowledge of school-related gender-based violence (SRGBV) issues and willingness to report increased significantly across all counties besides Grand Bassa, and differences were negligible by sex. We also asked Grade 6 students whether they disagreed with gender stereotype statements. Sixty percent of students at endline compared to only 48% at baseline, showed significant improvement in students’ perceptions of gender norms. However, a much lower percentage of students in Grand Bassa disagreed with at least four out of five gender stereotype statements compared to the other counties, especially River Gee. Primarily with reference to the qualitative data, LEARN outputs and intermediate outcomes are reflected upon below. Additional quantitative findings are provided and specified as such within the text. v | AIR.ORG LEARN Endline Evaluation Report Outputs and Intermediate Outcomes School Feeding and Gardens Relevance. Students and caregivers overwhelmingly agreed on the importance of education for both girls and boys. Students self-reported enjoying school and had high expectations for what they could achieve in the future if they were able to continue their education. Local stakeholders and beneficiaries agreed that the school feeding portion of the project was highly relevant and aligned with the needs of their communities, where parents were often faced with the choice between keeping their children out of school so they could engage in livelihood strategies and sending them to school hungry. Take-home rations for girls were also appreciated, though according to some stakeholders and beneficiaries, it wasimportant to also consider boys’ needs for such support given their own gender-specific pressures to leave school (e.g., to do physical labor or other work) or to clarify the rationale behind providing take-home rations to girls only. Gardens and larger farms would be a major asset in supporting school feeding interventions, in addition to providing Parent Teacher Associations (PTAs) with a source of reliable income that would alleviate their need to ask parents for money to support various school projects. School and community-level stakeholders working on school feeding indicated that they felt sufficiently trained and supported in fulfilling their roles, though there were some reports of inadequate materials (e.g., kitchen supplies). Efficiency. The school feeding program was slow to roll out from inception but has been on course since 2019. At endline, sporadic challenges remained related to reports of theft of commodities, kitchens not having adequate supplies, and PTAs sometimes not providing adequate condiments and dishes. LEARN worked more purposefully with PTAs to help them establish the school gardens meant to support school feeding activities. This required convincing parents to spend time carrying out the work. The Ministry of Agriculture admitted having trouble distributing gardening materials on time, and the growing season was missed by around 25% of school communities in 2022. Effectiveness and Impact. Respondents of all types across all communities were extremely positive about the school feeding program’s impact on attendance, learner focus and attentiveness, and relief for parents on food security at endline. According to the quantitative data, at baseline only 1% to 2% of students reported liking school because food was provided, but this proportion jumped to 15% overall at endline, with 21% of students noting food as a reason for their enjoyment of school in River Gee and 43% in Rivercess. Some learners complained that the rations were becoming monotonous, and there were rare instances of food supply being interrupted because of theft or condiments not being available. The potential of gardens was said to be increasingly recognized by parents and teachers. vi | AIR.ORG LEARN Endline Evaluation Report Sustainability. Endline data suggest that schools would not be able to continue feeding after the program concludes, even in the 64 LEARN communities where school gardens have been established, which are at this point only large enough to supplement food that is provided to the school. Commodities from other sources are essential, and SC is working to secure commitments from the government to try to achieve this more sustainably. Meanwhile, PTAs continue to struggle providing cooking materials, and kitchens regularly lack basic equipment. With an enhanced role for the PTAs and larger gardens and farming cooperatives, as well as production of Power Gari, which is planned for LEARN II, sustainability may become more likely. Literacy Champions and Trained Teachers Relevance. Improvement in teachers’ ability to teach literacy was a welcome advance in overall school quality, but LEARN’s goal of providing students with additional out-of-school support faced some challenges. Though students exhibit a general interest in and enthusiasm for education and literacy, since baseline it has been apparent that it is often difficult to find family members, tutors, or after-school teachers to adequately support students who were learning how to read. Thus, LEARN’s work with Literacy Champions and other community volunteers supporting literacy (e.g., community volunteers supporting Summer Reading Clubs and distributing Home Reading packets during COVID-19 closures) to provide out-of-school support and resources was highly relevant and well-received. Literacy Champions and teachers indicated that they appreciated the additional support provided by LEARN through trainings, monitoring visits, and teacher appreciation activities. Efficiency. Since midline, SC staff recognized there was high turnover or limited commitment among the teachers trained to be Literacy Champions and responsible for delivering much of the literacy boost component in the designated schools. Consequently, the investment in training those teachers to be the Literacy Champions was often lost. Acknowledging this, SC took steps to reduce the problem, including identifying volunteer non-teacher Literacy Champions within the community, being more attentive to the needs of the trained Literacy Champions (teachers or volunteers), such as providing additional training throughout the year, making regular monitoring visits to hear their requests, and ensuring regular provision of rations (the Literacy Champions’ form of compensation). SC also instituted other measures to encourage teachers in general to stay, which was particularly important in more rural and less well serviced areas, as teachers in these areas tended to try to relocate elsewhere. These measures included instituting teacher recognition activities and working with local education government officials (DEO and CEO) to try to improve pay and working conditions. Importantly, SC was also able to implement the Summer Reading Clubs in 2021, which had not been possible in 2020 because of COVID (instead, Home Learning Packets were distributed to try to substitute the learning). vii | AIR.ORG LEARN Endline Evaluation Report Effectiveness and Impact. At endline, beneficiary and stakeholder perceptions about the situation appear to have generally improved despite challenges encountered as a result of the COVID-19 closures that may have impacted on some of the learning outcomes observed (described in Outcomes, above). Teachers in literacy boost communities reflected on multiple topics that they had learned about in recent trainings on teaching literacy, and students recognized the different components of the instruction they had received. The reading clubs (after-school or weekend reading activities), summer reading camps, and other out-of-school literacy activities provided by the Literacy Champions were broadly welcomed by students and parents in the literacy boost communities. This is notable, given the challenges that SC had with maintaining Literacy Champions at midline, when students in literacy boost communities perceived the reading support they were given as largely indistinguishable from the support in other communities. Literacy Champions themselves were also generally positive about their ability to help children in K–Grade 2 to learn to read. However, Literacy Champions identified several challenges to carrying out their work. For example, all eight of the Literacy Champions mentioned that they did not feel adequately compensated for their work, which was demoralizing and led to attrition. Sustainability. The literacy component of LEARN has made huge gains in potential sustainability. First, Literacy Champions have shown great progress and gained popularity while working as volunteers. However, they indicate that working for food rations is not ideal. The stipends that some PTAs have collected may help convince some to continue their work, but their persistence is not guaranteed. Also, SC’s advocacy in favor of teachers may have had some impact on local MOE representatives, in addition to MOE’s procurement of vehicles for supervision, and adding volunteer teachers on MOE payroll. However, it is unclear whether this will lead to additional and enduring increase in the government’s attention to working conditions. At the same time, teachers will continue to impart the knowledge that they have gained through LEARN trainings, though the importance of follow-up training has been made clear. Access to Books Relevance. According to stakeholders and learners in the communities visited for qualitative research, access to books, as at midline, remained sporadic, though there was clearly a desire and need for them to help supplement other literacy measures. This was despite SC’s delivery of 118 copies of a new books to book banks in Grand Gedeh and River Gee in 2022. Frequent requests for more books were considered to be insufficiently responded to. Efficiency. At endline, there were no reports in the communities visited for qualitative research of community-based book banks existing, though some Literacy Champions are said to have books that they could share with students. Literacy Champions also had been trained to develop their own local reading materials to address the limited supply of books. viii | AIR.ORG LEARN Endline Evaluation Report Effectiveness and Impact. All communities mentioned the need for more learning materials, books in particular. Boys and girls reported often reading their notes at home because of lack of other reading materials. Literacy champions in some cases reported having books that they were able to lend to students, but this did not appear to be commonplace. Literacy champions described how SC trained them to create reading resources for them and their students to use. They explained that they developed their own materials to help teach children to read and also taught children how to make their own reading materials. According to the quantitative data, there was an increase in the amount of reading materials that students had access to at home. The proportion of students reporting that they have no reading materials at home decreased by 6 percentage points to a low 9%. Also, the proportion of students with access to holy books and textbooks at home increased from baseline to endline. Changes were generally uniform across counties. Parent Support for Literacy at Home Relevance. Aiding caregivers and others in the community in supporting reading at home helped fulfill a critical need, especially in non-literacy-boost communities that did not have access to reading clubs or camps. Providing illiterate parents with support was important, for such parents often lacked the confidence to support their children or believed the task required more time than they had. There was no familiarity with the “I Help My Child to Learn” tool, though parents did reflect on hearing radio jingles or LEARN or partner education radio programs that were encouraging them to support their children in their education. Efficiency. An important finding from the midline evaluation was that parents often did not feel empowered to engage with their children’s literacy education. SC continued working purposefully with caregivers to encourage more engagement with their children’s education at home, even given their self-perceived limitations (e.g., lack of literacy). Effectiveness and Impact. At endline, there were clear indications that this work had been effective. All caregiver groups reported that they tried to help their children as often and as much as possible and regularly asked about how they were doing. Students also perceived receiving good support at home, though they said they would benefit from having even more help, in particular having their own “study class teacher” or “home teacher” (as a personal tutor). Boys and girls perceived this report equally–none identified that boys or girls had particular challenges either during or after COVID-19 closures. According to the quantitative data, home literacy activities generally improved from baseline to endline. There were large variations between counties, with the largest proportion of students in Grand Gedeh and River Gee Counties reporting literacy activities at home in each category. ix | AIR.ORG LEARN Endline Evaluation Report Parent-Teacher Associations and Parent Engagement Relevance. PTAs serve not only as an important mechanism for communication between parents and teachers but also as a meansfor parents to mobilize other parents and community members to support school initiatives. PTAs can thus help to supplement services that are not always reliably provided by the government and whose provision by programs like LEARN will not continue indefinitely. At midline, the engagement of parents with their own children’s education was limited, primarily as a result of their own uncertainties about how to help their children. Meanwhile, teachers held the view that parents did not care. Efficiency. At midline many PTAs were said to be inactive both before and after COVID-19 closures; those that were active engaged in limited activities (fewer than expected of them if they were to assist with the LEARN interventions). Recognizing the need to work more with PTAs, SC conducted community outreach, including reinforcing and clarifying PTA roles and responsibilities, sensitizing PTAs to the need to increase the numbers and types of people involved with them (i.e., beyond a few active leaders), and requesting the MOE’s PTA Engagement Division to better support PTAs. SC also helped parents identify and implement strategies to support their children’s education, such as regularly asking about homework or supporting a child’s effort to identify a family member who could help complete assignments. Effectiveness and Impact. While the midline found that PTAs were relatively inactive, interviews with PTA members and other teachers and caregivers across the 12 qualitative sites at endline suggest much improvement in the PTAs’ understanding of their role and the activities they carried out. At endline, nearly all respondents across all the communities said that the PTAs were actively meeting their responsibilities to LEARN and were also performing non-LEARN activities (e.g., supporting payment of volunteers and large-scale school infrastructure projects). They continued to struggle getting more parents to be involved, often because parents did not see the benefit of joining. Regarding parent engagement with their own children, most parents at midterm were clear that they wanted to engage with their child’s education and would try to make the time. At the same time, there were others who were not sure how to engage with teachers, especially when they had limited education and lacked confidence, they understood what to speak to the teachers about. At endline, the situation seems to have improved: Caregivers in all 12 FGDs said that they engaged with their child’s teachers and felt free to do so. However, as at midline, many teachers at endline continued to lament what they considered to be limited parent engagement. They did admit that there were indeed parents who were very involved. Among the teachers commenting on who does engage with them, all but one said that it was generally the female caregivers who took the time to talk with teachers. x | AIR.ORG LEARN Endline Evaluation Report Sustainability. The role that PTAs, parents, and others in the community have begun to play in supporting children to read, including identifying or making their own teaching materials, is promising. This culture of literacy is likely to be maintained, especially given that it was built upon a strong foundation of valuing education and high student ambitions for success. School Health and Nutrition Activities Relevance. Students and teachers emphasized the importance of having a sanitary environment with functioning infrastructure, which was not always available at their school. School health and nutrition (SHN) activities around promotion of healthy practices and nutrition were evidently necessary given the relative lack of knowledge that students had on these topics, but in the context of school environments sometimes lacking basic necessities such as soap and water, healthy practices and nutrition promotion became secondary in importance. Efficiency. LEARN faced similar challenges with SHN champions as they did with Literacy Champions: SHN champions were sometimes transferred to a non-LEARN school after being trained. Effectiveness and Impact. School health clubs (SHCs) appear to be less active than at midline, when they were very active on school cleaning and handwashing activities as a result of COVID￾19 prevention protocols. Also, at endline, while SHCs remained focused on school cleaning activities, they faced newer challenges such as procuring regular access to clean water and soap; likely as a result of this emphasis on school cleaning and handwashing, there was relatively less attention to nutrition. At endline, despite the PTAs’ increased involvement in school feeding and gardening activities, they seemed to be relatively uninvolved in WASH and other SHN activities, beyond supporting occasional infrastructure and cleaning projects. SHN champions, responsible for regular cleaning of the school, were not always active, and many students in SHN schools complained of lack of adequate sanitation. The county-level differences reported in Outcomes may be attributed to the different types of COVID-19 response and resources allocated at the county-level, though this is not confirmed by the evidence in this evaluation. Also, some cooks and storekeepers reported inadequate materials and infrastructure to regularly maintain food safety procedures. In this context, where a clean school environment was said to be difficult to attain, it appears the attention of SHCs and SHN champions may have been less focused on hygiene behaviors such as handwashing and even less on nutrition (a finding supported by quantitative data showing no improvement in these two areas). Indeed, in FGDs and KIIs, rarely was nutrition education discussed as it related to LEARN activities. According to the quantitative data, xi | AIR.ORG LEARN Endline Evaluation Report knowledge of a balanced diet did not change from baseline to endline. Only 3% of students stated that they knew the definition of a balanced diet at endline, and of those, only nine students could successfully identify all three components of a healthy diet. The impact evaluation did not show any significant change in knowledge of a balanced diet due to any of the interventions, perhaps due to the small sample sizes. Also, it could be a result of nutrition￾related activities being started relatively later in LEARN, and prioritization given to COVID-19 related sanitation procedures, giving less time for potential impacts to be observed. Sustainability. Stakeholders and beneficiaries agree that having a clean environment is critical, and much collaborative work is done to achieve this. However, midline findings that showed active SHCs may have been mainly a reflection of the phase, because the COVID pandemic led to more stringent WASH-related measures becoming common across Liberia. SHN champions and SHCs appear to have lost momentum at endline, suggesting their work may not endure past LEARN. School-Related Gender-Based Violence Relevance. Despite widespread knowledge about the existence of school codes of conduct, including what items were included in the code, SRGBV remained a problem in schools across Liberia, indicating that knowledge does not translate into agreement with the codes or improved adherence to the code. Also, students at endline relatively often indicated fear of retribution for reporting a teacher’s infraction. The potential of such safety concerns to limit attendance and achievement is clear. Multiple interventions from partners and the government have tried to overcome these challenges, which highlights the clear relevance of SC’s work here. Efficiency. The MOE national Teacher Code of Conduct (TCOC) was, at endline, nearly finalized, though drafts of revisions were circulated and schools were re-introduced to them. As part of this re-introduction, SC utilized a social-behavior change approach in tasking community mobilizers with sharing SC-developed Safe Schools Stories with teachers, parents, and students independently. Each story had a lesson to teach about the context and rationale for parts of the TCOC and to raise awareness about the TCOC while also promoting appropriate behavior and response to violations. It is unclear whether these stories were rolled out only in Grand Bassa and Grand Gedeh or throughout LEARN communities. Sharing the stories via radio programming had also been planned, though this had not been rolled out at endline and was to be considered for LEARN II. xii | AIR.ORG LEARN Endline Evaluation Report Effectiveness and Impact. Respondents across all groups reported that they were aware of the school code of conduct but there was an evident degree of misunderstanding or blatant disregard for the code from both students and teachers. At endline, students generally understood that there was a process for reporting a teacher for a violation of the code of conduct, but some students noted that reporting teachers could result in repercussions and that they feared making a report. County-level differences observed in the Outcomes may be attributed to the proximity of the counties to Monrovia or other larger cities (e.g., communities in Grand Bassa) that are relatively well-served in terms of sensitization and reporting/follow￾up mechanisms around SRGBV. Sustainability. The 2013 MOE school code of conduct is institutionalized knowledge; however, the degree to which the specifics of the rules and regulations are understood and respected is questionable given open admission of violations by some principals and teachers. To be sustainable, such rules and regulations need to be fully understood and also fully agreed upon by stakeholders who will push for adherence and accountability. There is clearly still work to be done in this regard. Evaluation Limitations Some potential limitations of note that could arise include: Reliance on Self-Reported Data. The main limitation is that the quantitative approach relies on self-reported data from children for several socially and culturally sensitive subjects such as SRGBV. Although AIR adopts best practices in eliciting this information, this could still have some degree of measurement error, like data collected in other contexts on such sensitive topics. To mitigate this limitation, prior to the baseline data collection in 2018, AIR devoted considerable attention to cognitive testing of the survey instrument with students in Grades 2 and 6. In consultation with the local partners, AIR adjusted question phrasing to make sure children could understand the questions and feel comfortable answering. In addition, to further improve data reliability at endline, AIR incorporated some of these topics in qualitative interviews to triangulate with quantitative data. Internal Validity of Impact Evaluation. A key assumption for the reliability of the impact evaluation is that there is no contamination of schools in comparison groups. However, SC monitoring data from our midterm evaluation suggests that at least 13 comparison schools in Grand Gedeh received some form of literacy or WASH intervention from another donor during the study period. Given this caveat, the results from the impact evaluation should be interpreted with caution as there was potential contamination of the comparison schools. In other words, true program effects may be underestimated. xiii | AIR.ORG LEARN Endline Evaluation Report Internal Validity of Qualitative Findings. As with all qualitative research, results are not necessarily generalizable, but rather show the broad spectrum of types of perspectives that may be encountered across project beneficiaries and stakeholders. Because of this, the communities chosen purposefully represent the broad types of community across LEARN (rural, peri-urban, urban; the combined package, the base package). Difficulty accessing district education officers (DEOs). The qualitative field team struggled to interview four of 12 targeted DEOs despite multiple attempts to reach them in their office or on the phone. Recommendations Below, AIR presents recommendations based on key project outcomes, limitations, and lessons learned from the endline evaluation. Though LEARN is ending and most of the recommendations cannot be implemented at this point, they may be relevant to LEARN II and similar programming in the future.  Enhance literacy among non-readers. Further explore which types of students work with Literacy Champions or engage in other literacy boost interventions (e.g., reading clubs) to determine whether those who are already readers tend to seek this support more often. If existing readers tend to seek this help and non-readers do not, this could help explain why students who are already readers tend to improve while non-reading students do not. It may be beneficial to target non-readers or facilitate access to non-readers to literacy boost activities. Alternatively, if non-readers are being supported with such activities but still do not improve, then providing customized instruction based on their skill level may better help these less advanced students to progress.  Closely monitor MOE-hired teacher trainers to learn more about what they are focusing on as it relates to literacy. This will help SC to both contextualize literacy outcome findings and provide insights to help intervene where it seems necessary to better improve the desired outcomes (e.g., curriculum reform).  Continue using the option of mobilizing volunteer Literacy Champions instead of tasking teachers with the role but clarify with the volunteers the reasons their position is not, and will not be, compensated. At endline, volunteer Literacy Champions were effective and motivated, but did express some concerns with payment in take-home rations only.  Produce innovative and locally made reading materials. Continue empowering students and parents to create their own reading materials when there is a lack of content to read. Literacy champions have provided good examples of how children can use locally made materials (e.g., flashcards and transcribed stories narrated by community members) to enhance literacy. xiv | AIR.ORG LEARN Endline Evaluation Report  Continue to work with the government to better support and maintain teachers and other community volunteers supporting education initiatives (e.g., volunteer Literacy Champions). Advocacy in this regard would need to come from multiple partners regularly, for example as has been done already through the Education Sector Development Committee, but it is critical to continue to acknowledge to the government the degree to which teachers lament being underpaid and overworked and feel the government is not listening to their concerns. In the meantime, expanding on existing strategies to help acknowledge teachers’ work and provide supplementary compensation (e.g., through PTAs, or the STAR teachers intervention) could further help enhance teacher morale, attendance, and performance. Also, there remains the need to address the issue of frequent transfer of teachers to other schools, particularly those who have already been trained as Literacy Champions or SHN Champions.  Strengthen PTAs to support schools in the longer term. PTAs have critical roles in schools beyond LEARN activities. LEARN refresher trainings and meetings with PTAs after midline was effective in re-activating some PTAs that had lost momentum following the COVID closures or had been inactive for years prior. Working with PTAs to ensure that they have their own system for making and carrying out plans and remaining active without outside encouragement such as through LEARN can be critical in helping schools sustain themselves in the face of limited or sporadic government support. PTAs could have a more systematic role in supporting teachers who are feeling forgotten by the government or boosting the morale of Literacy Champions who lament not being paid cash for their work.  Attract more parents into PTAs. Continue stressing to PTA leaders the importance of including multiple parents and community members and train the leaders on strategies to attract parents and community members. One strategy is to convince parents that work done in collaboration with the PTA will ultimately provide compensation in the form of school improvements or parental influence over which activities are chosen.  Emphasize the importance of parents’ engagement in their children’s education and facilitate dialogues between parents and teachers about the challenges parents face in engaging with their children’s education. Teachers and principals can emphasize to parents the critical and constructive role they can play in enhancing their children’s education even without being educated themselves. This will also enable teachers to better understand the challenges that parents face and the assistance they in their efforts to support their children. With teachers, develop realistic strategies that parents and caregivers can use to encourage their children going forward.  Enhance PTAs’ understanding of the role of school gardens. Continue sensitizing PTA members to the active role that school gardens can play beyond supplementing school xv | AIR.ORG LEARN Endline Evaluation Report feeding activities. Rather, school gardens can be viewed as an income generation opportunity. For example, a larger garden could generate more income for PTA activities or help individual PTA members cover their children’s educational expenses. Meanwhile, it is critical that children are not exploited: teachers and students should be made aware that student work in the school garden is not meant to be done as punishment or demanded as free labor. Rather, all students, parents, and teachers can be expected make small contributions to the garden.  Reiterate to communities the rationale for providing girls with take-home rations (THRs): they are aimed at reducing the risk of sex for grades and grooming, demonstrating commitment to equality by giving girls a boost (critical given past and current evidence of boys performing better).  Ensure schools have adequate materials and infrastructure to maintain a healthy and safe environment, particularly in kitchens. While cooks and storekeepers demonstrate adequate understanding of food safety procedures, they lament lack of materials or poor infrastructure to ensure they can keep up to those standards.  Work with the government to get its commitment to support institutionalizing school feeding across Liberia schools. Not only is school feeding popular, it increasesthe attendance and performance of students while alleviating many caregivers’ concerns about the well￾being of their children. At the same time, a school garden and the PTA alone cannot sustain daily hot lunches; additional commodities are essential. Implementation of the LEARN II school feeding model will provide an important case study in how to effectively roll out and sustain school feeding.  Separate WASH and nutrition components rather than grouping them as SHN, and task different parties to manage each. SHCs demonstrated willingness and capacity to engage in school cleaning activities, and some were active in teaching fellow students about handwashing. However, improving nutrition was rarely mentioned, likely because of the already difficult task SHCs and SHN champions had in maintaining school cleanliness. Having separate individuals responsible for the nutrition component (e.g., dividing an SHCs into two “wings”) may help prevent the important issue of nutrition from being sidelined.  Respect and enforce the school code of conduct. With the revision of the school code of conduct will come opportunities for widespread sensitization around its content, including the opportunity to have dialogues with school personnel, caregivers, and students on their perspectives. This will help elucidate what is limiting enforcement of the code of conduct, such as misunderstanding of the content despite the ability to list items in the code, disagreement with some of the rules, and lack of alternative disciplinary strategies that are in accord with the code (i.e., strategies that could replace corporal punishment). xvi | AIR.ORG LEARN Endline Evaluation Report  Follow-up with schools on the status of their TCOC complaints mechanism to ensure it allows for children’s anonymity and protection, and that school leaders act on complaints made (or justify rationale for inaction) so that the system remains both safe and effective.  Track fidelity of implementation and contextualize findings and recommendations based on what has happened. Throughout project implementation, conduct regular assessments to identify gaps in implementation and work to fill those gaps appropriately. A robust monitoring system to closely track fidelity of implementation may benefit the project and lead to a more refined evaluation of the project’s impacts at endline. This will also be critical during the scaling-up and expansion to occur as part of LEARN II. Such a system could also aim to identify other implementers working on similar projects within the project’s catchment area, allowing for collaboration and the avoidance of complications. 1 | AIR.ORG LEARN Endline Evaluation Report 1. Introduction The U.S. Department of Agriculture (USDA), through the McGovern-Dole International Food for Education and Child Nutrition Program, has funded Save the Children (SC) to implement the Liberia Empowerment Through Attendance and Reading (LEARN) project between 2017 and 2022. The project aims to improve literacy outcomes of school-age children and enhance the use of health and dietary practices. This endline evaluation assesses the progress of LEARN in achieving the desired project outcomes and addresses the relevance, effectiveness, efficiency, sustainability, and impact of key program interventions through implementation. We also provide recommendations for the next LEARN II project based on lessons learned. This section outlines the project’s context and describes the interventions and the theory of change for the LEARN project. The next section outlines the evaluation approach, including the research questions, evaluation design, sampling, data collection methods, data analysis, and limitations of this study. The third section presents the endline evaluation findings from our mixed-methods approach, while the fourth section describes the LEARN impact evaluation results. Section five provides more contextual information from the qualitative findings and discusses the effects of COVID and other factors on the endline evaluation results findings. Finally, we conclude with lessons learned and the implications for the McGovern-Dole results framework and provide recommendations for the next iteration of the LEARN II project based on the key findings, limitations, and lessons learned from the endline evaluation. 1.1 Project Context Learning (e.g., reading skills) contributes more to a country’s economic growth than school attendance (e.g., years of education). A 10% increase in the share of students who reach basic literacy skill levels translates into an annual growth rate that is 0.3 percentage points higher than it would be otherwise for that country (Hanushek & Woessmann, 2009). Consequently, many international development projects have refocused their missions to keep children in school and ensure that children are learning while at school. Similarly, McGovern Dole projects aim to improve school enrollment, academic performance, and overall student health by providing school meals to students, teacher training and related support; raising community awareness on school related gender-based violence (SRGBV); and improving health and nutrition practices. 2 | AIR.ORG LEARN Endline Evaluation Report Liberian Education. The Liberian education system faces many challenges with respect to student attendance, rates of graduation, teacher quality, gender discrimination, gender-based violence, and basic educational attainment. While primary education is free and compulsory in Liberia, over 15% of children 6 to 14 years-old are not attending school and only 69% of students complete grade 6 (Liberia Ministry of Education, 2016a). Various cultural and economic factors drive late enrollment and low attendance in primary and secondary school, including national crises like the civil war and Ebola, as well as other structural issues. The Liberian Ministry of Education (MOE) has identified these issues as the most impactful:  Low family income relative to school costs and enrollment fees  Need to participate in household income generating activities instead of schooling  Negative attitudes toward late and overage enrollment in general, and  Parental beliefs about education. Food Security and Malnutrition. Despite significant progress following the end of the prolonged civil war in Liberia in 1996, food insecurity and malnutrition are still widespread. Most Liberians live below the poverty line, 49% of the population are food insecure, and nearly 30% of Liberian children are stunted due to malnutrition (UNICEF/WHO/World Bank, 2022). Food insecurity is not distributed evenly across the country, as nearly 80% of Monrovians are food secure, but only 50% are food secure nationally (USAID, 2016). This widespread food insecurity has a significant impact on student performance, particularly in rural areas that are hardest hit by the effects of COVID. Given that students in rural areas are already less food secure than their urban counterparts, the prolonged national shutdowns and economic shocks will only add to their food insecurity, which could further decrease student performance. Recognizing the limitations in the Liberian system of education, in 2016 the MOE released its “Getting to Best” Education Sector Plan (2017–2021), which included suites of strategic reforms aimed at increasing enrollment, gender and economic parity among students, improving teacher training programs, and reducing wasted government funds associated with “ghost” teachers (Liberia Ministry of Education, 2016b). Part of the MOE response to school improvement has included the partnership with NGOs to promote improved school management, in conjunction with other donor organizations funding large-scale school meal programs (WFP, 2012). These school feeding programs have the potential to defray the costs associated with attending school for poor families and encourage higher enrollment rates in vulnerable populations, while also seeking to improve student performance overall by decreasing the issues associated with malnutrition; however, the World Food Programme (WFP) has noted, in its own assessments of similar projects, that results are frequently mixed (WFP, 2017). 3 | AIR.ORG LEARN Endline Evaluation Report Sexual and Gender-Based Violence. Gender-based violence is an epidemic in Liberia, and the conditions following the end of the civil war and the outbreak of Ebola have exacerbated this problem. Medie (2013) estimate that 35% of young women will face gender-based violence in their lifetime, and this figure is considered largely underreported, due to the repression of cases by the police and other authority figures. In a study conducted by Columbia University and IRC in 2013, Stark et al. found that over 50% of women in Montserrado and Nimba Counties had experienced non-sexual domestic abuse, and approximately 20% of women had been victims of rape outside of marriage. Staggeringly, over 70% of women surveyed had experienced marital rape. Overage enrollment, common in Liberia, consistently exposes young girls to older young men who are enrolled in the same grade. Several studies have found that nearly one third of primary school and secondary school Liberian children report having engaged in transactional sex for financial gain, respect, or improved grades. The age and power differentials make it incredibly difficult for these youth to refuse sex (Parkes, 2016). A 2015 survey of secondary school students in Liberia revealed that 30% of girls and 22% of boys reported that they were forced to have sex. Peer abuse was common but many children in the same study spoke of sexual abuse by teachers and other school staff (Postmus et al., 2015). Currently, there is not strong evidence showing that government actions and policies have done anything to curb incidences of gender-based violence in schools, or in broader society. COVID Challenges. The challenges noted above have been exacerbated by the COVID pandemic, especially gender-related challenges. Students who are out of school for extended periods of time are known to experience significant learning loss, which has been explored extensively in the context of long summer holidays out of school (Marcotte & Hemelt, 2008). A recent study of the impact of school closures on learning loss in Liberia and other countries in Africa found that school closures lead half to over a year’s worth of learning loss (Angrist et al., 2021). The same study also found that learning deficits for a child in grade 3 could lead to 2.8 years of lost learning by grade 10. Liberian public schools closed in March 2020 and reopened in July 2020, a loss of approximately 80 instructional days. Further, the COVID pandemic, like any other external shock, might have aggravated sexual and gender-based violence, which has been labeled by global leaders as a “pandemic within a pandemic.” 12 1 https://www.savethechildren.org/content/dam/global/reports/emergency-humanitarian-response/ebola-rec-sierraleone.pdf 2 https://blogs.lse.ac.uk/africaatlse/2020/04/23/gender-based-violence-in-zimbabwe-a-pandemic-covid19-virus/ 4 | AIR.ORG LEARN Endline Evaluation Report 1.2 Project Description SC is implementing the $25 million LEARN from 2017 – 2022 in partnership with SC Liberia, Mercy Corps, and government partners including the MOE, the Ministry of Agriculture, and the Ministry of Health; to respond to some of the obstacles underlying low enrollment, attendance rate and literacy level; lack of health, nutrition, and hygiene knowledge; as well as food security challenges. LEARN is not the first McGovern Dole project in Liberia. Prior to LEARN, the World Food Program implemented the McGovern Dole project from 2013 to 2016. LEARN aims to reach 132,780 direct beneficiaries, 60,164 of whom (students) are expected to receive meals through school feeding activities in four counties(Grand Bassa, Grand Gedeh, Rivercess, and River Gee). A total of 220 started receiving program activities in the 2018–2019 school year. Error! Reference source not found. shows the map of the program locations in the four counties. LEARN program activities fall into three intervention packages designed to achieve USDA’s two strategic objectives: (a) improved the literacy of school-age children by enhancing the quality of instruction and increasing student attentiveness and attendance; and (b) increased the use of health and dietary practices by enhancing the knowledge of health and hygiene best practices, upgrading sanitation facilities, and improving food safety and storage systems. To respond to local context and to examine the effectiveness of different interventions, all schools receive the base package of school feeding activities, but not all targeted counties receive the same combination of LEARN interventions. In Grand Gedeh, 20 schools receive school feeding, literacy boost (LB), and school health and nutrition activities, while a different set of 22 schools receive only school feeding. Grand Bassa schools receive only the school feeding intervention, River Gee schools receive school feeding and LB, and schools in Rivercess receive school feeding and school health and nutrition activities. Error! Reference source not found. Exhibit 1. LEARN Geographical Coverage 5 | AIR.ORG LEARN Endline Evaluation Report shows the full list of activities for each of the three intervention packages across the four counties. Exhibit 2. Program Activity Packages County School Feeding Base Package (SF) Literacy Boost (LB) School Health & Nutrition (SHN)  Provide school meals  Provide take-home rations (THR) for girls (Grades 4-6)  Distribute deworming medications, vitamins, and minerals  Institute teacher recognition  Build/rehabilitate storerooms, kitchens, stoves, latrines  Establish Parent Teacher Associations and provide training on food preparation and storage, good health and nutrition, commodity management  Establish activities to promote literacy  Train teachers to lead Reading Camps  Establish libraries  Produce books & reading materials  Promote increase community awareness on SRGBV3  Establish school gardens  Train teachers to lead School Health Clubs to improve health and nutrition practices Grand Gedeh    River Gee   Grand Bassa  Rivercess   Source: Terms of Reference (TOR) COVID Adjustments to Programming. As result of COVID, the MOE closed schools from March to July 2020. The MOE developed a two-phase emergency response plan to (a) ensure continuous learning for students during the stay-at-home period and (b) prepare schools to safely reopen after the pandemic.4 In the summer of 2020, the MOE began a phase to allow schools to open on a “catch-up” schedule depending on student grade, 5 with a revised timetable for the 2020–2021 school year to run from December 2020 to August 2021 for all grades.6 Despite this catch-up period, returning to school may not have been feasible for many students in areas with particularly high economic hardship. According to the LEARN fourth quarter Workplan 3 All four counties, regardless of their intervention packages, also receive the promoting the code of conduct intervention. 4 Since midline data collection, LEARN programming has predominantly returned to normal. The Midterm Report provides a comprehensive overview of the government’s national response and the adaptations made to the LEARN program by SC. 5 Phase I: Grade 12 students resumption of classes from June 29 to September 2, 2020 for completion of academic year 2019– 2020; Phase II: Grades 6, 7, 8 and 9 students resumption of classes for completion of the academic year 2019–2020 from August 13 to October 9, 2020; Phase III: Grade 10 and 11 students resumption of classes for completion of the academic year 2019–2020 from October 5 to November 14, 2020; Phase IV: ECE to Grade 5 students complete academic year 2019–2020 from August 17 to October 16, 2020; continue to remain and learn at home. 6 The academic year in Liberia officially runs from September to June. The MOE revised the timeline for school year 2020–2021 to adjust for COVID school closures. 6 | AIR.ORG LEARN Endline Evaluation Report Adjustments Concept Note, under this revised schedule, students would have attended fewer instructional days than the typical 200 instructional days during a regular academic year. During COVID, SC also altered the modality to deliver school meals and distributed THRs to all students in the 220 schools served under LEARN. This THR distribution occurred following COVID preventative measures, recommendations from the government, and WHO guidance. SC returned to the regular school lunch modality in January 2021, when schools resumed for the 2020–2021 school year. 1.3 Results Framework LEARN activities aim to ensure the following:  teachers are recognized for their good practices, including consistent attendance;  teachers and students have better access to appropriately leveled reading materials and those reading materials are accessible in and out of the classroom;  teachers clearly understand their roles in quality literacy instruction;  children’s short-term hunger needs are met during the school day;  disincentives to education are decreased by lowering the prevalence and acceptance of SRGBV;  improved nutrition and health practices, including appropriate food preparation and storage, regular deworming, and increased use of clean water and improved sanitation;  enrollment campaigns are strategically targeted to reach out of school children; and  parents and community members have an increased understanding of the benefits of education and the role that they play to support their children’s learning; According to the LEARN theory of change (see Annex B), if the program is able to meet the above mentioned outputs, then schools in Liberia will demonstrate improved quality of literacy instruction, student attentiveness, and student attendance, because the MOE, schools, parents, community members, and students will have the resources and knowledge required to affect literacy performance and improve their nutrition. Critical assumptions of the results framework included: (a) school meals are delivered in a timely manner, (b) key stakeholders view LEARN and USDA McGovern Dole as non-controversial, (c) political changes in the program areas or Liberia will not adversely impact program operations, (d) commodities can move throughout the country without unexpected delay due to weather-related disruptions, (e) whenever possible, commodities will be transported during the dry season, and (f) project activities to engage PTAs and parents are sufficient to maintain continued interest in LEARN. As noted above, some of 7 | AIR.ORG LEARN Endline Evaluation Report these critical assumptions, especially those related to disruptions of food supply and PTA engagement were impacted due to the COVID-19 pandemic. Further, SC also considered the following external factors that could have impacted the critical assumptions of the LEARN results framework:  Liberia has limited infrastructure due to sporadic civil wars. Roads are in poor condition because of poor maintenance and heavy rains. The LEARN project plans to ensure that movement of commodities is efficient and secure; however, if extreme weather makes roads impassable, the LEARN project will experience delays. The project assumes that extreme weather events will be minimal throughout the course of the program, and whenever possible, commodities will be moved during the dry season.  Liberia is currently Ebola-free. However, Ebola is endemic to the region, and outbreaks could occur on a semi-regular basis. Through the course of the 2014-2015 epidemic, the country made greater strides in managing Ebola. However, if those systems are not maintained, a large-scale epidemic may occur again in the future, which may, again, shutter schools for months at a time. LEARN assumes that future Ebola outbreaks will be contained reasonably quickly and that program schools will remain open throughout the course of the program. 1.4 Purpose of the Evaluation SC selected AIR to design project and impact evaluations of the LEARN project. The project and impact evaluations were designed in parallel to maximize comparability in the outcome indicators and findings by using coordinated qualitative and quantitative methods. The project evaluation measures changes of the key performance indicators over the life of the project across all LEARN targeted counties. The impact evaluation, on the other hand, focuses on measuring the causal effects of LEARN activities on literacy, as well as knowledge, attitudes, and practices (KAP) outcomes among Grade 2 students in Grand Gedeh County only. The objectives of these evaluations together at endline are to:  Assess the progress of LEARN implementation  Examine the relevance and effectiveness of the interventions  Determine whether the project met its goals and make recommendations for LEARN II  Chronicle sustainability efforts to date  Summarize lessons learned to date. In the following sections, we describe the research design, objectives, evaluation methodologies (quantitative and qualitative), and outcomesfor the impact and project evaluations of the LEARN interventions at endline. Finally, we conclude with some discussion and recommendations. 8 | AIR.ORG LEARN Endline Evaluation Report 9 | AIR.ORG LEARN Endline Evaluation Report 2. Evaluation Approach This section provides an overview of the quantitative and qualitative designs for the LEARN impact and project evaluations, including research questions, sampling design, data collection methods and analysis, and limitations. 2.1 Evaluation Questions To address the evaluation objectives, the endline evaluation sought to answer questions related to literacy, health, nutrition, and SRGBV outcomes. The endline evaluation questions focused on five dimensions of project achievements based on the criteria defined by the Organization for Economic Co-operation and Development's Assistance Committee (OECD-DAC), 7 including: relevance, effectiveness, efficiency, sustainability, and impact of key program interventions. For each OECD criteria, the table in Annex C lists the key evaluation questions, data source, and data collection method used to address these questions. 2.2 Evaluation Design AIR used a mixed methods approach for the project and impact evaluations of LEARN. AIR completed the baseline data collection in April 2018 (first cohort) and September 2018 (second cohort, added in August 2018). AIR collected the endline data in March and April 2022. For the endline evaluation, AIR followed the same evaluation methodology used at baseline and endline to generate an appropriate comparison with the endline using USDA guidelines. To accurately reflect changes in program performance over time, AIR measured the same program indicators at all data collection points, replicating AIR’s same quantitative methodology and sampling strategy at baseline. For the impact evaluation, AIR measured the causal effect of the various LEARN activity packages on literacy and health KAP outcomes among Grade 2 students only in Grand Gedeh County. AIR complemented the quantitative component of the evaluation with qualitative methods to assess the relevance, effectiveness, efficiency, and sustainability of the LEARN project, as well as explore perceived impacts of the intervention. Qualitative analysis aimed to understand stakeholders’ perceptions of the design and implementation of the program, their experience implementing or participating in components of the program, expectations for improved outcomes, and other relevant contextual information. 7 The OECD Development Assistance Committee Criteria were revised in December 2019. https://www.oecd.org/dac/evaluation/revisedluation-criteria-dec-2019.pdf 10 | AIR.ORG LEARN Endline Evaluation Report 2.3 Sampling Methods This section describes our endline quantitative and qualitative sampling strategies. 2.3.1 Quantitative Sampling Strategy 2.3.1.1 Project Evaluation Design and Sampling The quantitative component of the project evaluation measuresthe progress of the performance indicators in outcomes related to core LEARN activities from baseline (2018) to endline (2022). To accurately reflect changes in program performance over time, AIR measured the same program indicators at baseline and endline. To track the key literacy and health KAP indicators over time, AIR sampled a cross-section of Grade 2 and Grade 6 students8 across all four LEARN counties. At baseline, AIR followed the recommendations from the United States Agency for International Development (USAID) Early Grade Reading Assessment (EGRA) Toolkit9 to confirm that a sample size of 830 second graders for the literacy outcomes and 498 sixth graders for the health KAP outcomes were needed. Based on these power calculations, AIR confirmed that a sample of 83 schools (with an average of 10 students per school) was needed for the project evaluation. AIR implemented the baseline data collection in 85 schools in April 2018 (first cohort). Additionally, AIR conducted the second baseline data collection in September 2018 in 61 separate schools after SC added more schools to the program (second cohort). 10 At endline, we followed a similar approach as at baseline and midterm to ensure a meaningful comparison across time and selected 85 schools to sample. Our sampling goal was twofold: (a) select a representative sample of all 220 LEARN schools (regardless of whether they were part of the first or second cohort) and (b) select a sample of schools which were assessed at baseline (across both cohorts). At midterm, we compared the literacy scores of the midterm project evaluation sample, with the literacy scores of the first cohort that best matched the timing of the baseline data collection to maximize comparability of outcomes. We employed this strategy because the reading outcomes of the first and second cohort samples were statistically significantly different, and this variation likely arose from differences in the timing of data collection at baseline. Therefore, for fully capturing changes in reading outcomes between baseline and endline, we wanted to hold all else constant by comparing reading outcomes measured at the same time of the year. 8 For Grade 6 students, AIR only focuses on health and nutrition KAP, SRGBV, and perceived gender norms. 9 RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. 10 Overall, AIR collected baseline data in 146 schools. 11 | AIR.ORG LEARN Endline Evaluation Report At endline, we intended to visit the same schools as at midline. However, after reviewing the enrollment numbers shared by SC, we found that many of the schools did not have the number of students needed to meet our sample targets. Therefore, we used purposive sampling to select the endline schools as follows:  Step 1: Starting with all the schools visited during the baseline evaluations, we removed any school with enrollment fewer than 5 students of any gender in either Grade 2 or 6.  Step 2: Any school with high enough enrollment from the midline sample was automatically included in the endline sample. 38 schools overlapped with the midline sample.  Step 3: The remaining 47 schools were then selected randomly proportional to the schools needed in each county and cohort to match the midline sample distribution.  Step 4: in each of the 85 selected schools to survey, using the enrollment lists for the 2021– 2022 school year provided by SC, we randomly selected 10 students (five boys and five girls) from Grade 2 and six students (three boys and three girls) from Grade 6. Then, SC obtained parental consent from students prior to the day of data collection. Thus, the endline sample sizes for Grades 2 and 6 were contingent on the number of students with signed consent forms who were present in class on the day of the survey. During fieldwork, some of the students with consent forms were not present in school and could not be found to be surveyed, therefore the enumerators tracked parents of other students who were present and obtained their consent prior to collecting student data. Columns [6] and [7] in Exhibit 3 show the final student sample sizes at endline. Exhibit 3. Sample Sizes for the Project Evaluation at Endline, by County a The 16 schools needed for the project evaluation in Grand Gedeh are also a subset of the impact evaluation sample. County Number of schools in LEARN [1] Cohort 1 LEARN schools [2] Cohort 2 LEARN schools [3] Cohort 1 Evaluatio n Sample [4] Cohort 2 Evaluation Sample [5] Total Grade 2 students (10 per school) [6] Total Grade 6 students (6-10 per school)b [7] Grand Bassa 95 40 55 15 21 360 304 Grand Gedeh 42 42 0 16a 0 160a 144 Rivercess 44 30 14 12 5 170 146 Rive Gee 39 35 4 13 3 160 144 Total 220 147 73 56 29 850 738 12 | AIR.ORG LEARN Endline Evaluation Report 2.3.1.2 Impact Evaluation Sampling and Design Schools In the impact evaluation, AIR measured the effect of the various LEARN activity packages on literacy and health KAP outcomes among Grade 2 students only in Grand Gedeh County. At baseline, AIR assigned schools randomly to two treatment arms (full package of program activities, or only school feeding in the base package activities) and a comparison group (not receiving any program activities). Our initial power analysis confirmed that a sample size of 1,320 students, equally divided into 22 schools each for the two treatment arms and the comparison group (66 schools in total), would be sufficient to detect the minimum detectable effect size (MDE) of 0.42 standard deviations with a 95% level of confidence.11 Specifically, at the end of baseline data collection, we created 18 clusters of schools not more than 10 kilometers apart based on the geographic location of each school, mapped using global positioning (GPS) system coordinates. Most clusters consisted of an average of three schools, but the two biggest clusters included 10 and eight schools.12 These two large clusters were assigned to different treatment arms. Keeping the two large clusters apart, we randomly assigned all 18 clusters into three groups: two treatment groups and one comparison group. Furthermore, to ensure that SC reaches its target number of beneficiaries, we designated the group of schools including the largest cluster of schools to receive the combined package of all program activities. The second group of schools including the second largest cluster of schools was designated to receive the basic school feeding package. The third group with the smaller clusters was assigned to serve as the comparison group which did not receive any program activities. At baseline, three factorsled to a smaller than expected number of available schools and students surveyed: (a) changes to SC’s implementation design, (b) inflated Educational Management Information System (EMIS) enrollment numbers, and (c) survey implementation during the rainy season making it harder to reach schools. Consequently, AIR surveyed at baseline a smaller sample of 55 open and active schools rather than 66 schools. In fact, 11 schools were inaccessible during the rainy season. Further, due to inflated EMIS enrollment numbers, within each of the 55 surveyed schools, AIR was able to survey an average of 12 students per school rather than 20 second graders. The smaller sample of schools and students than originally anticipated resulted in loss of power. With an average of 19 schools per treatment or comparison group and 12 students per school, the MDEs increased to 0.45 S.D. This means that the program activities need to be even more 11 The power calculations used the following additional assumptions: power (β) of 0.80, intra-cluster correlation of 0.25, and a correlation of other covariates with the measured outcomes of 0.50. 12 In two towns in Grand Gedeh, it was not feasible to create small clusters of three or four schools without running into contamination and spillover concerns. 13 | AIR.ORG LEARN Endline Evaluation Report effective than initially anticipated for their impact to be captured by the impact analysis. This limitation is important because LEARN may very well have positive effects that AIR will not be able to identify in the impact measurement. Only much larger effects can be estimated to be statistically significant with smaller sample sizes. Reassuringly, however, no bias was introduced by not surveying 11 schools (out of the original impact sample of 66). As described above, AIR clustered schools and randomly assigned them to treatment and comparison groups after baseline data collection. Thus, no systematic bias is expected by the 11 schools that AIR was unable to visit during baseline. Similar to midline, at endline, AIR collected data from the same 55 impact evaluation schools. Sampling Students Rather than following the same students over time, a different sample of students was sampled at baseline, midline, and endline. A cross-sectional sample of students is preferable to a cohort design because of the substantial probability of student attrition from school (Feldman & McKinlay, 1994). In addition, having independent samples surveyed every period minimizes the probability that the act of measurement itself influences subject behavior. Children from the same cohort may score better in a test when they take the same type of test multiple times, not because they know more, but because they are more used to taking that test. At midline, AIR resumed data collection due to the COVID outbreak at the beginning of the school year 2020–2021 and collected data from Grade 3 students. Collecting data from Grade 3 students at the beginning of the school year at midline helped us assess the effect of project interventions on students’ literacy skills at the end of their grade level for Grade 2. At endline, AIR followed a modified strategy for sampling students since we collected parental consent from students prior to data collection. That is, the goal of the impact evaluation was to sample 10 boys and 10 girls from Grade 2 in each of the 55 schools (for a total target sample size of 1,100 students) to assess the effect of the project interventions on literacy skills. Prior to data collection, we provided SC with a larger sample of students to ensure that there would be enough students present on the day of the survey with signed parental consent forms. Exhibit 4 shows the endline impact evaluation sample composition by school and student. 14 | AIR.ORG LEARN Endline Evaluation Report Exhibit 4. Numbers of Schools and Students in Impact Sample SF LB+SF+SHN Comparison Total Baseline Number of schools sampled 22 20 13 55 Number of Grade 2 students surveyed 213 280 188 681 Midline Number of schools sampled 22 20 13 55 Number of Grade 2 students surveyed 202 297 195 694 Endline Number of schools sampled 22 20 13 55 Number of Grade 2 students sampled 440 400 260 1100 Impact Design AIR used an intent-to-treat approach to estimate the impacts of the offer to the LEARN activities on the outcomes of participating students in a difference-in-differences (DID) framework. Exhibit 5 summarizes the impact evaluation design. The two treatment arms measure the impact of (a) the LEARN base package consisting of school feeding and associated activities; (b) the LEARN combined package, consisting of school feeding implemented together with the package of Literacy Boost (LB) activities, as well as SHN activities; and (c) the incremental effect of LEARN add-on activities (LB and SHN) relative to school feeding only. By measuring the effects at endline (relative to baseline), the evaluation team can understand the impact of LEARN activities in the early years of implementation, and measurements of effects at endline (relative to baseline) will reveal the impact of LEARN activities once they have matured/stabilized over four years. The DID estimation framework to analyze the collected data is summarized in Exhibit 6 below. For example, to measure the impact of the LEARN school feeding base package on outcomes of interest at endline (row 1 of Exhibit 6), the DID framework would measure the change in the average outcome in treatment group 1 (TG1) between endline and baseline [G-A] and difference out the change in the average outcome in the comparison group (CG) over the same period [I-C]. Because the experimental groups are statistically identical, the change in the CG represents the counterfactual trend, which TG1 would have followed in the absence of LEARN, allowing the evaluation team to disentangle the impact of LEARN. Subsequent rows of Exhibit 6 summarize the different types of treatment effects we can measure at endline. 15 | AIR.ORG LEARN Endline Evaluation Report Exhibit 5. Impact Evaluation Design for Grand Gedeh County *Newcomers to Grade 3 at the beginning of the school year will be used as a proxy for Grade 2 students due to COVID delays in data collection. Exhibit 6. Impact Evaluation Measurement Comparisons Impact Evaluation Measurement Period Intervention Impact measurement Endline Δ LEARN Base Package; SF only (TG1 vs CG) = [G-A] - [I-C] Δ LEARN Combined Package; SF+LB+SHN (TG2 vs CG) = [H-B] - [I-C] Δ LEARN Add-On Interventions; LB+SHN (TG2 vs TG1) = [H-B] - [G-A] 2.3.2 Qualitative Sampling Strategy Building on the approach at baseline and midline and considering the findings, the endline qualitative research focused on perceived changes at the school and community levels after five years of program implementation. At endline, the evaluation team aimed to gain a more in-depth understanding of project’s effectiveness, relevance, sustainability, perceived impact, and efficiency. The qualitative research took place, as it did for baseline and midline, in three intervention schools in each of the four counties, for a total of 111 Focus Group Discussions (FGDs) or Key Informant Interviews (KIIs) across 12 schools. The purposive sample of communities aimed to capture perspectives from varied schools and communities, based on locale (rural, peri-urban, urban) and intervention package (LB, combined package, base package). For selection within the communities, students were randomly selected from rosters of those in 16 | AIR.ORG LEARN Endline Evaluation Report grades 4, 5, or 6. Groups were split between girls and boys, but grade levels were mixed.13 If students who were selected were not available at the time of research, students from the oversample list were contacted. Caregivers were selected largely by convenience, finding those who were available during the day of research, though efforts were made to include at least two who were members of the Parent Teacher Association (PTA). To maintain consistency from baseline and midline, AIR followed a similar approach to select respondents for KIIs and FGDs. 14All planned community-level interviews were accomplished with the exception of four (of 12) DEOs who were not able to be reached despite multiple attempts in person and by mobile, and one community mobilizer who was away at a workshop at the time of research and was unavailable to be contacted by phone. Literacy champions were only expected in the communities with literacy boost (all River Gee communities and one Grand Gedeh combined package community). All completed interviews are summarized in Exhibit 7, below: Exhibit 7. Number of Respondents for the Project Evaluation, by County Interview Type Grand Bassa Grand Gedeh River Gee Rivercess Total Girl Student FGD 3 3 3 3 12 Boy Student FGD 3 3 3 3 12 Caregiver FGD 3 3 3 3 12 Teachers FGD 3 3 3 3 12 Principals KII 3 3 3 3 12 Community Mobilizer KII 3 3 3 2 11 Cook KII 3 3 3 3 12 Storekeeper KII 3 3 3 3 12 Lit Champ KII 1 - 1 3 - 4 Lit Champ KII 2 - 1 3 - 4 DEO KII 1 1 3 3 8 Total Community-level 25 27 33 26 111 There were also six national-level interviews (with relevant government, SC, and Mercy Corps [MC] staff) conducted by the lead qualitative researcher via mobile or Skype. 13 FGDs at baseline and midline were similarly structured, and the differences in grade levels within groups did not appear to affect the quality of the conversation or students’ comfort in participating; as such, this design was used again for endline. 14Our previous research showed that talking to principals and teachers separately led to more uninhibited responses, as there is often a power imbalance between these two positions. As such, AIR continued this practice from baseline and midline. Additionally, in the Liberia context, we found no need to separate parent FGDs by gender as mothers felt comfortable expressing their opinions even in the presence of their spouses and other male community members. 17 | AIR.ORG LEARN Endline Evaluation Report 2.4 Data Collection Methods This section provides detailed information on data sources, as well as the data collection plan, enumerator training, instrument field testing, human subject protection, and quality assurance at endline taking COVID implications into account. Data collection for LEARN endline was conducted simultaneously with data collection for the baseline evaluation of LEARN II. LEARN II is the second phase of LEARN, being implemented from 2021 – 2026, which will focus on reinforcing local capacity at every level to address GoL and USDA objectives to provide nutritious school feeding and quality literacy education in a healthy, supportive environment. Quantitative and qualitative data collection tools for LEARN endline included additional modules to capture information relevant for LEARN II baseline as described in below. 2.4.1 Data Sources To answer the research questions for the endline evaluation and to determine values for performance indicators, AIR collected and analyzed data from three sources: (a) a student survey; (b) the Literacy Boost Reading Assessment (LBRA); and (c) qualitative data collection tools, including KIIs and FGDs.15 The instruments were developed and adapted to the Liberian at baseline. AIRused the quantitative data to measure the program’s progress toward its objectives, and LEARN’s causal effect on the key outcomes. We also triangulated the quantitative findings with qualitative data that focused on the relevance, effectiveness, efficiency, sustainability, and perceived impact of project interventions. See Annex H. Survey Instruments for the survey instruments and Annex I. Qualitative Protocols for the qualitative protocols. 2.4.1.1 Student Survey At endline, AIR administered the same survey to Grade 2 and 6 students, including the LBRA, which was developed, cognitively tested, and used for the baseline and midline data collection with additional questions necessary for the baseline evaluation of LEARN II. Exhibit 8 presents an 15 We also conducted a school assessment to support SC which is not analyzed here. 16 A cognitive interview is an individual, face-to-face, in-depth interview that aims to understand how a respondent comprehends and responds to questions. 18 | AIR.ORG LEARN Endline Evaluation Report overview of the seven key topics that the survey covers for LEARN endline and outlines the additional modules that were included to capture information relevant for LEARN II baseline. Exhibit 8. Overview of Student Survey Key Topics Topics Types of Questions Background information  Demographic information (e.g., students’ age, main language spoken at home, etc.) Hygiene and health knowledge and practices  Handwashing knowledge (when one should wash hands)  Handwashing practices (when students wash their hands) Nutrition knowledge  Knowledge of a healthy diet (e.g., if a student knows what a balanced diet is) SRGBV  Knowledge of SRGBV behaviors (sexual and physical violence and harassment; bullying; corporal punishment)  Awareness on the existence of the code of conduct in school and its revised version  Knowledge of/propensity to use/confidence in reporting mechanisms to report instances of SRGBV  Perceived gender norms (Grade 6 only) School environment  Attitudes toward their school  Teacher attendance Home environment  Home literacy activities (e.g., if anyone reads to students or tells them a story)  Parent’s engagement in home learning  Reading culture at home Disability  Difficulty in seeing, hearing, talking, walking, and etc. During the baseline evaluation, there were some limitations in asking SRGBV-related questions from young students due to the sensitivity of the subject. SRGBV is a “known-secret” in Liberia but participants were unwilling to admit any history of sexual abuse of students in their schools. At midline, we added several simple questions to our instrument around teacher conduct to better capture the school climate related to SRGBV using guidance and specific survey items from the USAID SRGBV Conceptual Framework for Measuring SRGBV. These improved survey questions were maintained at endline.17 Literacy Boost Reading Assessment In addition to the student survey, Grade 2 studentsin both impact and project evaluation samples also took the LBRA. AIR developed the LBRA using Liberia second grade textbooks, calibrated to the Liberian context through the Liberia MOE, and field tested on Grade 2 students in non-project schools during baseline. To generate an appropriate comparison with the endline, AIR used the 17https://www.globalreadingnetwork.net/sites/default/files/eddata/Conceptual%20Framework%20for%20Measuring%20SRGB V_FINAL.pdf 19 | AIR.ORG LEARN Endline Evaluation Report same LBRA to measure the changes of literacy outcomes from baseline to endline. Using the same instrument between impact and project evaluation samples also helps maximize comparability in literacy outcomes and findings between the two evaluations. 2.4.1.3 Key Informant Interviews and Focus Group Discussions As mentioned above, AIR followed the midline qualitative approach for endline and conducted semi-structured focus group discussions (FGDs) with students (boys and girls in separate FGDs), caregivers (men and women in mixed FGDs), teachers (men and women in mixed FGDs), and key informant interviews (KIIs) with principals, government stakeholders, project staff, and other community members involved with the project. Community level KII protocols were designed for 30-45-minute conversations and FGDs for approximately 90 minutes. Interviews were conducted in Liberian English or, if needed, ethnic languages that the local researchers spoke (teams were formed around knowledge of languages in areas to be visited). The following exhibit presents an overview of the qualitative research protocols. Though discussion guides for the local qualitative team were written in Standard English, the qualitative team was experienced in interpreting questions into Liberian English during interviews, including rephrasing the wording of the questions to help the participants understand the question being asked, as is typical in qualitative semi-structured interviewing. Discussions between the lead qualitative researcher and the research team helped the research team to refine, and to practice delivering, all questions in Liberian English and local languages, as necessary. Exhibit 9. Overview of Topics Covered in Qualitative Protocols Topics Types of Questions (asked to groups / individuals) as relevant; see protocols in Appendix D)18 Protocol(s) Addressing Topic Background information  Background and role in project All Access to and value of education  Access to education in the community; barriers to access and full engagement (who is excluded)  Gender-equity of access  Whether parents are using the ‘I help my child to learn tool’  How confident parents feel in supporting their children’s learning and wellbeing Caregivers FGD Teachers FGD Principals KII 18 Discussion guides to be used by the qualitative team have detailed lines of inquiry and probes already written, to ensure that they solicit the required information. For national-level interviews with government and implementers, to occur while the local teams are in the field, the qualitative lead will design questions in advance of the interviews depending on (a) the person’s role; (b) initial findings from the evaluation. As such, detailed protocols for these interviews are not included in the Appendix. 20 | AIR.ORG LEARN Endline Evaluation Report Topics Types of Questions (asked to groups / individuals) as relevant; see protocols in Appendix D)18 Protocol(s) Addressing Topic School feeding/ nutrition  Existence of and quality of kitchen, gardens  Perceived effectiveness of feeding program; successes and areas for improvement  Knowledge of and agreement to ground rules on gardening activities; challenges to date Students FGD Caregivers FGD Teachers FGD Principals KII Community Mobilizers KII Cooks KII Storekeepers KII DEO KII SC / MC Project Staff National-level Ministry of Agriculture (as suggested by SC) School health clubs/ water, sanitation, and hygiene (WASH)/ nutrition  Perceived effectiveness of SHN champions and school health clubs on improving nutrition and WASH practices in schools  Progress on Save collaboration with community education officers (CEOs) and district education officers (DEOs) to provide training to the SHN Champions  Perceived effectiveness of community mobilizers Students FGD Caregivers FGD Teachers FGD Principals KII Community Mobilizers KII DEO KIIs National-level Ministry of Health (as suggested by SC) School literacy environment  How much, and how, students are exposed to literacy activities within the school environment (e.g., presence of library, teacher reading exercises)  Resources and encouragement teachers provide to students to read outside of school (e.g., can take home library books, working with parents/PTAs to encourage reading at home)  Feasibility of teachers with added load as Literacy Champions; related events Students FGD Teachers FGD Principals KII Literacy Champions KII Community Mobilizers KII National-level Ministry of Education (as suggested by SC) Home/ community literacy environment/ reading clubs  How much (and how?) students are exposed to literacy activities within the home (e.g., presence of books or other reading materials)  Whether literacy is valued in the home (e.g., if reading and doing homework is encouraged)  Existence/quality of community-based reading activities and resources (e.g., book banks, reading clubs, reading festivals (not yet started)), ease of accessibility to materials within  Reflections on summer reading clubs  Adequacy of training received to be Literacy Champion Students FGD Caregivers FGD Teachers FGD Principals KII Literacy Champions KII Community Mobilizers KII 21 | AIR.ORG LEARN Endline Evaluation Report Topics Types of Questions (asked to groups / individuals) as relevant; see protocols in Appendix D)18 Protocol(s) Addressing Topic SRGBV  Information on the extent to which students, parents, and teachers know about whether they are protected in the school by (a) a code of conduct that restricts SRGBV behaviors and (b) an effective referral and reporting mechanisms to report such behaviors if they do occur.  Positive discipline strategies (as alternative to corporal punishment) in place, and their effectiveness or limitations  Existence of/effectiveness of reporting mechanisms for students/teachers to use to report violations of school code of conduct  Whether parents are listening to the safe school stories Students FGD Caregivers FGD Teachers FGD Principals KII Community Mobilizers KII DEO KIIs National-level Ministry of Education (as suggested by SC) Parent￾Teacher Associations  Existence and activities of PTAs; specific successes and areas for improvement to enhance collaboration and effectiveness.  Degree to which parents in PTAs collaborate with teachers/principals  Perceived effectiveness of parent engagement messages on literacy Caregivers FGD Teachers FGD Principals KII Community Mobilizers KII In addition to community-level FGDs and KIIs, the lead qualitative researcher conducted open￾ended KIIs on phone or Skype with national-level Save and Mercy Corps staff, and relevant staff of ministries involved with LEARN (Agriculture, Education, Health), as recommended by Save. Questions asked to these respondents were based upon the respondent’s own role in LEARN, and to learn more about specific emerging findings from analysis of field-level qualitative data. These interviews also asked general question on greatest successes and key challenges of LEARN activities since midline; perceived capacity for sustainability, lessons learned, and recommendations for LEARN II. 2.4.1.4 School Assessment At endline, similar to baseline and midline, AIR conducted a school assessment. The purpose of the school assessment was to observe any changes in the status of students’ enrollment, attendance, and/or school characteristics. We included items on the checklist to collect observational data on safe food preparation, storage practices, and latrine cleanliness. In addition, we included items to better capture drop-our rates for students and teachers. These data are not analyzed in this report and will be shared with SC to triangulate self-reported responses from the interviews, as well as help SC identify gaps in resources so that it can use to make any necessary modifications for LEARN II. 22 | AIR.ORG LEARN Endline Evaluation Report 2.4.2 Human Subject Protection Prior to collecting data for the baseline evaluation for LEARN II, we received approval for our research protocols and instruments from the Institutional Review Board (IRB) of AIR, the University of Liberia Pacific Institute for Research and Evaluation, and SC’s Ethics Review Committee, ensuring that the evaluation (from baseline to endline) complies with local and international rules and procedures and meets the standards for the ethical research of children. This included protocols for ensuring that adequate health and safety measures related to COVID are followed (e.g., distancing and masking during interviews). During the enumerator training, AIR briefed enumerators on procedures for interviewing respondents, protecting respondents’ privacy and confidentiality, following COVID safety protocols during the survey,19 and securing the data. AIR also invited the SC Liberia team to provide enumerators with a refresher training on safeguarding children at school. To collect survey data from students and conduct focus groups with students, we obtained parental consent. Given the low literacy levels amongst parents, SC facilitated an awareness￾building session for PTAs, which included parents of children in the sampled schools. SC explained in detail the content of the consent forms at these meetings. At the end of these sessions, parents were invited to sign the consent forms, and AIR surveyed only those children whose parents completed the consent forms. Additionally, AIR asked for students’ assent before collecting data. This assured children that their participation was voluntary and that they could terminate the survey at any point. For all KIIs and FGDs, AIR received consent from adult participants. AIR also assured respondents that their participation was voluntary with referral mechanisms in place and that they could terminate the interview at any time. If respondents did not consent to recordings, we took detailed notes of the discussion instead. Our qualitative and quantitative field team received training on procedures for contacting respondents, protecting respondent privacy and confidentiality, child safeguarding, and securing data, thus ensuring high compliance with ethical guidelines to conduct research. Furthermore, after data collection, the evaluation team protected the privacy and confidentiality of respondents by storing the data on secure servers and separating personally identifiable information from the survey data. The data will be archived on the server at the end of the contract. 19 AIR shared a copy of COVID safety protocol with each of the enumerators. Also, enumerators were provided written agreement that they accept the risks and were comfortable moving forward with the mitigation measures we had in place. 23 | AIR.ORG LEARN Endline Evaluation Report 2.4.3 Data Collection Preparation AIR partnered with the Center for Action Research and Training (CART) for data collection. CART has worked with us to collect data for LEARN evaluations since the LEARN baseline in 2018. CART hired 33 enumerators – many of whom worked on the LEARN midline evaluation. AIR held an enumerator training from February 22 to 25, 2022. The AIR team led the in-person training of enumerators remotely20 in collaboration with the CART team leaders and fieldwork managers. Prior to the training, AIR reviewed all training tasks with CART’s director and fieldwork managers to ensure that, in case of connectivity issues, she would be able to continue leading the training. The training consisted of three days of theory-based classroom training and one day of pilot testing in a nearby school. During classroom training, enumerators learned: (a) the purpose of each survey question, (b) how to ask questions directed to vulnerable respondents (in this case, children under 18), (c) how to assess students’ literacy, (d) how to use tablets to implement the in-person surveys without an internet connection, and (e) how to survey respondents following COVID safety protocols. Pilot testing provided an opportunity for enumerators to practice with real respondents. Afterward, enumerators regrouped with the AIR team remotely to debrief and discuss any issues they encountered. Prior to data collection, the AIR qualitative lead held multiple remote training and discussion sessions with CART’s four qualitative researchers. CART’s researchers field-tested selected protocols such as FGDs with teachers and students, based on the availability of respondents in the pilot school, and regrouped remotely with the AIR team to debrief afterward. After pilot testing, the team met to discuss challenges such as comprehension (questions that confused respondents) and duration (insufficient time to complete all questions) and made necessary adjustments to the tools. The meetings also allowed the team to receive follow up training and opportunities to practice facilitation and notetaking in order to strengthen their interviewing and summarizing skills. 2.4.4 Data Collection Due to the new requirement of collecting parental consent (in prior evaluations consent was obtained from the school), fieldwork was delayed by two weeks while SC obtained consent. The quantitative team conducted fieldwork from March 16 to April 15, 2022. CART organized enumerators into two teams. Each team then split into groups of 2–4 who traveled to each school. One team focused on Grand Bassa, which had the largest sample, while the other team visited Grand Gedeh, Rivercess, and River Gee. The fieldwork managers, in collaboration with the MOE and school district offices, coordinated their school visits with school principals. All enumerators regrouped with their supervisors several times during the data collection to debrief, 20 Due to COVID, AIR could not undertake in-person training for the endline evaluation. 24 | AIR.ORG LEARN Endline Evaluation Report submit daily paper-based data collection logs, submit electronic surveys, and review and plan for the next days of data collection. The CART director and fieldwork managers were responsible for updating AIR’s project director on challenges and decisions. The AIR data specialist regularly downloaded the data through a secure server to run quality assurance checks and flagged the findings back to the team in the field to make additional decisions and adjustments as needed. For the qualitative data collection, two teams of two CART researchers each (three women and one man) collected data in the targeted schools. During the interviews and focus groups, one person led the discussion, while the other took notes. The male researcher was not present in any of the girl student FGDs (the female researcher took her own notes while facilitating). The local qualitative team summarized the main points of each session using a structured summary “field form” with one discussion question per page that paralleled the structure of the focus group or interview protocol. The summary synthesized the major points and salient themes and included verbatim quotations that addressed the supplemental evaluation questions. With the respondents’ permission CART also recorded all KIIs and FGDs as a back-up for the qualitative team to fill in gaps in their notes, as needed, on the same day that data collection occurred. The finalized detailed notes were entered into a Google Sheets database which was exported to Excel for the lead qualitative researcher’s subsequent coding and analysis. For quality assurance, within 48 hours after the first school’s KIIs and FGDs were completed, samples of notes pages and description of activities completed, and challenges encountered were sent to the lead qualitative researcher via WhatsApp. Feedback was provided to help to ensure high￾quality and complete data. The notes and recordings from the KIIs and FGDs were not shared outside the evaluation team. Throughout the fieldwork, all possible COVID protocols were followed to ensure the safety of our team, project stakeholders, and beneficiaries. The field team wore masks and followed social distancing when administering the evaluation instruments. They also carried hand sanitizers and extra disposable masks for respondents to wear, if comfortable, when collecting data. AIR also monitored COVID developments and relevant government guidelines closer to the fieldwork and worked closely with SC in case a new contingency plan was needed for the endline evaluation, which ended up not being necessary. 2.4.5 Challenges The data collection team faced several challenges. First, collecting parental consent added a new step to the process, which delayed data collection and reduced the number of available students to survey. While almost all parents consented, reaching those parents and then finding the students at schools proved difficult due to highly variable attendance. It was clear early on during fieldwork that many of the students whom we had consent to survey were not in attendance and 25 | AIR.ORG LEARN Endline Evaluation Report many of the students who were at school, had not been reached with the consent forms. To mitigate this, CART began sending a small group of enumerators to the schools’ towns ahead of time to seek out parents for their consent. Another challenge was the low enrollment and attendance in schools. The CART team reported 13 schools where there was no enrollment of 2nd and/or 6th graders at all. The team learned several reasons for low enrollment or attendance: (a) some students were dismissed for not paying tuition fees; (b) students were engaged in economic activities such as gold mining; (c) some schools have limited teaching staff, which discourages students from attending; (d) many of the schools closed for a semester break and after which many students failed to return to school. These school closures were another challenge the team faced. Many of the schools closed unexpectedly for a week, delaying the team’s efforts. To mitigate these challenges the team oversampled in any school that had extra students. Additionally, the team coordinated with the DEOs, principals, and SC field staff to assist with communication for the team’s visits. Finally, CART sent small teams to revisit certain schools multiple times in an effort to reach the target sample. In one case, the team visited a school five times in hope of finding additional students but instead found that most of the students were working on their families’ farms. In the end, the team was able to survey approximately 75% of the target sample. 2.5 Data Analysis Methods We started our data analysis with an exhaustive assessment of quality for both the quantitative and qualitative data before proceeding with data cleaning and analysis. 2.5.1 Quantitative Analysis After completing the field activities, we conducted a final review of the survey data, including:  Checking for data completeness  Checking for duplicate entries  Testing skip pattern logic of tablet survey programming  Data cleaning We then compiled the survey responses into a master file for the analysis. 2.5.1.1 Project Evaluation Analysis Overall Analysis Plan. To assess the progress of LEARN’s implementation at endline, like midline, we used a representative sample of 85 LEARN schools from first and second cohorts at endline. We compared the literacy scores of the endline project evaluation sample, with the literacy 26 | AIR.ORG LEARN Endline Evaluation Report scores of the first cohort that best matches the timing of the midline data collection to maximize comparability of outcomes. We employed this strategy because the reading outcomes of the first and second cohort samples were statistically significantly different, and this variation likely arises from differences in the timing of data collection at baseline. Therefore, for fully capturing changes in reading outcomes between baseline and endline, we wanted to hold all else constant by comparing reading outcomes measured at the same time of the year. To analyze the progress on other outcomes, we used the aggregated values from the representative random sample of 85 schools selected from both cohorts. Exhibit 10 shows the endline analysis plan for different outcomes. Project Evaluation Analysis. We first assessed the quality of the collected survey data by running frequency tabulations and counting missing responses. The evaluation team examined the frequency distributions for each survey question to ensure that all data are within a valid range. Then, the team used the survey data to construct the performance indicators required by USDA in a similar manner as at baseline and midline. To measure changes over time, AIR used a pre-post comparison using constructed means through clustered t-tests and corresponding p-values to highlight statistically significant differences. AIR used this methodology to assess and quantify LEARN’s progress by tracking changes in outcomes over time. The pre-post comparison method implicitly assumes that the program rollout for both cohorts is approximately the same and both cohorts receive the same level of exposure to the program interventions. As applicable, AIR disaggregated the findings by gender, county, and activity package. Exhibit 10. Endline Analysis Plan 27 | AIR.ORG LEARN Endline Evaluation Report This analysis can only suggest a correlation—not causal relationship—between the observed changes in outcomes and LEARN interventions such as school feeding or teacher training. For example, improvements in knowledge regarding hygiene could be due to other government programs between baseline and endline that provide hygiene-related health education. Self￾reported data on culturally and socially sensitive topics such as handwashing and hygiene, gender norms, and SRGBV may be subject to a social desirability bias.21 2.5.1.2 Impact Evaluation Analysis We estimated the program effects at endline using regression analysis of student outcomes. We used the following difference-in-differences (DID) specification for each treatment arm: 𝑌𝑖𝑠𝑡 = 𝛼 + 𝛽Postt + γ Treatments + Δ Treatments*Postt + Ψ Xist + 𝜀𝑖𝑠𝑡 Where:  Yist is the outcome of interest (e.g., reading with comprehension, letter recognition, handwashing knowledge and behavior, and nutrition knowledge) of student i in school s at time t 22  Post is an indicator variable, which takes the value of 0 if the outcome measurement is from baseline. It takes the value of 1 if the outcome measurement is from endline.  Treatment is an indicator variable equal to 0 for the comparison group and equal to 1 for the treatment group in consideration. We consider the following treatment groups: (a) school feeding only (the base package); (b) school feeding plus literacy boost and school health and nutrition (combined package); (c) groups a and b combined. Our comparison group does not receive any intervention.  The coefficient on the interaction term Treatment*Post (Δ) represents the DID estimate, i.e., the change in outcome from baseline to follow-up in the treatment group relative to the change in the same outcome for the comparison group  Xist includes a list of variables that the team used to control for their effect on outcomes to isolate the effect of the intervention, to improve precision in our estimates and to address any concerns regarding imbalance in treatment verses comparison groups (e.g., age, gender, number of household assets, English as main language at home)  𝜀𝑖𝑠𝑡 is the error term 21 Social desirability bias refers to tendency of research subjects to give what they perceive to be socially desirable responses, rather than responses that reflect their true feelings on sensitive issues. These results should be interpreted with caution. 22 Both dependent variables of interest are binary and we use a linear probability model. 28 | AIR.ORG LEARN Endline Evaluation Report Students in the same school might have similar observable and unobservable characteristics. Therefore, the outcomes of students within a particular school may be related to each other (i.e., the correlation between literacy outcomes of children in the same school may be high).23 To account for this correlation of error terms across students in the same school, we clustered the standard errors at the school level. Exhibit 11 visually shows the impact of the interventions to facilitate easier comprehension of the intuition of the approach. As shown, β is the time trend as represented by the comparison schools’ changes in outcomes. γ is the average difference in outcomes between the treatment and comparison schools at baseline. Our main outcome of interest is Δ, which we measured as the difference in the average outcome in treatment schools before and after treatment minus the difference in average outcome in the comparison schools before and after treatment, controlling for time effects. To study how results vary by gender, we also conducted subgroup analyses by sex. 2.5.2 Qualitative Analysis The qualitative data collection complemented the quantitative data by contextualizing the findings from the literacy assessment and the student health KAP assessment. The data were not interpreted as quantitative data as the nature of the qualitative research does not allow the results to be empirically generalizable. However, it offered critical perspectives to enrich the quantitative data, provided necessary context around the circumstances that might have influenced some of the quantitative findings. The qualitative data also provided additional information from beneficiaries and stakeholders about project activities that were not directly asked in the quantitative surveys, in addition to providing the critical perspectives of other stakeholders who were not reached with the quantitative component. The evaluation team relied on detailed notes and summary forms from the KIIs and FGDs to analyze the data, synthesize the findings, and identify major themes to address key evaluation 23 “The clustering problem is caused by the presence of a common unobserved random shock at the group level that will lead to correlation between all observations within each group” - Christian B Hansen. Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects. Journal of Econometrics, 140(2):670–694, 2007. Exhibit 11. DID Effect on Literacy Outcomes Source: AIR 29 | AIR.ORG LEARN Endline Evaluation Report questions. A Google Sheet--based qualitative database was built for the data collection team to enter their notes in English; this was exported into Excel at the end of data collection. As part of qualitative analysis, codes were established to help assess the relative response types provided across all the notes, paying attention to where (a) the qualitative data supported the quantitative data; (b) there were outliers; and (c) nuance was not captured by the quantitative tools. Also at this phase, quotations demonstrating key topics were pulled for use in the qualitative sections of this evaluation report. These data were also analyzed around the OECD criteria. 2.6 Evaluation Limitations Some potential limitations of note that could arise include: Reliance on Self-Reported Data. The main limitation is that the quantitative approach relies on self-reported data from children for several socially and culturally sensitive subjects such as SRGBV. Although AIR adopts best practices in eliciting this information, this could still have some degree of measurement error, like data collected in other contexts on such sensitive topics. To mitigate this limitation, prior to the baseline data collection in 2018, AIR devoted considerable attention to cognitive testing of the survey instrument with students in Grades 2 and 6. In consultation with the local partners, AIR adjusted question phrasing to make sure children could understand the questions and feel comfortable answering. In addition, to further improve data reliability at endline, AIR incorporated some of these topics in qualitative interviews to triangulate with quantitative data. Internal Validity of Impact Evaluation. A key assumption for the reliability of the impact evaluation is that there is no contamination of schools in comparison groups. However, SC monitoring data from our midterm evaluation suggests that at least 13 comparison schools in Grand Gedeh received some form of literacy or WASH intervention from another donor during the study period. Given this caveat, the results from the impact evaluation should be interpreted with caution as there was potential contamination of the comparison schools. In other words, true program effects may be underestimated. Internal Validity of Qualitative Findings. As with all qualitative research, results are not necessarily generalizable, but rather show the broad spectrum of types of perspectives that may be encountered across project beneficiaries and stakeholders. Because of this, the communities chosen purposefully represent the broad types of community across LEARN (rural, peri-urban, urban; the combined package, the base package). Difficulty accessing district education officers (DEOs). The qualitative field team struggled to interview four of 12 targeted DEOs despite multiple attempts to reach them in their office or on the phone. 30 | AIR.ORG LEARN Endline Evaluation Report 3. Project Evaluation Outcomes This section presents summary statistics from the student survey, including the LBRA. We performed balance checks for key demographic characteristics to ensure that the endline sample is statistically equivalent to the baseline sample. We first describe the project evaluation sample and show a summary of key performance indicators. We then present data from the student survey and the LBRA to examine the differences in outcomes between baseline and endline including disaggregation by sex, grade, and county. 24 In analyzing the quantitative performance data, we compared mean outcomes at baseline and endline by using t-tests and p-values to highlight statistically significant differences. This analysis can only suggest a correlation—not causal relationship—between the observed changes in outcomes and LEARN interventions such as school feeding or teacher training. For example, improvements in knowledge regarding hygiene could be due to other government programs between baseline and endline that provide hygiene related health education. Self-reported data on culturally and socially sensitive topics such as handwashing and hygiene, gender norms, and SRGBV may be subject to a social desirability bias. 25 Exhibit 12 presents the baseline and endline levels of the four key project evaluation outcomes required by the performance monitoring plan, disaggregated by sex. Annex D presents a complete table of endline levels for key McGovern-Dole performance indicators. We first describe the characteristics of schools and students sampled for the project evaluation; then, we discuss student reading outcomes, including factors such as home and school literacy environment, in addition to LBRA results. The section concludes with a review of baseline and endline values in handwashing and hygiene, nutrition, SRGBV, and disability. Whenever possible, findings are disaggregated by sex, grade, and county. Annex E and Annex F provide additional detail. 24 All the percentages in this section are rounded to the nearest whole number. 25 Social desirability bias refers to tendency of research subjects to give what they perceive to be socially desirable responses, rather than responses that reflect their true feelings on sensitive issues. Therefore, these results should be interpreted with caution 31 | AIR.ORG LEARN Endline Evaluation Report Exhibit 12. Baseline and Endline Levels for Key Project Indicators Source: Student survey, LBRA AIR calculation. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: The survey included 1372 Grade 2 students and 649 Grade 6 students at baseline; 1,029 Grade 2 students and 626 Grade 6 students at endline. Calculations for “Percentage of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters”, “Percentage of children in target schools who demonstrate improved knowledge and practices toward SRGBV prevention and response”, and “Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand grade level text” include only Grade 2 students per requirements of the PMP. 3.1 Evaluation Sample To measure progress toward outcomes from baseline to endline, we surveyed only students with parental consent. Although we aimed to survey 10 students in Grade 2 and 6 students in Grade 6 in each selected school in Grand Bassa, Rivercess, and River Gee, low enrollment and attendance rates in the field required the survey team to oversample students in larger schools within the same county. Exhibit 13 shows the total number of students who participated in the student survey, by county, at endline. 0% 64% 58% 1% 0% 64% 54% 1% 0% 72%** 54%*** 5%*** 1% 72%** 44%*** 4%** 0% 10% 20% 30% 40% 50% 60% 70% 80% Percentage of grades 2 and 6 students in target schools who can identify the components of a healthy diet Percentage of students who in target schools demonstrate improved knowledge and practices towards SGBV prevention and response Percentage of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand grade level text Endline Girls Endline Boys Baseline Girls Baseline Boys 32 | AIR.ORG LEARN Endline Evaluation Report Exhibit 13. Distribution of Endline Sample County Number of Schools Number of Students Endline Endline Grand Bassa 35 630 Grand Gedeh 16 467 Rivercess 17 208 River Gee 16 350 Total 84 a 1,655 Source: Student survey, authors’ calculations; Note: Grand Gedeh endline figure is the total sample size for Grand Gedeh in the project evaluation only. Additional schools and students were sampled in Grand Gedeh for the impact evaluation sample described in the Section 4. aOne school in Grand Bassa had no students present despite several attempts. 3.1.1 Student Characteristics Disaggregating the sample by grade and sex, we see that the student sample is relatively balanced between boys and girls at endline (Exhibit 14). Among the Grades 2 and 6 level students at endline, 53% and 47% were boys respectively, whereas 54% of Grade 2 students and 56% of Grade 6 students were boys at baseline. The sample is more balanced at endline in terms of sex compared to baseline. Exhibit 14. Student Sex Distribution, by Grade Source: Student survey, authors’ calculations. Grade 2 level students at endline averaged 12.1 years of age with a median of 12 years of age, both values nearly identical to baseline (Exhibit 15). The range, however, was wide at 6 to 25 years of age. The large age gap and high average could be the result of a government policy in 2001 that mandated primary education for children and eliminated fees. Before the enactment of thislaw, the high price of education and 14 years of civil conflict deterred parents from sending their children to school. After the new law was passed, many parents enrolled their children in school regardless of age. The age spread for Grade 6 is slightly larger than for Grade 2 with a range of 10 to 25 years of age. 54% 46% 56% 44% 55% 45% 53% 47% 47% 53% 51% 49% 0% 10% 20% 30% 40% 50% 60% Grade 2 Grade 2 Grade 6 Grade 6 Total Total Boys Girls Boys Girls Boys Girls Baseline Endline 33 | AIR.ORG LEARN Endline Evaluation Report Exhibit 15. Age Distribution, by Grade Grade Mean Median Range Baseline Endline Baseline Endline Baseline Endline Grade 2 12.3 12.1 12 12 5-19 7-22 Grade 6 16.3 15.8 16 16 8-25 10-25 Source: Student survey, authors’ calculations. Age is in years. 3.1.2 Household Characteristics To understand children’s backgrounds, we asked students about their household size, primary caregiver’s schooling, socioeconomic status, and language spoken at home. We found that the average household size, reported by Grade 2 students across counties at endline, was 7 people. The endline average was slightly lower than the baseline sample, which had an average of 8 people per household. Grand Bassa had a slightly higher average household size than other counties at nearly 8 people per household, while Rivercess had the lowest average at 7 people per household. The spread between minimum and maximum household sizes was large, between 2 and 18 people. Large household sizes could be explained by the fact that, in rural areas, multiple families often live together as one community; students might have also different definitions for households. However, “very large” households are rare, with only 13 out of 1,028 students who responded at endline said that their household contained over 15 members. This is much lower than at baseline, when 63 out of 1,371 students who responded to the question with a number claimed that their household had over 15 members. On average, 76% of students said that their mother was their caregiver at endline compared to 72% at baseline. Girls were more likely to report their mother as caregiver (80%) than boys (72%) at endline; meanwhile, 18% of boys reported their father as caregiver, compared to 12% of girls at endline. The decrease in reporting of father as caregiver from baseline to endline (from 22% to 18%) and increase in reporting of mother as caregiver (from 67% to 72%) amongst boys indicates the overall rise in reporting mother as caregiver was driven by boys. Interestingly, among students who reported their mother as the caregiver, 61% said their caregiver went to school as a child at endline; however, this rate rose to 82% when students cited their father as the caregiver (up slightly from 78% at baseline). Overall, proportions of students who reported that their caregiver went to school at endline (65%) were slightly higher to those at baseline (60%), and these results were significant at the 1% level; there were no large differences by sex, however there are significant differences by county and grade. 34 | AIR.ORG LEARN Endline Evaluation Report Grade 2 students at endline were more likely to have a caregiver who attended school (67%) than Grade 6 students (61%), and these results are statistically significant at the 1% level. Additionally, a higher proportion of students in Grand Gedeh County had a caregiver who attended school (74%) than other counties, with the lowest rate being in Grand Bassa County at 56%. Rivercess and River Gee Counties featured rates of caregivers attending school of 63% and 70% respectively. Most students reported English was the main language spoken at home both at baseline (69%) and endline (72%). As shown in Exhibit 16, there was a significant decrease of students reporting English as their main language in Grand Gedeh County, where the figure dropped from 71% at baseline to 60% at endline. Nearly all households in River Gee County speak English at home; Grand Bassa and Grand Gedeh Counties had lower rates of English-speaking despite a large increase (of 13 % points) in Grand Bassa County from baseline to endline. Exhibit 16. Proportion of Students for Whom English Is Their Main Language Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Baseline: N = 958 for Grand Bassa, 198 for Grand Gedeh, 438 for Rivercess, 427 for River Gee; Endline: N = 630 for Grand Bassa, 467 for Grand Gedeh, 208 for Rivercess, 350 for River Gee. There was a slight but statistically significant difference in the amount of assets that students had in their homes, from baseline to endline. On average, students at endline had 0.16 more assets (from a list of eight total) at their homes than students at baseline. This was driven by a large difference in the number of students reporting that they had no assets at home; at baseline, this represented 14% of student responses, whereas at endline this number drops to 2%. The distribution of assets was relatively similar for boys and girls across baseline and endline. By county, the proportion of students with above-average amounts of important assets in their homes was relatively similar across baseline and endline in both Grand Bassa and Rivercess Counties; however,there was a large jump in this proportion across baseline and endline in Grand Gedeh County and a large decrease in River Gee County (Exhibit 17). 54% 71% 76% 92% 67%*** 69% 60%*** 79% 95%** 72%** 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Grand Bassa Grand Gedeh Rivercess River Gee Overall Baseline Endline 35 | AIR.ORG LEARN Endline Evaluation Report Exhibit 17. Proportion of Students with Above-Average Amounts of Important Items at Home, by County Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline: N = 958 for Grand Bassa, 198 for Grand Gedeh, 438 for Rivercess, 427 for River Gee; Endline: N = 630 for Grand Bassa, 467 for Grand Gedeh, 208 for Rivercess, 350 for River Gee Further, a majority of students (77% at endline) said they had a holy book at home (Exhibit 18). Textbooks and schoolbooks represented the next most frequently cited book (62%), while storybooks and comics were the next most-cited non-textbook reading material at home (35% at endline). A larger proportion of students in Grand Gedeh County said that they had no reading materials at home (14% at endline) compared to the other counties (4% to 8% at endline). Exhibit 18. Availability of Reading Materials in the Home Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Baseline: N = 2021; Endline: N = 1655 46% 42% 58% 53% 49% 68%*** 58% 41%*** 0% 10% 20% 30% 40% 50% 60% 70% 80% Grand Bassa Grand Gedeh Rivercess River Gee Baseline Endline 9%*** 4%*** 18% 35% 62%*** 77%*** 15% 7% 20% 35% 49% 66% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% No books at home Newspapers Coloring and drawing books Storybooks/comics Textbooks/schoolbooks Holy book Baseline Endline 36 | AIR.ORG LEARN Endline Evaluation Report 3.2 Project Evaluation Endline Results This section presents results on key outcomes related to literacy, health, nutrition, and SRGBV. For reading related outcomes, we analyzed data collected from Grade 2 students who also took the LBRA. For health, nutrition, and SRGBV outcomes, we looked at students’ responses from Grades 2 and 6, separately. 3.2.1 Student Reading Outcomes This section presents changes over time for Grade 2 students’ responses to survey questions about the literacy environment at school and at home. The questions focused on four key areas: (a) the availability of reading materials in and out of school, (b) students’ home literacy environment, (c) students’ attitudes toward schooling, and (d) the presence of teachers in schools. After reporting on the results of these survey questions, we outline findings from the LBRA. While literacy boost activities under LEARN were only targeted to schools in Grand Gedeh and River Gee, we present findings on literacy outcomes for all four counties to provide better comparisons and a more nuanced understanding of how schools receiving literacy boost programming are doing relative to schools that not receiving those same activities. Moreover, it is possible that schools in Grand Bassa and Rivercess are benefitting from other donor-funded education programs, 26 which could be improving literacy outcomes for students in those areas. Therefore, presenting comparisons of changes in literacy outcomes for all four counties provides some insight into how the LEARN activities are performing in relation to other education programs in Liberia. 3.2.1.1 Availability of Reading Materials Grade 2 students reported about the availability of reading materials at school and in their community at large. We also asked how often students borrowed books from school, if available. At School. More than half of Grade 2 students reported that their school had books other than textbooks. This was a considerable improvement over baseline, despite some regional variations (see Exhibit E7 in Annex E) and our endline finding that 44% still report not having access to books other than textbooks at school (Exhibit 19). The proportion of students who reported being able to take these books home to read for free at endline was slightly higher in Grand Bassa (41%) than in other counties, especially River Gee (8%). While Grand Gedeh and River Gee Counties had the highest proportion of students responding that they had access to books at school at endline (65% and 68% respectively), both counties registered an increase in the proportion of students 26 For instance, the USAID-funded Accelerated Quality Education for Liberian Children program is currently operating in Grand Bassa, and even though the program is targeted at out-of-school children, the program could produce spillover effects in terms of improved educational quality and community support for education which improve outcomes for in-school students in this area. 37 | AIR.ORG LEARN Endline Evaluation Report responding that they cannot take the textbooks off campus. These two counties drive the overall increase in students responding that they cannot take books off campus from baseline to endline (Exhibit 19). The overall increase in students responding that they can take textbooks home for free is driven by responses from students in Grand Bassa County. Across all counties at endline, only 3 students out of the total responses of 1,022 reported having to pay to bring reading materials home, which is a slight improvement over baseline results. Exhibit 19. Access to Non-Textbook Reading Materials in School Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline: N = 1,358. Endline: N = 1,022 Exhibit 20 shows the frequency with which students borrowed books from school. This question was asked of the students who said that their school allowed them to take books home either for free or at a cost. On average, 78% of students said that they had borrowed non-textbooks in the past week at endline (adding up the overall percentages for those students responding “every day” [4%], “a few times” [34%], and “once during the week” [40%]), considerably higher than the 63% at baseline. There was also a major drop in the proportion of students responding that they never borrow books from school. As the Exhibit E8 in Annex E highlights, a far higher proportion of students in Grand Gedeh at baseline (71%) reported that they had never borrowed books from school. At endline, this proportion decreased to 28% in Grand Gedeh, which still had the highest rate of students never borrowing books than any other county. There were also significant decreases in this rate in Rivercess and River Gee Counties. Grand Gedeh had a large increase in the proportion of students saying that they had borrowed a book a few times in the past week, from only 8% at baseline to 37% at endline. Students in Rivercess dramatically increased their rate of borrowing books a few times per week, from 12% at baseline to 82% at endline. 0%*** 23%*** 30%** 46%*** 2% 15% 26% 57% 0% 10% 20% 30% 40% 50% 60% Yes, and can take home for a cost Yes, but can't take off campus Yes, and can take home for free No Baseline Endline 38 | AIR.ORG LEARN Endline Evaluation Report Exhibit 20. Frequency With Which Students Borrowed Non-Textbook Reading Materials to Take Home Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: of students who did not refuse to answer or did not know. Baseline: N = 374. Endline: N = 309. Outside of School or Home. On average, only 21% of students reported that they had read books other than textbooks/schoolbooks outside of school or their home (e.g., reading club, church, etc.) in the past week; the endline results show a slightly lower proportion of students in this sample who reported extracurricular reading at baseline (29%). There were no notable differences by sex. However,students in Grand Gedeh County reported reading non-schoolbooks outside of school at a much higher rate (35%) than students in other counties (proportions ranged between 15 and 27%). Of the students who reported that they read non-schoolbooks outside of school or the home, 78% said that they went to a friends’ or relatives’ for reading materials, followed by 17% who reported “other”. Just over one percent or fewer reported using reading clubs, religious buildings or community libraries. The proportion of students reporting that they did not know where to borrow a book decreased from baseline to endline from 53% to 48% (p < 0.05). The qualitative data tell a similar story: students report reading whatever resources they have access to at home which, in most cases, are not storybooks. Rather, they report most often that they are reading their own notes taken during school, textbooks, or religious texts. 3.2.1.2 Home Literacy Environment Examining household literacy practices can illuminate the level of children’s exposure to learning outside of school. Students exposed to literacy activities at home have better opportunities for reading acquisition (Kim, 2009). Numerous studies point to the role of the home literacy environment in influencing early reading skills – in particular, children’s exposure to print 22%** 40%*** 34% 4%*** 36% 36% 20% 7% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Never Once during the week A few times during the week Every day Baseline Endline 39 | AIR.ORG LEARN Endline Evaluation Report materials at home and opportunities to engage in reading with other household members (Hess, & Holloway, 1984; Dowd, Pisani, & Borisava, 2016). As such, the survey included questions to measure the home literacy environment, which we have conceptualized as the reading habits of family members and their engagement with students’ readings, as reported by students. Specifically, we asked students if they saw anyone reading at home and whether anyone in their household had encouraged them to study, read to them, or told them a story. Overall, all measures of home literacy activities increased from baseline to endline (Exhibit 21). In the overall sample, at endline, 66% of students said someone in their household helped them with their studies; 60% reported that someone read to them; 56% reported that they saw someone reading, and 41% stated that someone told them a story. All four changes are statistically significant at the 1% level. There was large variation in the home literacy environment between counties (Exhibit E9 in Annex E). A larger proportion of students in Grand Gedeh and River Gee Counties reported literacy activities at home in each category, compared to the two other counties. This is an encouraging result, as these were the two counties that received literacy boost programming. Household literacy activities generally increased in all counties except for Rivercess County, which saw considerable decreases in three out of four measures of home literacy activity. The qualitative data agree with these overall findings of improvement in home literacy, with few students reporting they have ‘nobody’ to help them read, and parents reporting that they find ways to help even despite their own challenges with literacy (elaborated on in the Qualitative Discussion). Exhibit 21. Household Literacy Activities in the Past Week Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages does not add up to 100%. Does not include students who refused to answer or answered “did not know” Baseline: N = 1,358. Endline: N = 1,029. 41%*** 56%*** 60%*** 66%*** 35% 47% 50% 61% 0% 10% 20% 30% 40% 50% 60% 70% Told student a story Saw someone reading Read to student Helped with studies Baseline Endline 40 | AIR.ORG LEARN Endline Evaluation Report The survey asked students to report on the specific family member involved in the four activities listed in Exhibit 21. Most students reported that their older brother was the one they saw reading last week (39%), that he helped them study (34%), and that he read to them (32%), while the majority reported that their father told them stories (22%) followed by their mother (21%). At baseline, these proportions are fairly similar, with their older brother being the person most reported to be seen reading last week (35%), to help them study (38%), and to read to them (38%), while their mother and father were most likely to tell them a story (21% and 22%, respectively). The similarity in baseline and endline proportions were confirmed by t-tests across the two rounds for the most common family member performing the stated tasks, which were not statistically significant at the 10% level. No major differences were found by county except in Rivercess County, where mothers were far more likely to tell a child a story than any other family member. No major differences were found when disaggregated by sex. 3.2.1.3 Student Attitudes Toward Schooling To assess students’ perceptions of their education, we asked Grade 2 students the reasons they liked or disliked school. At baseline, a majority of Grade 2 students (70%) said they liked school because they found their lessons and learning enjoyable. Students in Rivercess were particularly enthusiastic about lessons and learning: 75% reported them as a reason they like school at baseline. Overall enjoyment of lessons and learning increased to 86% at endline (p < 0.01) and was high across all counties, ranging from 72% in Rivercess County to 91% in Grand Bassa. At baseline, only 1% to 2% of students reported liking school because food was provided, but this proportion jumped to 15% overall at endline, with 21% of students noting food as a reason for their enjoyment in River Gee and 43% in Rivercess (p < 0.01 for all). There was also an overall 6 percentage point increase in the proportion of students who responded that they enjoyed classroom games (p < 0.01). Qualitative data support these findings, in particular showing students’ enthusiasm for teachers who they perceive to be doing their job well, which was reported in the majority of cases. Students described specific methods that teachers used to teach literacy that they enjoyed, including using songs, games, and group reading activities. The main reasons students reported disliking school at endline were their teacher’s punishments or physical violence from teachers (25% overall) and other students teasing or fighting with them (25%). These proportions increased from 13% and 9%, respectively, at baseline (p < 0.01 for both). The proportion of students reporting disliking receiving punishment from their teacher increased in all counties at endline, except for River Gee, where the difference in mean values was not significant at the 10% level. The proportion increased in the other counties: from 15% to 33% in Grand Bassa (p < 0.01), from 5% to 19% in Grand Gedeh (p < 0.01), and from 11% to 32% in Rivercess (p < 0.01). The qualitative data supports the finding that students’ main complaint about school, from baseline, midline and endline, is around receiving various punishments from 41 | AIR.ORG LEARN Endline Evaluation Report teachers, including beating and being forced to do physical labor. Notably, there were no reports of beating spelling at endline, which was a significant issue raised at midline. Students in qualitative research also sometimes reported feeling as though their teachers were not focused on teaching or had poor attendance, which was disappointing to students who were eager to learn. Finally, students commonly complained about the cleanliness of school grounds, and lack of certain infrastructure or materials (e.g., school fence, water, desks). These topics are further elaborated upon in the Qualitative Discussion. 3.2.1.4 Presence of Teachers in School There is a positive association between teachers’ attendance and students’ achievement (Ahn & Vigdor, 2010; Miller, 2012; Woods, 1990). Thus, we asked Grade 2 students about the consistent presence of teachers in school. Overall, 86% of students at endline stated that teachers came to class every day, which was the same value as at baseline (Exhibit 22). The proportion at endline is lowest in Grand Bassa, at 83%, followed by 84% in Grand Gedeh, 91% in Rivercess, and 93% in River Gee. Teacher attendance did not differ noticeably by grade, as reported by students. These results are consistent with student responses about whether they saw a teacher fail to show up in the last week. Exhibit 22. Teacher Attendance Source: Student survey. Authors’ calculations. Results do not include students who refused to answer or answered “Do not know”. Baseline: N =1,279. Endline = N = 702 The proportion of students noting teacher tardiness remained relatively stable over time, with a slight improvement by endline, suggesting that the LEARN program did not substantially affect teacher attendance or tardiness (Exhibit 23). Across all counties, 76% of students at baseline and 79% of students at endline said their teacher never arrived late to class. Teacher tardiness was 86% 9% 1% 4% 86% 10% 0%** 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Every day (5 days) A few times (2-4 days) Once during the week Never Baseline Endline 42 | AIR.ORG LEARN Endline Evaluation Report reported slightly more frequently in Grand Bassa and Grand Gedeh than in the other two counties: no students at endline reported that teachers arrived late at least once per week in Rivercess and River Gee, compared to 9% and 5% of students in Grand Bassa and Grand Gedeh, respectively (Exhibit E11 in Annex E). Exhibit 23. Teacher Tardiness Student survey, AIR calculation. Results do not include students who refused to answer or answered “Do not know”. Baseline: N = 1203. Endline: N = 679 The results for both attendance and tardiness should be interpreted with caution for two main reasons: (a) the team measured these outcomes based on the self-reported responses of young children in Grade 2, and (b) the team collected data during the rainy season at baseline when road conditions could have influenced teachers’ attendance and tardiness. However, qualitative data supports the finding, as was seen at baseline and midline as well, that teachers are generally present, but there are sporadic cases in which teachers are late or miss classes, to students’ evident disappointment. As the Qualitative Discussion details, teacher’s grievances around pay and support from government contributes to low morale and in some cases, it becomes necessary for teachers to supplement their income elsewhere, therefore missing school. 3.2.1.5 Literacy Outcomes This section provides the findings from the LBRA collected from Grade 2 students. Consistent with baseline, to measure the literacy skills of Grade 2 students at the end of their grade level, AIR administered an adapted version of the LBRA. Since the official language of instruction in Liberia is English, enumerators administered all subtests in English. The adapted version of the LBRA used for this evaluation consists of four subtests: 4% 10% 11% 76% 5% 7%** 8% 79%* 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Every day (5 days) A few times (2-4 days) Once during the week Never Baseline Endline 43 | AIR.ORG LEARN Endline Evaluation Report  Letter knowledge: The number of letter sounds that the student could identify, out of 26.  Word recognition: The number of words, out of the 20 most-used words from leveled textbooks that the student could read correctly. Recognition is defined as the student’s ability to read the word.  Decoding (invented word recognition): The number of invented words, out of 20, that students could decode correctly.  Reading comprehension – Reading aloud: Using a short story of 155 words, we assessed: » Fluency: The number of words read correctly in a minute » Accuracy: The percentage of words read correctly (untimed) – Comprehension: Ten comprehension questions related to the short story were asked orally in one of three conditions: » Reading comprehension, which applied to children who could read at least five words in the story correctly in 30 seconds. These children were identified as “readers.” » Listening comprehension, which applied to children who could not read five words in the story correctly in 30 seconds. The enumerator read the story aloud to these children, identified as “non-readers.” » Listening comprehension for “readers,” which applied to students who read at least five words correctly but gave up before attempting a significant portion of the passage or could not finish the passage. The enumerator read the rest of the story to them. Exhibit 24 shows a summary of Grade 2 students’ literacy skills at baseline and endline. Whereas only 1% of students could read and comprehend at baseline, that number increased to 5% by endline (p < 0.01). When disaggregating this outcome by reader status, we find that reading comprehension improved significantly from 35% to 61% of readers (p < 0.05) and accuracy improved significantly from 10% to 50% of readers (p < 0.01). However, we find a decrease in the proportion of readers from 17% to 11% (p < 0.01). We also find declines in other literacy skills over time, namely: listening comprehension, invented words, most used word recognition as well as knowledge of the letters of the alphabet. Overall, our results show a mixed pattern of findings. One explanation for the mixed pattern may be due to the interruption of schooling by the COVID pandemic and subsequent learning losses. 44 | AIR.ORG LEARN Endline Evaluation Report Exhibit 24. Grade 2 Students’ Literacy Skills Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline N = 758; Endline N = 1,029. There were 117 readers at endline; 127 readers at baseline. Letter Knowledge To assess students’ letter knowledge, enumerators showed students a chart of 26 letters in English and asked them to identify the sound of each letter. At endline, most students (83%) could identify 21 to 26 letters (Exhibit 25) which represents a decrease in letter knowledge from 90% at baseline (p < 0.01). This finding could be due to learning loss from the COVID pandemic. Across all counties, except for Rivercess, the average percentage of letters identified by students decreased. There was also a statistically significant decrease of 5 percentage points for girls and 11 percentage points for boys in the proportion of students that could identify at least 90% of the letters(p < 0.01), pointing to decreasing outcomes in this category across both boys and girls. 85%*** 23%*** 2%*** 11%*** 50%*** 61%** 15%** 14%** 5%*** 90% 38% 4% 17% 10% 35% 20% 25% 1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Letter Knowledge (% correct) Most used word recognition (% correct) Invented word recognition (% correct) Students classified as readers (5+ words correct in 30 seconds) Accuracy (% words correct in passage), readers only Proportion of reading comprehension questions correct (%), readers only Listening comprehension passed (%), non-readers only Listening comprehension passed (%), readers only % of students who can read and comprehend out of the total sample Baseline Endline 45 | AIR.ORG LEARN Endline Evaluation Report Exhibit 25. Letter Knowledge Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,029 at endline; N = 758 at baseline. Reading Skills and Reading Comprehension Similar to baseline, we classified students as readers if they could read at least 5 words in the LBRA story in 30 seconds. We found a statistically significant (p < 0.01) decrease in the proportion of students who could read at endline (11%) compared to baseline (17%). Grand Bassa had the highest percentage of readers with 16% at endline, followed by Rivercess with 11%, Grand Gedeh with 8%, and River Gee with 7% (Exhibit 26). The proportion of readers decreased the most in Grand Gedeh (from 22% to 8%) followed by River Gee (from 17% to 7%); the decline in Rivercess and the increase in Grand Bassa (from 14% to 16%) were not significant. Exhibit 26. Proportion of Readers, by County Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,029 at endline; N = 758 at baseline. 0% 0% 1% 1% 6% 90% 1% 2%*** 2% 3%** 9%** 83%*** 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% None 1-5 Letters 6-10 Letters 11-15 Letters 16-20 Letters 21-26 Letters Baseline Endline 14% 22% 17% 17% 17% 16% 8%*** 11% 7%*** 11%*** 0% 5% 10% 15% 20% 25% Grand Bassa Grand Gedeh Rivercess River Gee Overall Baseline Endline 46 | AIR.ORG LEARN Endline Evaluation Report We used the same LBRA passage to measure students’ fluency and accuracy. Fluency of readers increased from baseline (11 words/minute) to endline (24 words/minute), a significant difference at the 1% level. Students’ accuracy also increased from 59% of the time reading words correctly at baseline to 73% of the time at endline. Accuracy was highest for students in Grand Bassa (75%) and Rivercess (74%) even though these counties did not receive the literacy boost package. After readers read or non-readers listened to the whole passage, enumerators asked 10 comprehension questions:  Summary: One question that tests students’ ability to identify the main ideas of a reading passage.  Literal: Five questions in which the answer is clearly and explicitly stated in the passage.  Inferential: Three questions in which the answers are implied, rather than clearly stated in the passage.  Evaluative: One question that requires some level of cognitive and/or emotional judgment. To answer such a question, a child needs to use his/her personal opinion. Competency on the assessment is defined as the ability to answer at least 80% of the comprehension questions correctly. In general, readers were more successful than listeners (e.g., non-readers and readers who did not finish the passage) on the comprehension questions. Sixty￾one percent of readers (46 out of 75 eligible students) and just 14% of listeners (148 out of 1,026 eligible students) answered at least 80% of the comprehension questions correctly at endline. For readers, there was a statistically significant increase in the proportion of readers who were able to answer at least 80% of the comprehension questions correctly between baseline and endline (35% vs. 61%). The results for readers should be read with caution due to the low sample size of readers across baseline and endline. For listeners, there was a statistically significant decline in the proportion of listeners who were able to answer at least 80% of the comprehension questions correctly between baseline and endline (20% vs. 14%). Among the listeners, the decline was exhibited for both non-readers and for readers who did not finish the passage. In fact, 15% of non-readers and 14% of readers met the 80% competency standard at endline, down from 20% and 25%, respectively, at baseline (p < 0.05 for both). This outcome is likely driven by the overall reduction in the number of listeners from baseline to endline. We also analyzed the comprehension results by question types as shown in Exhibit 27. The proportion of students passing each type of question has remained broadly constant, except for evaluative questions that decreased from baseline from 72 to 55 percentage points (p < 0.01). 47 | AIR.ORG LEARN Endline Evaluation Report Exhibit 27. Comprehension Subtests Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. At endline, N = 1,029; Baseline. N = 758. We also examined changes in reading with comprehension over time. Specifically, we assessed the proportion of students classified as readers who were able to answer 80% of the comprehension questions correctly after reading the text. We find that there was a 4 percentage point increase in reading comprehension among boys at endline compared to baseline. In contrast for girls, there was a smaller increase of 2.5 percentage points over time. However, the 1.5 percentage point “differences-in differences” we see in reading comprehension among boys over time, relative to girls over time, are not statistically significant. Annex E shows further differences by county and by gender. 3.2.2 Other Key Student Outcomes This section describes the changes in key outcome indicators pertaining to hygiene and handwashing, nutrition, SRGBV and gender norms, as well as disability. The enumerators asked questions about these topics of both Grades 2 and 6 students, except for questions about gender norms, from which Grade 2 students were excluded. 3.2.2.1 Hygiene and Handwashing Practices To capture information on hygiene practices, enumerators first asked students whether they had washed their hands at all in the day prior to the survey, and with what. Most students (92%) reported that they had washed their hands, with slight differences by sex (93 and 91% for boys and girls, respectively) and no differences by grade. By county, this outcome was slightly more common in River Gee (96%) and Grand Gedeh (93%) than in Grand Bassa (90%) and Rivercess (89%) at endline. For a deeper understanding of students’ knowledge and handwashing practices, we also developed questions to compare students’ knowledge of appropriate handwashing behavior to their actual conduct. Survey questions focused on handwashing at three critical moments: (a) after using the toilet to defecate, (b) after using the toilet to urinate, and (c) before eating food. 75% 72% 34% 13% 73% 55%*** 37% 14% 0% 20% 40% 60% 80% Inferential Evaluative Literal Summary Baseline Endline 48 | AIR.ORG LEARN Endline Evaluation Report Exhibit 28 shows knowledge and behaviors relating to critical handwashing. Overall, handwashing knowledge increased by 1 percentage point from baseline to endline. While we see significant improvements in Grand Bassa and Grand Gedeh, knowledge levels among students in River Gee and Rivercess witnessed a statistically significant decrease relative to baseline (see results by county in Exhibit E13 in Annex E). The data also show that self-reported critical handwashing behaviors increased by 6 percentage points (p < 0.10), with statistically significant increases in Grand Bassa and Grand Gedeh (p < 0.10) and decreases in Rivercess and River Gee. Across all the counties, Grand Gedeh had the largest proportion of students with handwashing knowledge (27%) and recommended handwashing practices (23%). Exhibit 28. Student Knowledge Versus Practice of Critical Handwashing Moments Student survey, AIR calculation. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline: N = 2,021. Endline: N = 1,655 3.2.2.2 Nutrition Practices and Knowledge Practices. We asked Grade 2 and 6 students how frequently they eat each day. Students consume two to three meals daily, on average (Exhibit 29). Statistically significant differences at the 1% level emerged for the share of students eating three or more meals daily, which increased from baseline, and for the share of students eating one daily meal, which decreased relative to baseline. Negligible, though statistically significant, differences emerged when the data were disaggregated by sex or county. Across counties, students eating three meals a day was higher in River Gee (34%) and Grand Gedeh (32%) relative to Rivercess (29%) and Grand Bassa (22%). While no statistically significant differences emerged by gender for students who reported eating three or more meals per day, such differences emerge for those reporting they only eat one meal per day. Boys (11%) more often reported as such rather than girls (9%), representing a small yet statistically significant difference at the 10% level. 15%*** 22%** 9% 19% 0% 5% 10% 15% 20% 25% Handwashing knowledge Handwashing self-reported behavior Baseline Endline 49 | AIR.ORG LEARN Endline Evaluation Report Exhibit 29. Number of Times a Student Eats per Day Student survey, AIR calculation. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline: N = 2,021. Endline: N = 1,655 Knowledge. To determine whether students could identify the components of a healthy diet, the survey asked students to identify the three components of a balanced diet, defined as go, glow, and grow foods. Based on the terminology SC used to train students, “Go” foods are defined as foods that give one energy to play and learn; “Glow” foods are defined as foods that protect one’s body from disease; and “Grow” food are defined as foods that help body grow. The nutrition knowledge remained low similar to baseline. Only 3% of students (48 students out of 1,645) stated that they knew the definition of a balanced diet at endline, and, of those, 1% (9 students) could successfully identify all three components of a healthy diet. We also asked students how they thought food should be divided between boys and girls, whether they thought that one sex should get more, or that the food should be divided equally. At baseline, overall, 46% of students felt that food should be divided equally, though 31% responded that boys should get more food, and 19% said that girls should get more food. At endline, the proportion of students responding that food should be shared equally increased to 63% (p < 0.01), while the proportion reporting that one sex or the other should receive more decreased to 21% for boys and 15% for girls (p < 0.01 for both). Students in River Gee had the greatest increase in the proportion of students reporting that food should be shared equally, with a 35 percentage point increase from baseline to endline (42% to 77%, p < 0.01). The proportion of students responding that food should be shared equally in the other counties also increased significantly from baseline to endline (p < 0.01). 0% 13% 3% 58% 24% 2% 0% 7%*** 1%*** 56% 33%*** 4%*** 0% 10% 20% 30% 40% 50% 60% 70% I eat once per day and sometimes not at all Once per day Sometimes two times, sometimes one time per day Twice per day Three times per day More than three times per day Baseline Endline 50 | AIR.ORG LEARN Endline Evaluation Report 3.2.2.3 Sexual and Gender-Based Violence and Gender Norms Background. At midline, there had been little work done on revising the school codes of conduct, nor on providing refresher training to school staff or students on the contents of the existing code of conduct. The qualitative data at baseline and midline did show that students and teachers did have a good general idea about what sorts of behavior were not acceptable (e.g., corporal punishment / beating, physical labor as punishment, teachers having inappropriate relationships with students), but there was an evident degree to which those guidelines were not actually followed and, for some, outrightly disagreed with, particularly around corporal punishment. At endline, the revised TCOC was nearing finalization and drafts had been disseminated to local level Ministry of Education actors. Prior to the completion of the revised TCOC, the project devised a set of ‘Safe School Stories’. SC utilized a social-behavior change approach in tasking community mobilizers with sharing these Safe Schools Stories with teachers, parents, and students independently. Each of these stories had a lesson to teach about the context and rationale for parts of the TCOC, raise awareness about the TCOC, and promote appropriate teacher behavior, and parent or student response to violations. Not all communities were expected to have been reached at the time of endline data collection. As part of this re-introduction, Rules/Code of Conduct for Teachers Knowledge. Overall, a majority of students (78%) said that rules existed for how teachers should treat students at school, similar to baseline (79%). Data show large differences by county. While 98% of students in Rivercess and 95% of students in River Gee said rules exist at endline, only 76% in Grand Gedeh and 64% in Grand Bassa noted the same. Exhibit 30 shows the changes in student knowledge of their own code of conduct by county and round, which demonstrates statistically significant increases in student knowledge of the code of conduct relative to baseline across all counties. These increases are particularly noteworthy among students in Grand Gedeh, Rivercess, and River Gee (p < 0.01). Students tended to know that rules prevented teachers from physically harming students. Differences by sex and region were negligible. As Exhibit 31 shows, 49% of students at baseline and 87% at endline stated that teachers were not allowed to beat students, a statistically significant increase at the 1% level. While less than a third said teachers are not allowed to use humiliating language on students at endline, this represents a statistically significant increase from baseline at the 1% level. 51 | AIR.ORG LEARN Endline Evaluation Report Exhibit 30. Student Knowledge of Code of Conduct, by County Student survey, AIR calculation. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline: N = 2,021. Endline: N = 1,539 Similar to midline, the qualitative data at endline agree that students knew of a code of conduct in the school and were able to correctly name many of its components. These components include: teachers and staff must not partake in bribery, abuse, and rape, must not have relationships with students, commit corporal punishment, discriminate by gender, discriminate against children with disabilities, commit fraud, use humiliating language/lack of respect or practice favoritism, use drugs and alcohol, or have persistent absences. The qualitative data also showed that teacher corporal punishment continued, with students across communities reporting it occurring to boys and girls. However, encouragingly given the attention SC gave to the matter, there were no reports of beating spelling – a common report at midline – taking place in any of the schools Exhibit 31. Student Identification of Rules to Guide Teacher Behavior Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Baseline: N = 1,816. Endline: N = 1,539. 69% 63% 73% 66% 68% 64%* 76%*** 98%*** 95%*** 78%*** 0% 20% 40% 60% 80% 100% 120% Grand Bassa Grand Gedeh Rivercess River Gee Overall Baseline Endline 22%*** 32%*** 29%*** 87%*** 14% 22% 22% 49% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Teachers are not allowed to ask students for money Teachers are not allowed to be in a relationship with students Teachers are not allowed to use humiliating language on students Teachers are not allowed to beat students Baseline Endline 52 | AIR.ORG LEARN Endline Evaluation Report At endline, students were also asked whether they knew of any other general rules for teachers in school. Approximately 42% of students said ‘yes’ with the lowest proportion in Grand Bassa (24%) and the largest in Rivercess (86%). For those who mentioned additional rules, the most reported rule was that teachers are not allowed to come to school drunk or high on drugs (64%). Forty-eight percent of students said that teachers should not steal from school, while 30% noted that teachers were not allowed to arrive to school late or leave early. Only 16% of students responded that teachers are not allowed to fail to show up to school. Students reported that they typically heard of these other rules from their head teachers or principals (54%) followed by the teachers themselves (29%). Comparatively fewer students reported hearing about these rules from their parents or other students (13% and 10%, respectively). Rules in Practice. We asked students whether they ever heard of a teacher lying to get what they want or stealing things from school, a teacher offering money to get what they want, a teacher making a comment about a student’s body inappropriately,27 a teacher touching a child inappropriately, a teacher coming to school drunk or high on drugs, a teacher teasing or calling children names, or a teacher treating one student better than the others. Fewer than 10% of students at endline reported hearing of a teacher performing these acts individually. However, 10% of students at endline reported a teacher failed to show up at school in the last week. These results are consistent with midterm findings, suggesting that teachers largely abide by their rules of conduct as reported by students. Reports of Disciplinary Practices. When asked about discipline at school at endline, 44% of boys and 52% of girls said that teachers forced them to clean or work at school if they behaved poorly; 41% of boys and 38% of girls reported being given extra work; and 29% of boys and 24% of girls reported physical violence. These results contrastslightly from those reported at baseline, where 55% of students said that teachers forced them to clean or work at school; 27% reported being given extra work; and 41% reported physical violence. 28 At endline, there is some variability in the disciplinary actions taken by county. In Grand Bassa and Rivercess, being forced to clean or work was the most common response for boys and girls at endline whereas being given extra work was the most common disciplinary action reported for boys in Grand Gedeh and River Gee at endline. Among girls, being forced to clean or work was the most common response at endline apart from River Gee where being given extra assignments was slightly more common. 27 Inappropriately here is defined as discussing or touching a child’s front part, behind part, or chest part. 28 Note: comparisons between baseline and endline for these outcomes is challenging as the questions were not asked by gender at baseline. 53 | AIR.ORG LEARN Endline Evaluation Report The team also asked students whether they believed that students were afraid to go to school for fear of being punished by teachers. Regardless of gender, most students reported that their peers were never or rarely afraid to go to school for fear of punishment. Nine percent of students reported witnessing a teacher using corporal punishment in the last week at endline. The largest proportion of students reporting witnessing the use of corporal punishment by a teacher was 12% in River Gee, and the lowest proportion was 3% in Rivercess. On average, boys and girls were reported as being equally likely to receive corporal punishment in the last week (2 times), and there was also no difference in the reporting regarding the number of times students heard about a teacher teasing a boy or girl student in the last week (2 times). Willingness to Report. The enumerators asked students about their knowledge of actions to take if they are teased or touched at school in a way that they dislike (which left open-ended the committer of the act—a teacher, another child, an administrator, etc.). Among those that said they would report these incidences (100%), a large majority of students at endline (76%) reported that they would speak to their teacher, followed by 51% who said that they would go to the principal or registrar. In contrast, at baseline the proportions were 89% reporting that they would speak to their teacher, and 31% would speak to the principal or registrar (p < 0.01 for both). A Proxy for Willingness to Report SRGBV. Synthesizing the above information to create an index for SRGBV knowledge and practice, we developed three measurements to indirectly gauge students’ willingness and ability to report SRGBV incidents: (a) proportion of students who understand school rules and codes of conduct, (b) proportion of students who indicated that they would report cases of bad behavior, and (c) proportion of students who reported any type of corporal or psychological teacher discipline. We considered students to be knowledgeable about codes of conduct if they reported that rules exist to guide teachers’ behavior and could describe at least one of these rules to the enumerator. Across the sample, 78% of students (an increase from 68% at baseline) stated that their schools had a code of conduct regulating teachers’ behavior. Nearly all students stated this in Rivercess (98%) and River Gee (95%), while proportions were also high for students in Grand Gedeh (76%) and Grand Bassa (64%). There was no statistically significant difference in knowledge of the code of conduct by sex at endline. We considered students to be willing to report SRGBV incidents (defined in the survey as being teased or touched in an uncomfortable way) if they could identify the person that they would speak to in such cases. Although knowing whom to contact does not guarantee that the student would report an incident, the survey could not ask students directly if they would report an incident because of the sensitivity of the topic. We therefore assume that students who could readily name a contact person might be inclined to report SRGBV incidents and used this 54 | AIR.ORG LEARN Endline Evaluation Report information as a proxy for willingness to report. In this context, like baseline (98%), all students in our sample were willing to report inappropriate situations at school that they witnessed or in which they were personally involved (p < 0.01). No differences were found by county, grade, or sex. These large proportions of students stating their willingness to report is likely a byproduct of social desirability bias rather than true responses. However, Tourangeau and Yan (2007) argue that students would be more likely to report incidences if they are less/not stigmatized or sensitive (for the child). While social desirability may still be at play, this dimension could also potentially explain the high willingness to report. A thorough analysis of students’ willingness to report would require a separate, rigorous study. The qualitative data show that most students, as at baseline and midline, were able to provide a list of rules and regulations that teachers were expected to follow and explained the process that was to occur in case of a violation. At midline, few students reported fear of retribution upon reporting an infraction; however, at endline, this claim was relatively more common. Even among those not fearing retribution, a concern shared by students was that principals would not punish teachers, who were already in such short supply. This topic is further elaborated upon in the Qualitative Discussion. For the third measurement in our index of willingness to report, we analyzed students’ responses to questions regarding teachers’ disciplinary practices. We considered teachers as having engaged in corporal or psychological punishment if students reported them as having taken part in any of the disciplinary tactics. By this definition, across the board, regardless of sex, grade, or county, more than 99% of students stated that their teachers used some form of corporal or psychological punishment. As the proportion of students stating that their teachers used corporal or psychological punishment was already high a baseline, we see no significant changes in this outcome at endline. Our index suggests that students in all grades would willingly report inappropriate teasing or touching in school. However, given that teachers regularly disciplined students using corporal or psychological disciplinary strategies and that most students are not aware that their teachers are subject to a code of conduct, there is cause for concern. Such knowledge of a code of conduct does not guarantee that students would report their teachers should they violate the code. However, this knowledge regarding guidelines could influence students’ perceptions of the school climate and permissible behavior. Studies have shown that students feel more encouraged to share their thoughts if they hold positive perceptions of their relationships with their teachers in the classroom, and such sentiments may depend on whether their teachers abide by a code of conduct. 55 | AIR.ORG LEARN Endline Evaluation Report Gender Norms. To obtain information on students’ perceptions of gender norms, we asked Grade 6 students only whether they agreed or disagreed with a series of five statements (Exhibit 32). Exhibit 32. Student Perceptions of Gender Norms, by Gender Statement Disagreed Boys Girls Total Base End Base End Base End If a boy touches a girl at school, it’s because the girl did something to attract him. 70% 67% 78% 79% 73% 73% There are times when a boy needs to beat his girlfriend. 36% 31% 28% 28% 33% 29% Girls like to be teased by boys. 48% 63%*** 61% 76%*** 54% 70%*** When girls wear short skirts, they are telling boys or men to touch them. 50% 68%*** 47% 68%*** 49% 68%*** For girls to get good grades, they sometimes have to let their teachers touch them or love them. 69% 78%*** 79% 85%** 73% 82%*** Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Endline: N = 295 for boys and 329 for girls. Baseline N = 364 for boys and 285 for girls. Next, we defined a threshold for students holding less biased and better perceptions of gender norms if they disagreed with at least four of the five statements on gender stereotypes listed above. Sixty percent of students reached the threshold at endline compared to only 48% at baseline, showing significant improvement in students’ perceptions of gender norms (Exhibit 33). There were significant county variations: a much lower percentage of students in Grand Bassa disagreed with at least four out of five gender norms statements (48%) compared to the other counties, especially River Gee (72%). These results should be interpreted with caution due to the social desirability bias inherent in self-reported responses to such questions, especially in Liberia where this subject is a known challenge (Parkes, 2016). 56 | AIR.ORG LEARN Endline Evaluation Report Exhibit 33. Students Who Disagreed with Four out of Five Gender Norms, by County Source: Student survey. Authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. At baseline, N = 649. At endline, N = 625. Disability We assessed visual, auditory, or physical impairments that may impede students’ ability to learn in the classroom by asking students the same short set of questions from the Washington Group Questions that we asked at baseline. These questions reflect current thinking and measurement of child functioning.Although this leading conceptual framework assesses a multitude of areas in which children may experience functional difficulties, in our survey, we asked students only questions related to difficulties in seeing, hearing, or walking. We directed these questions to students in both Grades 2 and 6. Our results showed that, across grades, sex, and counties, most children (more than 95% at baseline and endline) reported that they did not have any kinds of disability in these domains. We did not find any notable differences by sex, grade, or county or by round. However, given stigma around disability, students may be under reporting their disabilities. It may also be possible that students are not fully aware of their disabilities. Future research needs to carefully study measurement issues in assessing disability patterns in students. 43% 61% 48% 49% 50% 48% 66% 55% 72%*** 60%*** 0% 20% 40% 60% 80% Grand Bassa Grand Gedeh Rivercess River Gee Overall Baseline Endline 57 | AIR.ORG LEARN Endline Evaluation Report 4. Impact Evaluation Findings This section presents the findings from the quantitative impact analyses enriched by contextual information from the qualitative data. First, we describe the sample of Grade 2 level students in all 54 active schools in Grand Gedeh who completed the student survey and the LBRA at both baseline and endline. Then, we present the results of the main outcomes of interest for the impact evaluation, i.e., literacy outcomes (Section 4.2.1) and health and nutrition outcomes (Section 4.2.2). 4.1 Impact Evaluation Samples This section presents the composition of schools and distribution of students across the three evaluation arms: school feeding, school feeding, literacy boost, and school health and nutrition, and comparison schools. We provide descriptive statistics of key student and household characteristics (as reported in the student survey). We report information separately by the three groups of schools (treatment and comparison groups) and indicate whether the differences between each of the treatment arms (school feeding or school feeding and literacy boost) and comparison schools are statistically significant at endline. These equivalence checks help verify and comparison for the differences between each of the treatment arms and comparison group in the regression analysis. 4.1.1 School Compositions The project evaluation sample includes schools in four counties, but the impact evaluation sample includes only schools in Grand Gedeh. As explained above, based on the geographic location of each school in Grand Gedeh, we created 18 clusters with the 54 schools. Then, we randomly assigned all 18 clusters to 3 groups – 2 treatment and 1 comparison group. Schools in each treatment group received different LEARN interventions (Exhibit 34). 58 | AIR.ORG LEARN Endline Evaluation Report Exhibit 34. Treatment Interventions Treatment Group 1 SF only Treatment Group 2 Combination of SF + LB + SHN Comparison Group Business as usual  Provide school meals  Provide THRs for girls (Grades 4- 6)a  Distribute deworming medications, vitamins, and minerals  Institute teacher recognition  Build/rehabilitate storerooms, kitchens, stoves, latrines  Establish PTAs  Provide training on PTAs, food preparation & storage, good health & nutrition, commodity management  SF package in full  LB package, including: – Establish activities to promote literacy – Train teachers to lead Reading Camps – Establish libraries – Produce books & reading materials – Promote increase in community awareness on SRGBV  SHN package, including – Establish school gardens  Improve health and nutrition practices by training teachers to lead SHCs Schools in this group will not receive either of the packages and serve as a comparison group for the project’s impact evaluation Source: SC TOR. a During the COVID school closures, LEARN distributed the school meals in the form of THRs to all students, unconditional on attendance. As Exhibit 35 shows, 22 schools were assigned to the school feeding treatment group, 20 schools to the school feeding, literacy boost, and school health and nutrition group, and 12 to the comparison group with no intervention. In an attempt to address the smaller-than-anticipated number of students to survey (either because of inflated enrollment rates or higher absence rates), the evaluation team oversampled from some of the larger schools. In replacing students, we kept the sex ratio balanced to the extent that the sex-ratio of present students allowed.29 In total, we surveyed 721 students across the three groups in Grand Gedeh. Parents provided written consent for the children in their school. We also asked students for their oral assent; none of the students refused to participate in the survey. Exhibit 35. Numbers of Schools and Students in the Impact Sample Unit Students Surveyed at Baseline Students Surveyed at Endline SF only SF+LB+SHN Comparison SF only SF+LB+SHN Comparison Sampled schools 22 20 13 22 20 12* Surveyed students 213 280 188 244 321 156 Overall 681 721 Source: Student survey. Authors’ calculations. One of the schools did not have any students to sample at the time of visit which reduced the number of schools from 13 to 12. 29 If a girl was not available at follow-up, we randomly selected another girl from present students. However, if there were not enough girls available on the day of the school visit, we replaced her with a boy in the same class to maintain the power. 59 | AIR.ORG LEARN Endline Evaluation Report 4.1.2 Student Compositions and Characteristics Sample Compositions. Though the average age of the Grade 2 students was similar across both treatment arms and the comparison group at about 11.5–12 years old, the range of ages varied slightly (there were more significantly older students in both treatment groups). The endline sample consisted of 46% girls and 54% boys (Exhibit 36). Exhibit 36. Composition of Students’ Sex in the Impact Sample at Endline Treatment/Comparison Condition Boys Girls Percent Number Percent Number Overall SF 56% 136 44% 108 244 LB+SF+SHN 53% 169 47% 152 321 Comparison 54% 84 46% 72 156 Overall 54% 389 46% 332 721 Source: Student survey. Authors’ calculations. Main Language Spoken at Home. When we compared the main language spoken at home across the treatment and comparison groups, we found that most households speak English at home. The data revealed no major differences, except for a slightly lower percentage of households in the comparison group speaking English and a higher percentage speaking Krahn. Household Size. Each treatment group was similar in household size with, on average, a little over seven people, which was not significantly different compared to the comparison group. Socioeconomic Status. To obtain a better understanding of students’ socioeconomic status across each group, we examined the differences between each treatment arm and the comparison group on the household’s possession of eight durable goods, including cell phone, electricity, icebox, bicycle, TV, motorbike, or a tuk-tuk. Students reported owning an average of about two of these goods. The differences between treatment groups were negligible. Enrollment and Grade Repetition. As the key performance indicators focus on students who received the program for at least 2 years, we examine differences in students’ reported grade repetition and movement into the treatment schools. Students in school feeding, literacy boost, and school health and nutrition were less likely to report ever repeating any grades (21.5%) compared to students in school feeding (22.5%) and comparison schools (29%). Of those who repeated a grade, Exhibit 37 shows the proportion of students repeating Kindergarten and Grade 1 by treatment group. Boys and girls are equally likely to report repeating any grade (23%) and those that reported repeating any grade are similarly likely to report repeating Kindergarten (45%) or Grade 1 (42%). 60 | AIR.ORG LEARN Endline Evaluation Report Exhibit 37. Grade Repetition Source: Student survey. Authors’ calculations. 4.2 Impact Evaluation Results We used a DID method to estimate the effect of the school feeding intervention and the combined effect of school feeding and literacy boost and school health and nutrition on improvements in reading with comprehension, letter recognition, handwashing knowledge and behaviors, and nutrition knowledge. In addition, we study effects on being a reader. We present the results from the specified regressions for the base package (school feeding), the combined package (school feeding, literacy boost, and school health and nutrition), and the added benefit of the additional interventions (literacy boost and school health and nutrition) below. We also present results for being part of any treatment/intervention (base or combined package arms). This allows us to have a larger sample to estimate if any of the intervention activities (as a bundle) had an impact. We also present the results from the subgroup analyses assessing the differential impacts of the packages by sex for each key outcome.30 We structured each exhibit in this section as follows: each row presents the key program effects on main confirmatory outcomes (e.g., letter knowledge, reading with comprehension, handwashing behaviors and nutrition knowledge). The first three rows present results for overall sample, followed by results disaggregated by gender. The first column presentstreatment effects for combined package compared to control group; the second column shows treatment effects for base package compared to control group; the third column shows treatment effects for the combined package and the base package (bundled together) compared to control group; and the fourth column shows the additional (marginal) treatment effects for combined package compared to the base package group. Exhibit 38 shows effects on literacy outcomes, while Exhibit 39 shows the effects on health and nutrition outcomes. In Annex E (Tables E15 and E16) we report our estimates in standardized effect sizes. 30 We also conducted additional robustness checks looking at differential impacts by language spoken at home, by wealth, caregiver’s schooling or student’s age. None of these additional robustness exercises yielded any meaningful differences by subgroup so we do not show these results. We also considered the use of a variable that tracks duration for which students have stayed in their current schools. However, the sample size of non-missing responses for this variable was very small (~30), so we chose not to disaggregate analyses further. 49% 44% 55% 38% 35% 43% 0% 10% 20% 30% 40% 50% 60% Kindergarten Grade 1 Kindergarten Grade 1 Kindergarten Grade 1 Comparison SF SF+LB+SHN 61 | AIR.ORG LEARN Endline Evaluation Report All estimates control for sex, age in years (and age squared to allow for non-linear effects of age), an indicator for the main language being English, dummies for time (endline vs baseline), relevant treatment groups as well as total number of assets. For robustness, we tried estimates without any additional covariates, beyond adding dummies for treatment/intervention and time (required to estimate DID model). Estimates for regressions without controls give similar findings and are not reported. Adding different covariates such as student attendance or index of child friendly reading material does not change our results either. 4.2.1 Reading Outcomes This section discusses the effects of the school feeding and school feeding, literacy boost, and school health and nutrition packages, and the added value of the combined package to students’ reading with comprehension skills (i.e., students who can read 5 more words or more correctly in 30 seconds and respond to 80% of the comprehension questions). Exhibit 38. Overall Impact of LEARN on Key Learning Outcomes, by Treatment and Gender Outcomes Combined Package vs Control Group Base Package vs Control Group Combined And Base Package vs Control Group Combined vs Base Package Reading comprehension .034** (.016) .012 (.009) .024** (.011) .021 (.018) Knowledge of letters -.053 (.036) .061 (.111) -.016 (.098) -.14* (.077) Reader .113 (.094) .048 (.086) .087 (.079) .063 (.096) Observations 945 801 1402 1058 Girls Reading comprehension .006 (.006) .018 (.012) .01* (.006) -.013 (.015) Knowledge of letters -.067 (.13) .104 (.133) .001 (.124) -.167* (.089) Reader .04 (0.127) .060 (0.121) .051 (0.112) -.12 (0.111) Observations 448 368 656 496 Boys Reading comprehension .063** (.028) .004 (.008) .04** (.019) .048* (.025) Knowledge of letters -.078 (.113) .029 (.13) -.033 (.11) -.121 (.099) Reader .182* (0.100) .029(0.083) .119 (0.083) .129 (0.090) Observations 497 433 746 562 Source: Student survey; authors’ calculations; * p < 0.1 ** p < 0.05 *** p < 0.01; Standard errors shown in parentheses are clustered at the school level. Only key DID (Treatment X Post) key impact estimate are shown for relevant treatment and comparison/control groups and relevant outcomes. We first show estimates for overall sample, and then we disaggregate results by gender. We control for dummies for time (endline vs baseline), relevant treatment groups as well as age, age squared, gender, total number of assets and English as main spoken language at home. 4.2.1.1 Readers with Comprehension Students in schools that receive the combined package are 3.4 percentage points more likely than students in comparison schools to read with comprehension (p < 0.05). In contrast, we do not find any similar impact on reading comprehension skills for students in the base package 62 | AIR.ORG LEARN Endline Evaluation Report (school feeding only) schools compared to comparison schools. When we examine effects on any of the intervention schools (base or combined package), we find a 2.4 percentage point increase in reading comprehension relative to control group. The muted effects of the combined and the base package interventions seem to be driven by the null effects for the base package schools. Subgroup Analysis. Overall, boys seem to improve more than girls in reading comprehension. Boys in schools that receive the combined package schools are 6.3 percentage points more likely than boys in comparison schools to read with comprehension (p < 0.05). In contrast, we do not find any similar impact on reading comprehension skills for girls who receive the combined package. When we examine effects on any of the intervention schools (base or combined package), we find a 1 percentage point increase in reading comprehension relative to control group for girls, but a 4-percentage point increase for boys. We find null effects for the base package schools for boys and girls, though in terms of magnitude girls seem to benefit more than boys from the base package. Comparison of Treatment Arms. The estimated effects on reading comprehension appear to be larger in magnitude for students in the combined package schools compared to the base package schools. However, we only find a statistically significant value-added effect of the combined package over the base package for boys. For girls, the base package seems to have larger effects than combined package, though effects are not statistically significant. 4.2.1.2 Readers To investigate what explains the patterns in reader comprehension scores, we estimated effects on being able to read, i.e., students who can read 5 more words or more correctly in 30 seconds. Students in schools that received the combined package schools were 11.3 percentage points more likely than students in the comparison schools to be readers, but these effects are not statistically significant. We also do not find any statistically significant impact for alternative treatment group comparison. Subgroup Analysis. The overall intervention improved reading for boys, but not girls. Boys in schools that received the combined package were 18.2 percentage points more likely than boys in the comparison schools to be readers (p < 0.10). We find null effects for other intervention/treatment comparisons (e.g., for the base package only) Comparison of Treatment Arms. The estimated effects on reading were larger in magnitude for students in the combined package schools compared to the base package schools, but for boys only. However, we don’t find statistically significant differences between students in the base package schools compared to combined package schools, either for overall sample or for boys/girls sample. 63 | AIR.ORG LEARN Endline Evaluation Report 4.2.1.3 Letter Knowledge We considered students who could name at least 90% of the 26 letters of the English alphabet as “letter knowledgeable.”Error! Reference source not found. Results in Exhibit 38 show that exposure to neither the base package nor the combined package increased Grade 2 students’ ability to identify letters. Subgroup Analysis. We do not find any significant effects of either treatment on letter recognition for boys or girls. This finding is consistent with findings from midline analysis. Comparison of the Treatment Arms. Lastly, we examine the added value of the additional program components: the literacy boost and school health and nutrition interventions on letter knowledge. We find that students in schools receiving the full package performed worse on letter recognition tasks compared to students in schools receiving the school feeding package only (-14 percentage points, p < 0.10). Furthermore, we find that girls in schools that received combined package do worse than girls in schools with the base package only (-16 percentage points, p < 0.10), but there is no statistically significant effects on letter knowledge for boys in the combined vs. the base package schools. The results comparing the combined and the base packages are somewhat surprising as we generally find the combined package to do better for reading comprehension scores than the base package. One possibility is that knowledge of letters is an indicator that is relevant for readers and non-readers alike. The combined package may benefit more children to become readers but may not be geared toward helping the average Grade 2 student who does not know letters to become more literate in terms of knowledge of letters. Furthermore, when additional program activities are added in a combined package (e.g., school gardens or literacy boost training for teachers), teachers end up focusing more on reading and helping the readers or help with school gardens but may give less time to help the average student improve their knowledge of letters. Alderman et al. (2012) find evidence of unintended negative effects for school meal programs on at least one learning outcome, which they attribute due to potential time related demands of some of the school meal activity programs. Therefore, this effect can be thought of as a substitution effect, where fewer resources are being devoted to children who don’t know letters to learn letters and more are being spent on those who already know letters or can read. 4.2.2 Health and Nutrition Outcomes We also examined outcome variables related to handwashing and nutrition knowledge and behaviors (Exhibit 39). Students were asked to report on their own handwashing behaviors and knowledge of correct handwashing practices, as well as whether they could identify a balanced diet. 64 | AIR.ORG LEARN Endline Evaluation Report Exhibit 39. Overall Impact of LEARN on Key Nutrition and WASH Outcomes, by Treatment Group and Gender Outcomes Combined Package vs Control Group Base Package vs Control Group Combined And Base Package vs Control Group Combined vs Base Package Handwashing behavior .114 (.074) .078 (.072) .097 (.067) .04 (.054) Knowledge of handwashing .007 (.056) -.024 (.066) -.008 (.053) .033 (.059) Knowledge of balanced diet -.004 (.007) .007 (.011) .001 (.007) -.01 (.009) Observations 945 801 1402 1058 Girls Handwashing behavior .169* (.088) .07 (.092) .127 (.08) .084 (.083) Knowledge of handwashing .027 (.085) -.018 (.098) .009 (.079) .034 (.087) Knowledge of balanced diet NA .008 (.008) .003 (.003) -.009 (.009) Observations 448 368 656 496 Boys Handwashing behavior .075 (.079) .088 (.077) .08 (.073) -.005 (.056) Knowledge of handwashing -.004 (.069) -.038 (.077) -.017 (.066) .027 (.063) Knowledge of balanced diet -.006 (.013) .005 (.015) -.001 (.013) -.01 (.012) Observations 497 433 746 562 Source: Student survey; authors’ calculations; p < 0.1 ** p < 0.05 *** p < 0.01; Standard errors shown in parentheses are clustered at the school level. Sample is restricted to girls. Only key DID (Treatment X Post) key impact estimate are shown for relevant treatment and comparison/control groups and relevant outcomes. We first show estimates for overall sample, and then we disaggregate results by gender. We control for dummies for time (endline vs baseline), relevant treatment groups as well as age, age squared, total number of assets and English as main spoken language at home. 4.2.2.1 Self-Reported Handwashing Behaviors To assess changes in students’ handwashing behaviors, we examine students’ self-reports of washing their hands at three critical moments the day before the interview: after defecation, after urination, and before eating food. Students were classified as having proper critical handwashing behaviors if they reported washing their hands in each of these instances. Overall, we do not find any impact of the LEARN program on reporting handwashing behaviors. While we would not necessarily expect to find impacts in the school feeding only group, we would expect that students receiving the additional school health and nutrition interventions to have improved outcomes. This absence of an impact in the school feeding, literacy boost, and school health and nutrition schools suggests that the interventions were not implemented with fidelity or that children in comparison schools realized similar improvements in critical handwashing behaviors, perhaps because of increased COVID prevention messaging. Based on the qualitative data, it appears as though schools lacked basic materials that would facilitate regular handwashing: buckets, soap, and water were often reported to be unavailable or irregular. In the period soon after the re-opening of schools in 2020, when midline data was collected, schools across Liberia were generally more equipped for handwashing and were 65 | AIR.ORG LEARN Endline Evaluation Report receiving regular handwashing reminders to avoid COVID. This was indeed the case according to respondents in the LEARN schools visited for qualitative research. Also, students included in communities with SHCs reported ongoing concerns with the cleanliness of their schools, especially toilets, which were meant to be maintained with the help of the School Health Clubs. As such, relative to these fundamental concerns, it may be the case that handwashing behaviors were secondary. This issue is further elaborated on in the Qualitative Discussion. Subgroup Analysis. When examining differential impacts for handwashing behaviors by sex, we find that handwashing behaviors for girls in combined package schools increased (16.9 percentage points; p < 0.10), but we found no other statistically significant effects on either girls or boys. Comparison of Treatment Arms. We found no statistically significant effects on any of the samples we studied (overall or disaggregated by sex). 4.2.2.2 Knowledge of Handwashing Behaviors We also examined students’ opinions of critical handwashing behaviors. We constructed an indicator of critical handwashing knowledge based on their responses on whether individuals should wash their hands after the same three critical moments. As with handwashing behaviors, we find no impact of LEARN programming on handwashing knowledge. Subgroup Analysis. We do not find any evidence of overall or differential impacts of the program packages on knowledge of critical handwashing behaviors. Boys and girls appear to have the same level of knowledge of proper behaviors. Comparison of Treatment Arms. We find no significant additional impact of the literacy boost and school health and nutrition interventions compared to the impact of the school feeding only intervention. Neither treatment package produces a significant effect on students’ knowledge of the critical moments for handwashing. 4.2.2.3 Nutrition: Knowledge of a Balanced Diet To determine whether students could identify the components of a healthy diet, the survey asked students to identify the three elements of a balanced diet, defined and taught by SC as go, glow, and grow foods. We find no evidence of a significant impact on students’ nutrition knowledge.31 31 Some of the nutrition information may have been provided in a format different from "go, grow, and glow" and instead drew from the MOH's guidance. Further research is needed to investe the reasons behind the null effect on nutrition. 66 | AIR.ORG LEARN Endline Evaluation Report The qualitative data suggest that student, teacher, and caregivers priorities related to the variety of SHN interventions were primarily around having a clean school environment with working infrastructure. As mentioned above with respect to handwashing, and further discussed below, schools often lacked basic items such as soap, water, buckets, and clean toilets; in this context, attention from SHCs and SHN champions may have been less focused on nutrition. Indeed, in FGDs and KIIs, rarely was nutrition education discussed as it related to LEARN activities. Subgroup Analysis. We do not find any evidence of overall or differential impacts of the program packages on knowledge of balanced diet. Boys and girls, both appear to have low levels of knowledge about balanced diet. Comparison of Treatment Arms. Assessing the additional impact of the literacy boost and school health and nutrition interventions on knowledge of balanced diet, we find no value added over the school feeding only intervention. Again, given the low proportion of students who were able to identify a balanced diet, these results should be interpreted with caution. 67 | AIR.ORG LEARN Endline Evaluation Report 5. Qualitative Discussion The research team assessed the relevance, effectiveness, efficiency, perceived impact, and sustainability of the LEARN interventions with respect to the research questions related to literacy, nutrition, WASH, and SRGBV themes (i.e., relevant to the McGovern-Dole results framework). We conducted KIIs and FGDs with beneficiaries at the school and community levels, as well as with project staff and government officials at the national and local levels. The findings are based on data analysis from 12 intervention communities visited (three per county) that had different variations of the LEARN packages (see Exhibit 40). Exhibit 40. Qualitative Intervention Sites, by LEARN Packages Delivered County School Feeding Base Package (SF) Literacy Boost (LB) School Health & Nutrition (SHN) Grand Bassa 1 x 2 x 3 x Grand Gedeh 1 x x x 2 x 3 x Rivercess 1 x x 2 x x 3 x x River Gee 1 x x 2 x x 3 x x As described in Section 2.3, we interviewed 350 community-level stakeholders (166 female, 184 male) from the 12 intervention sites. Boy (n=80) and girl (n=78) students were in Grades 4, 5, and/or 6 and ranged in age from 11 to 20, with an average age of 15.5 and median age of 16. The 50 female and 22 male parents/caregivers ranged in age from 22 to 89, with a median age of 42. Of those, just over half (n=38) had no education; a similar distribution of the remainder had elementary (n=10), junior high (n=14), or high school education (n=10). Teachers were mostly male (n=45 males and n=12 females); all but one principal was male. Most teachers had college and most of the remainder had high school education; just one (a volunteer teacher) had only junior high. All but one of the principals had college education; the remaining one had high school (Exhibit 41). 68 | AIR.ORG LEARN Endline Evaluation Report Exhibit 41. Respondents’ Educational Attainment, by Interview Type and Intervention Stakeholder Education Level Students (n=158) Teachers (n=58) Caregivers (n=72) Principals (n=12) LCs (n=8) Cooks (n=12) Storekeepers (n=12) DEOs (n=8) College / Teaching Certif. 0% 62% 0% 83% 88% 0% 0% 100% High School 0% 36% 14% 17% 13% 0% 42% 0% Jr. High 0% 2% 19% 0% 0% 0% 0% 0% Primary 100% 0% 14% 0% 0% 0% 17% 0% None 0% 0% 53% 0% 0% 100% 42% 0% Total 100% 100% 100% 100% 100% 100% 100% 100% Source: 2022 Endline qualitative interviews in all 12 intervention sites for a total of 340 stakeholders at school and community levels (excludes 10 SC staff community mobilizers for whom education data was not collected). Authors’ calculations. Key informant interviews via phone or Skype were also held with three senior staff from Save Liberia and one from MercyCorps Liberia, three national staff from the Government of Liberia (one from the Ministry of Agriculture School Garden Division and one from the Ministry of Education School Feeding Unit), and one Ministry of Education County Education Officer (Grand Gedeh). Below, we present the qualitative findings at endline by using the five OECD-DAC criteria and across each criteria addressing the evaluation questions (Annex C) through exploring each of the outcome areas. Each criteria section begins with a summary of findings. 5.1 Relevance The research team assessed the relevance of LEARN interventions, asking whether project activities were appropriately addressing needs and the extent to which stakeholder voices were considered in improving activities to better address those needs through interviews and FGDs with project implementers, national and local government officials, and community level stakeholders and beneficiaries. This section discusses the following indicators related to relevance: 69 | AIR.ORG LEARN Endline Evaluation Report Exhibit 42. Relevance-Related Indicators and Findings Category Activity Indicators Key findings Enhancing school quality through school feeding and teacher support Advocating for prioritizing education Student and parent support for education Students and parents want education for all; challenge is around ensuring it is prioritized given limited money and other livelihoods￾related challenges School feeding Agreement from stakeholders on need Agreement that meals contribute to increased enrollment, attention, performance; highly popular School gardens Agreement from stakeholders on need From midline, growing agreement from communities that gardens are worth effort Improving literacy Training teachers Agreement from teachers and students on need Teachers needed additional support and materials (books) for teaching reading Literacy champions / reading clubs Agreement from students on need Students need support outside of school and sometimes what they can access at home is limited; support outside of school is very helpful School health and nutrition SHN champions and school health clubs (SHCs) Agreement from stakeholders on need Clean school is important to students, caregivers, and teachers; without specific intervention, school grounds become dirty; clear need for system to ensure cleanliness. SRGBV Sensitization on codes of conduct Knowledge of, agreement with, and following of code SRGBV is still common, including in some cases out in the open, so more work is needed around ensuring codes are agreed with and followed 5.1.1 There is Need to Enhance School Quality (School Feeding, Teacher Attendance, and Teacher Performance) to Increase Enrollment, Attendance, and Achievement 5.1.1.1 Students Enjoy and Value Education As in the baseline and midline evaluations, respondents of all types and communities at endline saw the value in education (broadly) and literacy (specifically). Respondents welcomed any help for their children to improve their education, including literacy, which was generally seen as essential for success in life. All students self-reported that they enjoyed school and thought their education was important. Students also revealed high expectations about what they could achieve in their future, provided they were able to continue their schooling. For example, boy students said, 70 | AIR.ORG LEARN Endline Evaluation Report “P1- I want to learn so that I can be big person in the country. I want to be a president; I’m going to school because I want to be a lawyer. P2- I love school because I want to work in the hospital. P3- I love school because I want to be a clan chief. I want to control my parents and help make them rich.” - (Boy students, Grand Gedeh) Similarly, girl students said, “P1-I come to school to learn book because when you graduate from high school, nobody will bluff you, I want to make my own money. P2- I want to learn book because I want to graduate and sit in my own office so that no one can bluff me… P3- I come to school to learn so that I won’t depend on anybody in the future and be able to write my own letter. P4- I come to school because I want be a doctor, I want treat people and also want to lean about health because it tells you more about the human body. P5- when you learn you can be someone good in the future. I want to be nurse because we don’t have nurse in our family, I want to treat people and have my own drugstore.” – (Girl students, Grand Gedeh) Occasionally, students or teachers reported that ‘some’ students did not take school seriously or did not like learning. In many cases, students’ perceptions were gendered: boys said that girls didn’t focus on school because they were ‘chasing boys’ or at clubs. Girls said that boys didn’t focus on school because they were ‘out working’ or ‘chasing girls.’ Teachers also noted there were indeed some students who did not take their lessons as seriously as others though their perceptions were less gendered than those of the students. While there may be some social desirability bias reflected in the responses of students about their own values about education, it is evident that education was, nominally, very important to students and parents. Despite overwhelmingly positive attitudes about the importance of education, existing challenges endured between midline and endline, and new enrollment and access challenges emerged. In 2021, the MOE instituted school fees ($1,000 Liberian dollars) for 2021–2022 school year. Because SC staff and community members thought this would reduce enrolment, project staff worked with MOE staff to allow parents to pay the school fees in installments, which was effective in preventing a drop in enrollment. Monitoring data collected by SC into 2022 revealed that enrollment had actually not declined by that time, though it is possible that many of the students had enrolled late and missed weeks or months of instruction. 71 | AIR.ORG LEARN Endline Evaluation Report As it relates to regular attendance, students continued to face pressures to engage in livelihood strategies, help with domestic tasks, or tend to other family or personal matters at the expense of their education. Respondents often said that parents still had a difficult choice about whether to send their children to work – essential to help supplement the family’s income – or to school. In some cases, students were able to attend but struggled to find time after school to complete assignments or study. 5.1.1.2 School Feeding Remains an Incentive to Send Children to School As at midline, all stakeholders remained overwhelmingly positive about the potential of school feeding to increase enrollment, attendance, and achievement of young people in schools. Respondents also agreed that it was difficult for parents to send their children to school with food or money to purchase lunch, so this could discourage them from sending them at all. Regular access to lunch was a huge incentive for parents to send their children to school regularly. Equally, it was an incentive for students to attend, knowing they would not have to worry about where to find lunch or to spend the school day hungry. Having adequate nutrition was also an essential component for young people to maximize attention and learning. At endline, one issue related to the relevance of the school meals emerged: students reported wishing for more variety in their meals from day to day. 5.1.1.3 Teacher Morale and Attendance Remains a Concern According to multiple beneficiaries and stakeholders, teacher morale and attendance continued to be a concern at endline. In addition, multiple teachers were not on the government payroll, working with the hopes of PTAs or others pooling money to pay them a small stipend. Even those on the government payroll lamented their low salary, inadequate materials, and frequently late payments. For example, one principal said, “They should come down on the government to pay teachers. Teachers are not taking pay while they are giving the Zoes and elders motor bikes that cost 3000 united states dollars who are not making any impact in society. Teachers are suffering especially those that are in the rural areas. Also, those that are been called for large teachers meeting or school meetings they should include the principal and allow them to give their opinion and speak out in such meetings.” – (Principal, Rivercess) While parents and students highly appreciated teachers for their work, they also said the quality was not always sufficient. The qualitative data suggests that teachers were generally in attendance, but some were late or when present, were not motivated. 72 | AIR.ORG LEARN Endline Evaluation Report Perhaps a reflection of teachers’ financial challenges and efforts to address them while continuing as teachers, the qualitative data showed multiple reports of paying for grades by students across the communities. For example, one girls’ FGD in Rivercess discussed, “P1-The teachers can encourage us not to study because when we fail we can give the teachers money for grades and they can pass us. Some of us can steal our ma’s money while some of us can ask our boyfriends for money…. P7- I’m selling so when I fail I can take the money from there to pay the teacher for grade. My mother doesn’t know… Most boys are mechanic so they can’t pay attention in class because they can pay money. … The principal just got to know about it and he got vex with the teachers and advise them. Our parents noticed that we can’t read but we have good grades so they attacked the principal.” – (Girl students, Rivercess) In response to these known challenges and the resulting limited impact teachers could have on students, SC worked with local education officers to try to improve teachers’ conditions, while also working in the schools to improve teacher morale and conduct. 5.1.2 Students Are Eager to Become Good Readers As with students’ general interest in and enthusiasm for education, students were eager to become good readers. None of the students in FGDs reported being uninterested in reading; none of the parents in FGDs reported that their own children were uninterested in reading. Students did, however, identify barriers they experienced when trying to improve their literacy. As with baseline and midline, a key challenge for students to have improved literacy was finding people to adequately support their reading both in and outside school. Within the qualitative sample, there appeared to be substantially more parent engagement and efforts to help children do schoolwork at home. Students also reported more often than at midterm that they were seeking help from older siblings or friends. At endline, relatively few boys and girls said that nobody reads with them, which was commonly reported at baseline and midline. Despite these improvements in support, students often mentioned needing additional support such as a “study class teacher” (personal tutor paid by the family) or additional in-school and after-school support. SC recognized that improved instruction within school was critical, but, especially given the limited control that SC had on in-school quality (relying on MOE trainers and supervisors), students also needed access to outside support such as through Reading Clubs, “study classes” (which could include Reading Clubs or after-school opportunities for homework help), summer reading camps, and personal tutors (a “study class teacher” or “home teacher”). 73 | AIR.ORG LEARN Endline Evaluation Report There were indications that books were still scarce and students, still wanting to read, resorted to whatever they could find. For example, students reported reading and re-reading their notes from school for lack of anything else available. Others read the bible, or, when possible, textbooks that they were able to bring home, “I study my notes because I don’t have reading books.” – (Boy student, Grand Bassa) 5.1.3 Clear Need to Reduce School-Related Gender-Based Violence There is clear relevance of SCs work as it relates to trying to lessen instances of school-related gender-based violence, a widely recognized problem across Liberia (and globally). Not only is it an issue around ensuring student physical and mental well-being and safety, but as it relates to LEARN’s desired outcomes, SRGBV can have negative impacts on student attendance and achievement. As it relates to LEARN, the baseline and midline evaluations identified multiple instances of SRGBV, including corporal punishment and verbal abuse from teachers, teachers administering physical labor punishments, and bullying or teasing between students. At midline, multiple schools reported a literacy game called ‘beating spelling’ in which teachers would provide two students a word, and whomever of the two students who spelled the word correctly first was able to hit the other student. Rarely, teachers sexually harassing or assaulting girl students were reported (while rare, one may assume that such incidents may be occurring more than is reported). 5.1.4 Clear Need for Improving School Health and Nutrition SHN activities around promotion of healthy practices are evidently necessary given the relative lack of student knowledge on these topics (evidenced in the quantitative data); however, in the context of school environments sometimes lacking necessities such as clean toilets, soap, buckets, and water, healthy practices and nutrition promotion appeared to be secondary in importance. At endline, stakeholders across all communities agreed that having a clean school environment with good access to clean water and regularly practiced hygiene was critical. Interventions related to improvement of WASH were welcome, and there was an explicit desire among students and parents across schools to improve WASH further. The need to remain vigilant about COVID prevention measures further demonstrated the importance of hygiene. Students also complained around issues with cleanliness of school grounds or broken infrastructure, which required both effort from individuals contribute to cleaning and improvement tasks, and organization from PTAs to pool human and financial resources to support those improvements. As such, SCs work in this regard remains highly relevant and work through PTAs and SHCs has contributed to some improvements but has many areas for improvement. 74 | AIR.ORG LEARN Endline Evaluation Report 5.2 Efficiency The research team used interviews with project implementers and community-level stakeholders to assess efficiency and timeliness for achieving objectives, including internal and external factors affecting implementation of project activities. This section discusses the following indicators related to efficiency: Exhibit 43. Efficiency-Related Indicators and Findings Category Activity Indicators Key findings Enhancing school quality through school feeding and teacher support PTA activities Building capacity of PTAs SC conducted community outreach with PTAs to clarify roles; also worked with MOE to help strengthen their work with PTAs School Feeding On-time delivery of commodities; adequate supplies in kitchens Generally on-time; few issues with theft of commodities that led to delays; challenges with providing adequate materials to kitchens School Gardens Agreement from PTAs to establish and maintain gardens Good progress since midline showing PTAs that gardens can help supplement school feeding and be a source of income for PTA and other community members. Improving literacy Training teachers Teachers equipped and willing to teach literacy Good progress working with teachers to enhance morale and skills (e.g., staff recognition, trainings, liaising with local MOEs to improve conditions) Literacy champions / reading clubs Literacy champions adequately trained and delivering activities Good progress in ensuring LCs stay in their posts (support to LCs to enhance morale, and with MOE to reduce transfers; use of volunteer LCs instead of teachers). LCs lack some materials, including books. School health and nutrition SHCs SHCs trained and equipped SHCs are mobilized and have women involved, but lack basic materials to complete tasks (e.g., buckets, soap); PTAs provided limited support; success in achieving gender parity is unclear SHN Champions SHN Champions trained SHN champions have been trained but there has been high turnover (transfers) so some are not in place SRGBV Codes of Conduct Codes revised and adopted Code nearly finalized; draft code used to prepare safe schools stories to share with communities across two counties. 75 | AIR.ORG LEARN Endline Evaluation Report 5.2.1 SC Has Enhanced School Quality Through School Feeding and Teacher Support 5.2.1.1 Parent-Teacher Associations Increased Activities Since Midline At midline, many PTAs were said to be inactive; those that were active had limited activities (less than was expected of them for LEARN interventions). Recognizing the need to work more with PTAs, SC conducted community outreach including:  Reinforcing and clarifying PTA roles and responsibilities (e.g., overseeing the school garden, pooling resources for school, oversight of school feeding)  More sensitization with PTAs to increase the number and types of people who engage, emphasizing that PTAs should not just be made up of ‘executives’ (principal and one parent) but should have multiple parents and non-parent community members in attendance to ensure relevant issues are addressed.  Increased and more purposive engagement with the MOE (PTA engagement division) to provide better support and coordination to PTAs. LEARN national and community-level staff attended some of the meetings with PTAs and MOE to understand the context and provide feedback. 5.2.1.2 School Feeding is Efficiently Delivered and Managed As with respondents at midline, respondents at endline were overwhelmingly positive about the efficiency of the school feeding program. In rare cases, as with midline, there were reports of late delivery of food such that meals were missed. In some cases, a finding that emerged more in the endline data collection than midline, the late food was due to theft, which SC also recognized to be a problem from midline. In some cases, the theft was on a larger scale, leading SC and government to procure warehouses to safely store the food before it was used in schools. In other cases, though, it was said that some school personnel stole some of the food. One girl student in Grand Gedeh said, “They didn’t give us food last week because they said food finish… the people can sell the food that’s why the food can’t stay long.” – (Girl student, Grand Gedeh) None of the complaints from midline regarding students and parents having to pay for meals were made at endline, though students and teachers said that sometimes there were not adequate condiments or dishes (the responsibility of PTAs). PTAs also said they continued to struggle to support the feeding component of the project by contributing to supplies for school meals (“soup kind”) and money to support meals, WASH, and other school projects. 76 | AIR.ORG LEARN Endline Evaluation Report At endline, four of 12 schools reported inadequate materials to support the kitchen’s work to provide school meals; varying quality of the kitchens with cooks, storekeepers, and other school staff reporting damaged or inadequate cooking infrastructure; and difficulty with supplies and maintenance, even with rehabilitated kitchens. These were also challenges at midline, indicating challenges with the LEARN activities meant to support rehabilitation of kitchens, and provision of materials, to facilitate food preparation for school meals across all sites. 5.2.1.3 Advocacy and Support for School Gardens Has Been Effective After a substantial delay in the garden intervention at midline, some schools were reportedly using food to supplement school feeding commodities or, if they had not established gardens yet, they had plans to do so. School gardens had been envisioned for all schools in Rivercess and some schools in Grand Gedeh (combined package sites) to help support both school feeding with supplementary produce, and to provide PTAs with additional income from selling the produce for school-related activities. A staff member from MOA shared that because of the challenge in getting LEARN school gardening materials by May 2022, they missed planting season (there would be limited rainwater by August). As such, the MOA estimated that about 75% of the schools were reached in time but the remainder were so remote, combined with bad road network, that it was not possible. A staff member from the MOE school feeding division also recognized the staffing and procedural challenges with school garden implementation, “The school garden component was not delivered as expected. When I went on the joint monitoring visit, I saw that most of the garden did not do well. There is only one agriculture specialist for both Rivercess and Grand Gedeh and there is no agriculture monitoring system and coordination for each county. There should be coordinator and monitor for each county. They should be able to provide implementation tools and train the PTA to be able to manage the garden.” – (Director, School Feeding Unit MOE) Also, in some communities, access to land for the garden was an issue. For this, LEARN worked with the Ministry of Internal Affairs (MOIA) to try to relocate locations, which further delayed implementation of gardens. Government also recognized the challenge in getting community members on board for the gardens and, even if established, maintaining them to productivity. In addition to supporting delivery of gardening materials, SC worked toward advocating for PTAs and other community members to support the gardening activities. They conducted sensitization with parents about the positive role gardens could play both in terms of supplementing school meals, but also to help raise money for school-related activities. 77 | AIR.ORG LEARN Endline Evaluation Report 5.2.2 Literacy Boost Activities Have Been Improved 5.2.2.1 Literacy Champions Have Been Trained and Mobilized Since midline, SC staff saw high turnover of LCs in the literacy boost schools (all River Gee schools and combined package Grand Gedeh schools) who were responsible for delivering much of the literacy component of the project. This turnover preceded COVID, stemming from challenges with keeping LCs interested in staying in rural communities, accepting payment in food rations, and DEOs and CEOs transferring teachers (many of whom are also LCs) frequently. As such, the investment SC provided in training those LCs was often lost. Compounding this challenge was COVID, during which LCs were tasked with helping students in the communities use the Home Learning Packet, which (as detailed in the midline evaluation) was mixed in implementation and effect. The midline evaluation further explains that LCs were slow to re-establish reading clubs, and that literacy instruction at the classroom level had been limited due to interruptions in training and support to teachers. Acknowledging this, SC took steps with the LCs including being more supportive to their needs such as providing additional training throughout the year, making regular monitoring visits to hear their requests, and ensuring regular provision of rations (LC’s form of compensation). SC also instituted other activities to encourage teachers in general to stay, which was particularly important in more rural and lesser serviced areas where teachers tended to try to relocate away from, such as instituting teacher recognition activities. For example, under the LEARN STAR teacher initiative, students, parents, and PTAs identified and rewarded high-performing teachers. SC also worked with local education government officials (DEO and CEO) to try to improve pay and other conditions for teachers, such as procuring vehicles to better supervise and support teachers, and adding volunteer teachers to payroll. As a result, according to SC staff, turnover has been limited since midline and students and teachers are quite positive about the work LCs have done (elaborated on in next section). Importantly, SC was also able to implement the Summer Reading Clubs in 2021, which had not been possible in 2020 due to COVID. The positive perspective on these clubs is also elaborated in next section. 5.2.3.2 Students Lack Books and Learning Materials at School As mentioned above, a key literacy-related challenge identified at baseline and midline was that students lacked materials to read at school. Through LEARN, SC and USDA aimed to leverage work done by MOE and USAID Read Liberia to provide literacy materials to all children and teachers in Grades 1 and 2, and training to all teachers. For example, LEARN provided all Literacy Champions with boxes of books and materials that they could use in their work. Also, according to SC, 118 copies of a new books were added to the book banks in Grand Gedeh and River Gee. However, at endline, respondents did not mention existence of “book banks”, but rather mentioned accessing some books at school via teachers. Even then,students continued to struggle to access 78 | AIR.ORG LEARN Endline Evaluation Report reading materials, especially those that they could borrow (though a system of loaning books was not necessarily expected at the time of research as it had only been rolled out to five schools in Grand Gedeh). Some LCs did mention having books that they would share with students. There was one report from a student that teachers would loan books at a fee (it was unclear whether these books were his/her own books, or from an established book bank). LCs also had been trained to develop their own local reading materials to address the issue with limited book supply (elaborated on in next section). 5.2.4 School Health and Nutrition Activities Have Lost Some Momentum Since Midline While at endline, SHCs were said to exist in all four communities meant to have them; respondents said that both men and women were active in the SHCs, though it was not possible to confirm that there was indeed gender parity. SHCs were also said to be active in multiple schools without the SHC component of the LEARN intervention; however, in many cases, they appear to have lost some momentum. Also, it appears that some of the SHCs that existed in the period immediately following the reopening of schools after COVID closures had since stopped their activities; for example, one girl in Grand Bassa said of her SHC, “It was active last year but this year it is not active.” – (Girl student, Grand Bassa) Similarly, a teacher in River Gee said, “The school health club was very active especially during the COVID time. We use to place handwashing buckets at strategic locations on the campus. We also located dump site. We haven’t taken messages to the community from the time the school health clubs were established. We used to tell students to wash their hands when they come to school and the information should be extended to every community.” - (Teacher, River Gee) The SHN champions also faced challenges up to endline. For example, SC recognized that, like the issue faced by LCs at midline, there was a problem with SHN champions leaving or being transferred after receiving their training. As such, SC understood that there was inadequate understanding of school health and nutrition among beneficiaries and continued to work with SHCs and SHN champions to enhance knowledge. Also notably, there appears to have been limited work with PTAs on their tasks related to SHN, given how infrequently they mentioned this aspect of their responsibilities in qualitative interviews. It is important to note also that nutrition￾related activities were started relatively later in LEARN, with more attention being given to the COVID-19 prevention efforts that focused on WASH, giving less time for potential nutrition￾related impacts to be observed. 79 | AIR.ORG LEARN Endline Evaluation Report 5.2.5 SRGBV Addressed by Sharing Safe School Stories With Teachers, Students and Parents At endline, the revised TCOC – an MOE task that was supported by SC and other country office NGO actors – was nearing finalization and drafts had been disseminated to local level MOE actors. To work toward addressing the issue of SRGBV before the revised TCOC was more widely disseminated, SC devised a set of ‘Safe School Stories’. SC utilized a social behavior change approach in tasking community mobilizers with sharing these Safe Schools Stories with teachers, parents, and students independently. Each story had a lesson to teach about the context and rationale for parts of the TCOC, raise awareness about the TCOC, and promote appropriate teacher behavior, and parent or student response to violations. Not all communities were expected to have been reached at the time of endline data collection. There were plans to share the stories also via radio programming, though this had not been rolled out at endline and was to be considered for LEARN II. 5.3 Effectiveness and Impact (Perceived) The endline evaluation examined perceived impacts of the project’s key intervention activities, including school feeding, literacy, WASH, and SRGBV. Interview topics focused on the successes and challenges of program implementation in achieving its desired outputs and outcomes. This section discusses the following indicators related to effectiveness and perceived impact: Exhibit 44. Effectiveness and Perceived Impact-Related Indicators and Findings Category Activity Indicators Key findings Enhancing school quality through school feeding and teacher support PTA activities Activities completed related to LEARN and other school tasks Much improvement from midline from PTAs across multiple areas. Increased commitment and activity. Continue to struggle to get more parents involved. Parents in general feel more comfortable to engage with teachers, though teachers consider them generally uninterested. Also stronger engagement more generally from parents in their children’s education School Feeding Perception that feeding has helped Widespread agreement that school feeding has enhanced enrollment, attendance, and achievement among students though meals could be more varied; some concern that take-home rations should also go to boys. School Gardens Establishment of gardens to supplement school lunches All PTAs have established gardens or have plans to do so; recognize the importance and benefit gardens can have for school feeding and more. Some are lacking adequate materials. 80 | AIR.ORG LEARN Endline Evaluation Report Improving literacy Training teachers Improved teacher morale and quality of literacy instruction Teachers specify strategies they use for teaching literacy and are positive about it; students appreciate this instruction; stands out from midline where it was difficult to distinguish LEARN-specific pedagogy. Some reports of paying for grades Literacy champions / reading clubs Improved Literacy Champion morale and quality of literacy instruction LCs and reading clubs are appreciated by students and said to be effective though potentially they are most impactful for existing readers. Work on developing innovative reading and instruction materials is promising despite reported lack of books. LC morale is improved and they feel equipped to do their work, but express dissatisfaction with payment in rations. Literacy at home Culture of reading at home is increased Good progress toward establishing a culture of reading at the home; parents are more frequently engaged even if they do not know how to read; books are still difficult to obtain School health and nutrition WASH Knowledge and Practice School is clean and students have good hygiene practices Schools are frequently said to be dirty and lacking basic products (soap, buckets, and clean water); weekly cleaning activities to be done by SHCs are not guaranteed; PTAs are not highly involved. Nutrition Knowledge and Practice Students are knowledgeable about nutrition There is little work to be observed as it relates to nutrition; few respondents commented on this component of SHN activities. Nutrition seems to have been dropped in priority compared to hygiene. SRGBV Codes of Conduct Codes are agreed with and adhered to Knowledge of codes is widespread, but still SRGBV practices remain relatively widely reported, in particular teachers beating students, physical labor for punishment, and accepting payment for grades. 5.3.1. SC Has Been Largely Successful in Improving School Quality (School Feeding, Teacher Attendance, and Teacher Performance) 5.3.1.1 PTAs Have Improved Understanding of Roles and Responsibilities While the midline evaluation found that PTAs were relatively inactive, interviews with PTA members and other teachers and caregivers at endline suggest much improvement in terms of the PTAs’ understanding of their role and activities. PTAs were tasked with supporting gardening activities, mobilizing parents to support purchasing of condiments for school feeding, and overseeing school feeding implementation. At endline, nearly all respondents across communities said PTAs carried out their responsibilities for LEARN. One principal in Rivercess described the newly active PTA at his school, 81 | AIR.ORG LEARN Endline Evaluation Report “Yes, it been active since LEARN came because LEARN offered the PTA training. They tell the PTA to take responsibility of the school garden and the PTA have meeting to decide what to give to the volunteer teaches.” – (Principal, Rivercess) Beyond LEARN-specific activities, caregivers and teachers described how PTAs contributed to other non-LEARN-specific tasks such as encouraging parents to send their children to school; monitoring teacher attendance; pooling money to pay volunteer teachers, Literacy Champions, cooks; and fundraising and mobilizing for school-related infrastructure repair or cleaning projects. In one case (Grand Bassa), a PTA purchased land to relocate a school that had been unsafely close to a busy road. PTAs did not indicate their role in supporting nutrition as a component of the broader SHN initiatives. PTAs continued to face some challenges, mainly with engaging multiple members of the community. In one community in River Gee, the principal and PTA leader lamented that only teachers were involved in meetings despite the role parents needed to play in making decisions and actively helping the school. Parents are said to not be involved because they do not see the benefit, in particular, “There is not much willingness from parent to join because they don’t want to be volunteers once money is not involved.” – (Principal, Grand Bassa) Caregivers in Grand Bassa said, “Some parents thought that committee was going to be sharing money in the meeting or they were going to have direct benefit from the meeting, but they saw that that was not happening, they stopped coming.” – (Caregiver, Grand Bassa) One principal in Rivercess pointed out, “The PTA needs to improve on the attendance of the meeting. They need to encourage the parents to attend and pay their dues because the more they pay their dues, it will us get what we need. They need to set rules that if any parent refuse to attend the meeting, you will be charged $50 Liberian dollars.” – (Principal, Rivercess) 82 | AIR.ORG LEARN Endline Evaluation Report According to one caregiver in Rivercess, the PTA was not very active due to parent inattendance, “No active PTA now. We used to have PTA chairman here and they used to send citation to the parents but everything have stopped…. We have PTA chairman here but when the teachers call for meeting the parents can’t come… The government give some money to the principal on behalf of the students but the principal said that he misplaced the money so because of that the PTA got dissolved.” – (Caregivers, Rivercess) Finally, two communities described challenges with leadership, “The current executive has been in power for almost four years. This has affected the work of the PTA in that the people no longer want to work with the old leadership. But we plan to have an election of the new executive second week in May.” – (Caregivers, River Gee) Similarly, a principal in Rivercess said that PTA members were unhappy that the present chair was not letting others take leadership, and that reduced their desire to contribute to the PTA. 5.3.2.1 School Feeding As at midline, respondents of all types across communities at endline were extremely positive about school feeding’s impact on attendance, learner focus and attentiveness, and relief for parents on food security. For example, one girl in Grand Gedeh said, “More people started coming to the school because of the food as compare to when food was not here.” – (Girl student, River Gee) As described in the PTA section above, PTAs have been critical in implementing the school feeding component, and SC’s continued work with PTAs to strengthen their capacity in this regard appears to be effective. The qualitative data suggest that while there is broad appreciation for school meals, students would appreciate more diversity, “The food can’t be sweet, every day is one soup, but at least it can make us stay in school and listen to the teachers.” – (Girl student, River Gee) 83 | AIR.ORG LEARN Endline Evaluation Report Also, there were reports of meals not being prepared well in nine of 24 student FGDs. For example, students reported that there were bugs in the prepared beans or the beans were not fully cooked, “The food can't be sweet, no enough pepper, salt, or cube. The bugs can be in the beans because they can keep it for long.” – (Girl student, Grand Bassa) Based on the qualitative data, school meal provider (cooks and storekeepers) morale was generally high. Nearly all of them said that they had received adequate training, were happy about the positive impact the school feeding program had had on students, and felt well￾supported by school staff and SC. For example, one storekeeper said, “It can’t give me hard time [it is not hard for me] to serve the cook. Absolutely I haven’t encountered any challenges in this work. But if for any reason I have challenges, I can tell the principal or Z (the Community mobiliser).” – (Storekeeper, River Gee) A cook in Rivercess said, “Nothing is giving me hard time, everything is okay.” – (Cook, Rivercess) When asked to share any challenges, cooks most often reported having kitchens in need of repair (reported by seven of 12 cooks) or basic materials (reported by seven of 12 cooks); storekeepers commented on needing better doors and walls to storerooms to keep out pests (reported by three of 12 storekeepers, all of whom also mentioned that they had since fixed the doors themselves). There were rare reports that some of the food received was not sufficient for distributing to the cooks each the month (reported by two of twelve storekeepers), and reports that food received was sometimes spoiled or had bugs inside (reported by one of 24 cooks and storekeepers who, in response to these instances, would check the food upon receipt and return it to be replaced). Two of the 12 storekeepers complained that their food rations were late and one said she would prefer cash payment. Importantly, none indicated that they would not continue their jobs without better payment or conditions. 84 | AIR.ORG LEARN Endline Evaluation Report Though not recognized explicitly by cooks to be a challenge, the qualitative data suggest that they lacked additional food supplies to diversify and enhance the nutrition of the meals. In only three of the twelve communities did cooks and storekeepers mention that they cooked more than beans and rice. One cook indicated that she was aware of the need to diversify, but that this was not easy, “They said, we should change the soup we cook for the children because they will get tired with the same beans every day… We are making farm but the farm can’t make it, we can only plant cassava and other soup like greens, okra, and bitter ball but it will not be enough.” – (Cook, Rivercess) KIIs with other cooks indicate that they did not regularly have access to any foods beyond the commodities provided (i.e., rice, beans, oil) by LEARN and condiments provided (i.e., Vita, salt, pepper) by the PTA – though even those condiments were not always available if the PTA had not been successful in procuring them (usually through collecting money from parents). The qualitative data suggest that access to animal protein or vegetables depended on whether others in the community provided it, and that it was not necessarily regular and sometimes at a cost to parents: For example, a cook in Grand Bassa said, “The principal can tell the teacher to collect money for the food, they are both men and women. Sometimes, the men can buy meat and the women can buy fish and pig feet for the food.” – (Cook, Grand Bassa) A cook in Rivercess said, “Sometimes, the principal can ask the children to bring greens to change the soup when we don’t have greens in the garden… the students are not responsible to bring the greens but we can just talk to them to bring it.” – (Cook, Rivercess) A cook in another community in Grand Bassa said, “[The community mobilizer] also told us to put fish in the beans every and the money is provided by the students in the sum of 25 Liberian dollars per student per day.” – (Cook, Grand Bassa) 85 | AIR.ORG LEARN Endline Evaluation Report Take-home rations for girls were also viewed positively by many who recognized that girls faced unique pressures and challenges as it related to staying in school. One county education officer explained, “They [all students] are benefiting the same but the girls have an edge, because they are encouraging them to stay in school and to retain in school by providing extra ration for the girls to take home. There are benefits for them, especially parents who are taking care of their girls, most of the girls who are in school and are staying with their parents.” – (CEO, Grand Gedeh) However, there were indications from boy students and other stakeholders that focus on girls for take-home rations could be leading to some resentment since many boys, too, were facing challenges in staying in school. For example, one group of caregivers in Rivercess said, “P7- We are appealing that the food be given to the boys as well. Most girls are not serious again but the boys are serious. P1- They should give the food to all the boys and girls. They said that they don’t want the girls to get pregnant, so they used to give it to the girls.” – (Caregivers, Rivercess) This suggests a need for the project to reiterate to communities the rationale for providing girls with THRs – reducing the risk of sex for grades and grooming, demonstrating commitment to equality by giving girls a boost given past and current evidence of boys performing better in many cases. 5.3.2.2 School Gardens Between midline and endline, SC explained, there were two main barriers limiting the establishment of school gardens. First, parents had a perceived lack of time to work on the garden or seeing a reason to spend time on a school garden when they have their own farms or livelihood strategies to pursue. Second, as mentioned in the Efficiency section, delivery of gardening materials was delayed or had still not occurred for many schools. At endline, in all parents and teachers’ interviews, the potential was recognized whether there was an established garden or not. This can be attributed, to some extent, to SCs work sensitizing PTAs and others about the important role that gardens could play in helping supplement school meals, helping PTAs raise money by selling produce, and eventually (a focus of LEARN II), expanding gardens into farms to allow parents to earn additional income to support school￾related expenses or discourage them from taking their children out of school to work. For example, some parents noted that school gardens would help diversify the type of food that 86 | AIR.ORG LEARN Endline Evaluation Report children get at school, about which there were some complaints that it was becoming boring to children. Parents also noted the potential in PTAs earning income from selling the garden produce; that income could be used to support purchase of condiments, plates, and cutlery, which was part of PTAs’ recognized responsibilities for LEARN and would need to be sustained after LEARN. One principal reported, “The PTA understands that the LEARN project will one day come to an end and that is why they have initiated in have a garden for the school. I just want the PTA to focus on the garden even though they said they may need material from the LEARN project.” – (Principal, River Gee) 5.3.1. Parent Engagement with Teachers At endline, there seems to have been improvement in parent engagement with teachers as compared to midline, when some parents said they lacked time or felt uncomfortable doing so. At endline, caregivers in all 12 FGDs said they engaged with their child’s teachers and felt free to do so. Some parents did so only when they saw a need, as one group explained, “P1: I can check their lesson, if I am not satisfied with the notes, I can approach the teacher to tell him because I been in school and I know the kind of notes that need to be given to certain class and I can ask about how my children are doing school… P2: for me, I can go to the campus and tell the teacher to please pay attention on my child because he can tell me he can’t understand some things in the class but he be afraid to tell the teacher.” – (Caregiver, Grand Gedeh) Other caregivers checked in more regularly, whether they had a specific concern or not. However, many teachers continued to lament what they considered to be limited parent engagement despite none of the parents interviewed saying that they did not engage. This disconnect – teachers perceiving parents to be generally unengaged despite this not appearing to be an issue with the parents sampled – was also a finding at midline. Of course, this may simply reflect selection bias in terms of the types of parents who were available for interview, or social desirability bias in that parents were not being fully honest about the extent to which they engaged. It could also, however, indicate that teachers continued to misunderstand the extent to which parents did want to and try to engage. One teacher elaborated, 87 | AIR.ORG LEARN Endline Evaluation Report “I can say 95% of the parents don’t ask me [about their child’s education]. You can only see them during time of sharing grades sheets. They would not even know whether the child can come to school but just say we are giving grade sheets tomorrow, and you will see the whole campus filled with parents. Some of the parents are farmers and are not really interested in their child school. Imagine some parents can be asking me when are we closing school just for their children to join them on the farms… Some parents don’t even care if their child wears shoes or not to come to school. Some students even you send them home because of school fees their parents won’t bother.” - (Teachers, Rivercess) While teachers said that most often, parents did not engage them, they did point out that this was not true of every parent. Some parents were quite involved, which teachers were happy about. Among all teachers commenting on who does engage them, all but one said that it was generally the female caregivers who took the time to engage teachers. Only one teacher had a different view, “Females don’t normally engage teachers because most of them are not educated. The males can engage us more.” – (Teacher, Grand Gedeh) Teachers also said there were some occasions in which parents would visit, for example when grades came out or to discuss school fees, but that regular check-ins on progress was less common. 5.3.2 Literacy Support Has Improved Inside and Outside the Classroom 5.3.2.1 Pedagogical Approaches in the Classroom Have Improved At endline teachers reflected on multiple topics that they had learnt in recent trainings on teaching literacy, and students recognized the different components of the literacy instruction they received. This stands out at endline, though it is important to recognize that the sample of students in the FGDs may not have captured the types of students who the quantitative data identified to have not progressed since baseline. This stands out from midline, where qualitative sites observed were largely indistinguishable from one another in terms of literacy approaches used within the classrooms, despite there being both Literacy Boost and non-Literacy Boost communities in the sample. As mentioned already in Relevance, students paying for grades was a problem reported in multiple communities, 88 | AIR.ORG LEARN Endline Evaluation Report “Some students are buying grades, sometimes, they can pay $200 or $150 Liberian dollars. We that can come to school every day, when we have problem, they can’t help us…more girls are paying money to teacher for grade. Most girls can’t study because they can pay money.” – (Girl students, Grand Gedeh) There were also reports of teachers requiring work from students or payment to administer examinations, “Teachers are still asking us to pay money when we fail in test on campus, the teacher can ask us to go and fix bricks at his house before he can write the test on the board and tell us to put money in the test $100 Liberian dollars.” – (Girl students, River Gee) One student reported his success in reporting the matter to the principal, “When we were taking the first period test, in English I got 98%, the teacher say we should go to his house to work. I put my name down but didn’t there. He said why I didn’t go to his house so he put 60% on my grades sheet to I took the paper to Aunty Theresa (the principal) and that is how they change my grade to 98% and they gave the teacher warning letter.” – (Boy student, River Gee) 5.3.2.2 Students Perceive More Support At Home and Parents Often Say They Offer Support At endline, all caregiver groups reported that they tried to help their children as often as much as possible, and regularly asked about how they are doing. This indicates good progress from midline, where parent engagement was relatively low for a variety of reported reasons including lack of time (exacerbated by COVID-19 when parents were often said to be prioritizing going out to earn money for basic necessities for their families) and perceived lack of ability (e.g., illiteracy) or, according to teachers in particular, parents’ lack of concern. Parents did not indicate providing more help to boys or girls – they said that they’d help them equally. Recognizing this, SC worked purposefully with caregivers to encourage more engagement in their children’s education at home, even given their own perceived limitations in ability (e.g., not being able to read). Students also perceived receiving good support at home, which stands out from baseline and midline where they often reported having no support. For example, in all FGDs, multiple students said that they were able to seek outside help from somebody, including from older siblings, friends, and “study class teachers” or “home teachers” (personal tutors). Notably, six of the 24 student FGDs specifically reported that their fathers supported their reading at home, which was 89 | AIR.ORG LEARN Endline Evaluation Report not identified at baseline or midline. Only two of the 24 student FGDs had someone who said that they had nobody at all to support their reading at home. At the same time, even those students getting some support said that more support would be beneficial. Not observed at midline, multiple students mentioned needing a “study class teacher” that they had observed others able to access. Boys and girls perceived this report equally – none identified that boys or girls had particular challenges either during or after COVID-19 closures. Caregivers were not familiar with SC’s “I help my child to learn” tool in any of the communities. In five of the ten communities visited for qualitative research (none in Grand Bassa), when asked about this tool, respondents did not indicate any awareness of the tool, but rather mentioned a radio program that encouraged them to help their children to read: “It is good because we need to focus on our children so we can benefit from them tomorrow… I have used it by sending [the children to school] and focusing on their lessons.” – (Caregiver, River Gee) One caregiver in Grand Gedeh described the content of it in some detail that suggested it encouraged parents to become more innovative in the ways they could help their children, “Yes, I heard of it. If you don’t have chalk or the materials at home you can stone from the ground. If you want to draw, you can use char coal. It is call local materials. We are benefiting from it. It makes the children busy at home. A child taught her mother about the local material, and she was happy. Save taught us and we are passing it on to the children in the community.” – (Caregiver, Grand Gedeh) 5.3.2.3 Literacy Champions Are Trained, Engaged, And Effective Among the communities visited for qualitative research and students interviewed at endline, there appeared to be substantially more parent engagement and efforts to help children do schoolwork outside at home. Students also reported notably more often than at midterm that they were seeking help from older siblings or friends. Relatively few boys and girls said that nobody reads with them, which was commonly reported at baseline and midline. Despite these improvements in having resources, students often mentioned needing additional support such as a “study class teacher” (a personal tutor paid for by the family) or additional in-school and after-school support to achieve even more. As such, the reading clubs, camps, and other out-of￾school literacy report provided by LCs were generally welcomed by students and parents in the LB communities. This is notable, given the challenges SC had with maintaining LCs as of midline, when LB communities were largely indistinguishable from non-LB communities in terms of perspectives from students about resources to support their reading. 90 | AIR.ORG LEARN Endline Evaluation Report LCs were also generally positive about their ability to help children age K to Grade 2 learn to read; as one said, “The learning aspect is the best because right now 80-85% of the children in the summer reading club and in the school can read well. The books have helped the children to read. It boosts the children reading.” – (Literacy Champion, Grand Gedeh) However, the LCs noted that not all children benefitted, sometimes because parents did not send their children to those activities, opting instead to have them work on the farm, or because they did not have adequate transportation and/or lunch during those extra activities. Still, LCs worked to convince parents of the importance of the sessions, as one recalled, “At first it was difficult because some parents were discouraged. We started to encourage the parents. The first time we had 180 children in the community, because of that the parents were encouraged. The children could recite the words and they are improving and because of that the parents are happy.” – (Literacy champion, Grand Gedeh) As mentioned earlier, SC made efforts since midline to maintain LCs in communities through better supporting their needs while also working with local education officials to limit transfers. At endline, there has been evident progress toward successfully encouraging LCs to stay through SC’s work engaging more regularly with LCs and teachers to hear their concerns and act on them. For example, more training sessions were added per year because of LCs request for more regular training and this was well-received, as one Literacy Champion said, “The children are not the only one learning, myself I’m learning. I have been trained through workshops and it has built my knowledge. I am able to manage the children.” – (Literacy Champion 2, Grand Gedeh) There was also progress made to convince DEOs and CEOs, via dialogues and work within regional “literacy clusters” in which SC conveyed the importance of supporting teachers, to not transfer teachers trained by LEARN them after their training. This was said to be especially important in River Gee, where teachers tended to not want to be placed or stay. 91 | AIR.ORG LEARN Endline Evaluation Report LCs identified several challenges to their work. All eight LCs said they did not feel adequately compensated for their work, which was demoralizing and had led to other LCs leaving their position. LCs in two communities mentioned not getting their expected compensation (rations) on time. Some had received nothing at the time of the evaluation for work they had done in the previous summer’s camp. Even those who had been compensated on time said that the amount was inadequate and cash or greatly increased rations would be more appropriate. Interestingly, in one LB community, girl students described a for-payment study class that they (being older students) could take. This group was also not aware of the existence of LCs in the community, though they were present. 5.3.2.4. Access to Books Remains Sporadic Access to books at home and in school, as at midline, remained sporadic. Book borrowing was not fully established at the time of endline, so a system wherein students were bringing books home was not necessarily expected. The endline research found that boys and girls reported often reading their notes at home because of lack of other reading materials at home. Though the borrowing system was not expected to be in place everywhere, books were expected to be available at schools (primarily through USAID’s Let’s Read program) and also in reading clubs via Literacy Champions who were provided with books and other resources to help produce local reading materials to use with students. While some communities indicated having access to books and resources through teachers and LCs, all communities mentioned the need for more learning materials, and books in particular. To deal with this challenge, LCs described how SC trained them, and provided materials for them, to create reading resources for they and students to use, given the difficulty in acquiring books for everyone. LCs explained how they developed their own materials to help teach children, and teach children how to make their own reading materials, as one said, “We developed materials (local materials) to make letters. We have also developed toys to help them. I have trained them how to read well. Some impacts is that student can now read fluently and they can develop materials on their own.” – (Literacy Champion 1, Grand Gedeh) Another said, “I developed some items and teach the children how to fix their own books… I was taught about some local materials like fixing flash cards, and how to teach it to the children.” – (Literacy Champion 2, Grand Gedeh) 92 | AIR.ORG LEARN Endline Evaluation Report In communities without LCs, there were reports from two different schools of students paying teachers for books, “I get my books from people that are selling it, my teacher collects my money to buy my books. I pay 150 to him.” – (Boy student, Grand Bassa) “My teacher can give me the book and I can pay $100 Liberian dollars for it.” – (Boy student, Rivercess) In another community, girl students reported paying for photocopies of books, “The principal can photocopy the reading book and we can buy it to go read. We can pay $20 Liberian dollars. The school don’t have reading books.” – (Girl student, Rivercess) 5.3.3.1 Students Lament Lack of Cleanliness in Schools At endline, students across all communities lamented about the lack of cleanliness of their schools, though some reported more of a severe situation than others. Interestingly, the situation did not appear better for the communities with SHCs as compared to the communities without them. It appears that, while SHCs existed, they were not reliably fulfilling their responsibilities. For example, girls in the combined package site in Grand Gedeh said, “P5- The campus is not clean, the wall is dirty, it is not painted, the toilet is not clean, no tissue is in the bathroom we can use copybook sheet to clean ourselves. P4- The school toilet is nasty and the school campus. They can’t take care of the campus. The grass is growing everywhere on the campus. The care taker is responsible to clean the campus but they can’t clean it good. P3- the campus is bad off. The zinc [roof] is leaking, so when it rains, the class can be wet, the class floor is burst, so when it rains, mud can be in the class room. We have caretaker but she can’t clean the campus, we can sometimes clean the campus ourselves. When we tell the teacher, they can tell us to clean it ourselves. Boys and girls can use the same bathroom.” – (Girl students, Grand Gedeh) Importantly, this above quote reveals a disconnect between the girl students’ understanding of the role of SHCs, and what the teachers say is in place, 93 | AIR.ORG LEARN Endline Evaluation Report “The health club had been active. Save told us to prepare our plan and we did that. They said that they need 30 students, 15 males and 15 females, they give us T. shirt. Every Monday and Friday we can clean the campus, the bathroom and on devotion line, we can carry on awareness about personal hygiene…. it’s effective because the campus is always clean and the toilet is always clean” – (Teachers, Grand Gedeh). Similarly, girls in Rivercess said, “The bathroom is dirty because the flies can be all over the bathroom and teacher are part of the school health committee [School Health Clubs] and the students are responsible to clean the campus, both boys and girls are part of the health club. The teacher can select student from the 4th to 6th grade to clean the campus on some Friday but these few days, the teachers had not told us clean to clean it.” – (Girl students, Rivercess) However, the principal in that same school told a different story, “The school has a health club. they are responsible to make sure all the handwashing buckets have water, clean the bathroom and put water in in it and also encourage their friends to wash their hands. There are 25 persons in the health club, they are divided into five sections, five persons work every day. Both boys and girls are part of the health club, 10males and 15 females and each have their heads. Yes, it is effective and I am involved. I make sure that all the handwashing station has water, and especially when the children use the bathroom. Yes, they can share it with their friends and community. We are planning to go in the community to carry on health awareness.” – (Principal, Rivercess) Meanwhile, at endline, PTAs were relatively inactive with respect to WASH and other SHN activities. For example, in FGDs they generally did not describe their work as it related to SHN aside from occasionally mentioning tasks to clean up the school. They never mentioned work related to nutrition. Other stakeholders also commented that PTAs were somewhat inactive in terms of WASH activities and none noted any work PTAs did as it related to nutrition. Limited PTA engagement was also found at midline, though it is especially notable to see this at endline given that PTAs’ engagement in other domains (gardens, feeding) was relatively stronger than it was at midline. Also at endline, SHCs appeared to have become less active than they were at midline, when they were largely applauded by students and school staff. At endline, the work of SHN champions was rarely described. All the principals of the schools with the SHN intervention were able to clearly articulate these processes, so it seems as though the main challenge is in effectively mobilizing the students to do it reliably. 94 | AIR.ORG LEARN Endline Evaluation Report The findings from midline that showed SHCs being highly engaged in school cleaning activities, and relatively fewer concerns from students regarding access to basic cleaning and water materials as compared to endline. This may be a reflection of the COVID-related sanitation measures that were more stringently applied at midline (in the period so soon after the initial 2020 COVID closures) as compared to endline. Such measures, at endline, appear to have been reduced and the SC-specific impact is more clearly limited. The challenge that schools had in attaining a sanitary environment because of challenges accessing basic materials like soap and buckets may have negatively impacted other SHC work on hygiene behaviors (e.g., handwashing) and even more so on nutrition, which the quantitative data showed no improvement on at endline. Indeed, in FGDs and KIIs, nutrition education was almost never discussed as it related to LEARN activities. 5.3.3.2. Health and Nutrition Practices for School Feeding are Established but Inadequate The qualitative data suggest that cooks and storekeepers felt adequately trained for their duties, but face challenges in reliably applying those practices. For example, they were able to give examples of some of their responsibilities that they said they regularly practiced, “Save the Children gave us training how to prepare the children food. They say we should prepare the children food and it must be done [cooked]. The food must not burn. Save the children told us to cover our hair before cooking and we should not wear our ring while cooking. They said maybe while cooking the ring might fall into the food and mistakenly given to any child. Save the children also told us that if you are sick you shouldn’t come to cook. They also said that we should also wear our nose masks when we are sharing food. That our kitchen must be clean at all time and we shouldn’t keep dirty water in the kitchen. The training has been enough because all that we are supposed to do is what they told us which I have just explain.” – (Cook, River Gee) Similarly, another cook said, “Yes, I received training about cooking. They said, when get to the kitchen, I should not allow the students to be there because it will be risky for them, there should be no toilet around the kitchen, cover food because of the flies. The flies will sit on the toilet and it will also sit the food and the children will get sick when they eat the food. We should be clean before coming to work and I think it was Save the children. Yes, it was good and I am doing all I was taught.” – (Cook, Grand Bassa) 95 | AIR.ORG LEARN Endline Evaluation Report However, six of the cooks pointed out that it was not always easy to adhere to all food safety guidelines, given challenges with adequate materials and kitchens. Still, they all explained how they went out of their way to try to alleviate the problem to keep the food safe. For example, one cook said that she lacked a table in the kitchen to serve the food, so she fixed her own structure with bricks and wood to prevent serving food on the floor. Two other cooks explained that the clean water source nearby was spoiled, so they walked extra distance to carry water back to the kitchen. One cook explained that, “The kitchen has nothing around it, so the goat and cow can go and scatter the place. The goat can pupu and pepe [defecate and urinate] under the kitchen, I have to take broom and sweep every morning. The principal can complain to the community people that have those cattle to control them but they promise to fence the kitchen. The kitchen was built long time ago but they have rebuilt it yet, we fix the fire halt with dirt and it can always spoil so we have to fix it all the time.” – (Cook, Grand Gedeh) In one rare example, a cook shared that she was asked to serve expired food for lack of any non￾expired food available (it is unclear from the quotation whether or not she did actually cook it), “If the food expires, we are instructed by the teachers to cook it.” – (Cook, Grand Bassa) Beyond these examples, across all interviews, there is no additional evidence of cooks violating food safety protocols either knowingly or inadvertently. 5.3.4 Codes of Conduct are Well-Known, but Not Necessarily Agreed with or Followed Little had been done by MOE as of the endline on revising school codes of conduct, so the analysis of the work in this regard is limited to SC’s sensitization around existing codes of conduct. Respondents across all groups reported that they were aware of the content: teachers and staff must not partake in bribery, abuse, and rape, must not have relationships with students, commit corporal punishment, discriminate by sex, discriminate against children with disabilities, commit fraud, have persistent absences, use humiliating language/lack of respect, use drugs and alcohol use, and practice favoritism. Midline showed that despite a broad awareness of the content in the code of conduct, there was an evident degree of either misunderstanding or blatant disregard for it. 96 | AIR.ORG LEARN Endline Evaluation Report At endline, students in all FGDs were able to articulate a number of the components of a school code of conduct, including that teachers should not beat students, teachers should not have relationships with students, teachers should not accept payment for grades. However, there were multiple reports across the FGDs with students of beating (and other forms of physical punishment) and payment for grades. However, as at midline, students continued to report instances of physical punishment including beating and being given manual labor tasks, “The thing I don’t like the school for is because they can beat on students and give you hard punishment to do when you misbehave; [another student] yes oh, they can beat too much and they can give you large grass to hook.” – (Girl student, Grand Gedeh) There were also clear indications that students continue to accept that corporal punishment is acceptable, as when one girl said, “I like how the teachers can behave. They can be treating us fair. For example, if some people are causing noise in class and the teacher beats them some of the students can say the teacher is not treating them fair because the teacher didn’t beat the whole class. So, the teacher can only beat those who did wrong.” – (Girl student, River Gee) In one community, students said physical abuse had become less prevalent, “Teachers used to beat on us before but now they are not beating on us again.” – (Girl students, River Gee) Other non-physical but harmful punishments included being sent out of the classroom as a form of punishment, in some cases for two weeks, and not being given a chance to make up the work, “When we are taking test and you do something, they will send you outside to hook grass then you will miss your test and they will not give it to you.” – (Girl student, Grand Gedeh) 97 | AIR.ORG LEARN Endline Evaluation Report As at midline, students generally understood that a process for reporting a teacher for a violation of the code of conduct would involve elevating it to a principal or DEO, or telling one’s parents, who would then speak to the principal. Others noted that certain violations should involve the police. However, more frequently than at midline, students noted that reporting teachers could result in repercussions and therefore they feared doing so, as girls said, “When we find out the everyone is involved, we should take the complaints to the police station or carry the complaint to our parents. And when we carry their complaints, they will beat on us and they will not give us good grades and they will make you shame in front of everybody and take you to class to class and ask the others students to boo at you.” – (Girl student, Grand Gedeh) Students also said that reporting teachers often led to nothing, for example, “We can put our complaints in the box…, we can feel comfortable [but] the last time we reported a teacher pressing tete (breast) but he denied it to the principal and nothing was done about it. The teacher is still around.” - (Boy student, Rivercess) In two groups, students said that they feared reporting to the principal because if they did not have adequate evidence of their claim, they’d be given an “NTR – Never to Return.” Students in the schools in Grand Bassa and Rivercess mentioned a complaint box from Mercy Corps where they could anonymously report that helped them to “be brave” or as a girls’ FGD said, “The boys and girls can be afraid to carry their complaint [to the principal] because they will fail us so we will just write it put it in the box.” – (Girl students, Rivercess) 5.4 Sustainability To assess the sustainability of project interventions, the research team asked stakeholders about necessary inputs and system required for sustainability, as well as factors likely to influence sustainability. Interviewers also asked respondents to recommend strategies for continuing activities after the project ends. 98 | AIR.ORG LEARN Endline Evaluation Report Exhibit 45. Sustainability-Related Indicators and Findings Category Activity Indicators Key findings Enhancing school quality through school feeding and teacher support PTA activities PTA ownership of tasks; adequate financial and human resources PTA activities will likely continue given the strong momentum they have gained at endline. Support from government will be helpful. School Feeding School and local MOE ownership of task School feeding will not continue without outside support to provide the major commodities needed (e.g., rice). With those commodities, though, schools (through PTAs) demonstrate the ability to carry out the school feeding including providing supplemental condiments, on their own. School Gardens School and PTA ownership of task PTAs have shown enthusiasm for gardens and will likely continue trying to establish or expand gardens to supplement school feeding. However, they may struggle if basic materials are not provided and they are unable to raise adequate funds at the school-level. Improving literacy Literacy champions / reading clubs Commitment of Literacy Champions to stay Without financial support (e.g., provided by the PTA), Literacy Champions will likely not continue in their role. However, the culture of reading that has been established at the household level may remain. School health and nutrition SHCs and SHN Champions Degree to which they will sustain after LEARN There is little evidence that SHN champions and SHCs will sustain after LEARN, given they have lost much momentum by endline, and major barriers include inadequate materials. Meanwhile, PTAs are not supporting this component as actively as others. 5.4.1 There Have Been Successful Efforts Toward Increasing LEARN Sustainability SC has been successful in efforts to engage Ministry of Agriculture, Ministry of Education, and Ministry of Health to support activities and eventually take on some degree of ownership to enhance the likelihood that components of LEARN will be sustained. This work had become increasingly urgent as LEARN came to an end, and SC was clear from midline that getting government ‘on board’ had been a challenge. SC noted that it was difficult for government staff to understand the program’s multiple components and the need to convey technical information on various types of interventions, 99 | AIR.ORG LEARN Endline Evaluation Report “Increasing government understanding of LEARN has been a huge challenge. Mainly claimed to not understand what it [LEARN activities] was aiming to do, the combined package in particular.” – (SC National Staff) At endline, SC provided frequent technical support to Monrovia-based and field-based government staff to mitigate this challenge, in some cases simplifying it into two main objectives: (a) increase learning, and (b) increase safety. Another effective approach was inviting staff on monitoring visits so they could see LEARN activities. As a result, according to senior implementing staff, “government has been more on board recently.” With this increased understanding of what LEARN is, SC engaged in more purposive interaction with government, having regular meetings and engaging in joint work-planning sessions during which government was said to be responsive. Government had also been providing ‘token monetary support’ to school feeding interventions to show their commitment to school feeding. One of SC’s hopes for the launch of LEARN II (June 2022) is that it would publicly show and confirm the government’s role, an important milestone that has been achieved because of the work done as part of LEARN. Also, interviews with one CEO and eight DEOs show increasing understanding about what LEARN is doing, and agreement that government has an important role in sustaining the work. For example, one CEO indicated that there was a clear intention and multiple efforts on his and his staff’s part to follow-up to check on teacher and student performance, but noted challenges with accessing communities and often, when they do reach a community, the school is not open because of some other event, “Some of the children are not available because of sometimes community’s issues- like when a person dies in the town, sometimes the school will be closed for two to three days and they will not meet the children, and so the time you supposed to take to do follow up will be wasted… Additional support is getting the DEO fully involved by empowering them to be able to monitor the project.” – (CEO, Grand Gedeh) 5.4.2 School Feeding Will Not Be Sustained Without Direct Support Despite the popularity and successful rollout of the school feeding program, there was widespread recognition that there would be challenges with sustainability of school feeding at the end of LEARN without SC’s direct support. For example, a teacher said, 100 | AIR.ORG LEARN Endline Evaluation Report “The program will be difficult to sustain because the back-up will not be there. Even if the parents want to help, it will not be an inch to what Save the Children is doing.” – (Teacher, River Gee) Similarly, a storekeeper said, “I am not sure if they will continue because to cook for whole group is very expensive and they don’t have the money.” – (Storekeeper, Grand Gedeh) Government agreed that sustainability would be challenging but saw opportunity in terms of ensuring that community members had some ownership of the work, “No, it will not continue because they are not taking ownership and what is needed is to encourage the communities and motivate them to take it as their own and take ownership…Get the communities involved and not individuals… To get the larger communities involved to feel the project as their own and not put special people in front of it.” – (MOA School Feeding Division 1) “If we continue with the schools according to the plan we have and make sure they work in line with us, I am sure they will continue after the project ends. The schools garden will continue especially the planting of the eddoes, cassava and potatoes because the students are having interest in looking at the crops and learning about them and also benefiting from its produce, but it will just be some communities that will continue this.” – (MOA School Feeding Division 2) 5.4.3 Literacy Champion Activities Will Not Be Sustained, But Home Literacy Environment Will Endure At endline, a key challenge threatening the sustainability of LEARN was the frequent complaint about compensation from LCs, cooks, and storekeepers, who are, through LEARN, paid in food rations. Rather, cash or substantially more take-home rations were requested. All eight LCs interviewed (two per each LB community) expressed disappointment with their compensation (rations) and would have preferred either more rations or cash. While some did receive money from PTAs who collected funds for school volunteers, it was not enough for any of them. Four of them said they had no intentions to continue in their role unless they received more compensation going forward, 101 | AIR.ORG LEARN Endline Evaluation Report “I will continue If my name goes back on government payroll if not, I will leave the work and be in the community teaching the children in the form of study class for soap money. I already have the knowledge from LEARN… [Going forward] for LEARN, they should give me incentives [cash] or add my ration up to 10 kilos with oil and beans.” – (Literacy Champion 2, River Gee) Another agreed to be a LC without realizing there was no financial compensation. Two LCs explained that previous LCs had dropped from their positions because of the limited compensation; they remained because they cared about the children’s education, “I have been Literacy Champion for 5 months now, I came to be selected because those that were originally train as Literacy Champions drop because they said no money is it at least to buy soup, no token, so they drop that’s how I told my registrar for us to continue their role because the program is a good thing for the children and I don’t want them to be sitting down doing nothing.” – (Literacy Champion 2, River Gee) Parents and teachers also recognized this need, “I recommend that they should give the teachers that are part of the Literacy Champion with money for their stipend instead of the food they give.” – (Caregiver, Grand Gedeh) A promising finding from endline was the report from six communities that PTAs were mobilizing to collect money to pay volunteer teachers, including some LCs, cooks, and storekeepers. 5.4.4 School Health and Nutrition Activities Will Continue to Lose Momentum Without Direct Support SHNs and SHCs appear to have lost momentum at endline, suggesting their work may not endure past LEARN. At midline, students, teachers, and principals agreed that it was critical to have a clean environment, and the SHCs were active in all schools visited in helping to achieve this. They regularly took part in cleaning the school and were said to be active in teaching others about basic hygiene practices. It was thought, at midline, that these types of behaviors would likely endure where SHCs continued to be encouraged because the SHCs endured post-COVID even in school communities without active school health and nutrition intervention from SC. However, endline findings suggest that SHCs may have been at least partially a reflection of the post-COVID-closures phase when more stringent WASH-related measures were common across Liberia. 102 | AIR.ORG LEARN Endline Evaluation Report 5.4.5 Codes of Conduct Will Unlikely Be Applied Without Increased Sensitization The 2013 MOE school code of conduct is institutionalized; however, the degree to which the specifics of the rules and regulations are understood and respected is questionable given some open admissions of violations by principals and teachers. To be sustainable, such rules and regulations not only need to be fully understood, but fully agreed upon by stakeholders who will continue to push for adherence to these protocols. There is still work to be done in this regard. 103 | AIR.ORG LEARN Endline Evaluation Report 6. Conclusions This report presents the endline findings of the impact and project evaluations of the LEARN project in four counties in Liberia: Grand Bassa, Grand Gedeh, Rivercess, and River Gee. While there was clear improvement from baseline to endline in school and home literacy environments, SRGBV, and handwashing knowledge, literacy results were mixed and, in some cases, outcomes worsened from baseline to endline.32 Generally, there was high variation in the outcomes across the four counties. This section summarizes key findings related to the main research questions in two categories: (a) with respect to the key project interventions within the McGovern-Dole results framework and (b) based on the five evaluation criteria (i.e., relevance, effectiveness, efficiency, perceived impact, and sustainability). 6.1 Key Outcomes, by LEARN Interventions Several key reading outcomes worsened from baseline to endline, and these outcomes did not always vary uniformly between counties. Like at baseline, at endline students were mostly successful at identifying letters but struggled beyond that to identify full words. While 85% of the Grade 2 students could identify letters at endline, only 23% could identify words, and only 11% were classified as readers. These outcomes generally decreased similarly across counties except for students classified as readers, whose proportion decreased the most in Grand Gedeh and River Gee Counties, by 14 and 10 percentage points, respectively. Improvements in literacy outcomes among readers compared with non-readers point to heterogeneous impacts of the COVID pandemic. For example, the percentage of words read accurately improved greatly from 10% at baseline to 51% at endline, and fluency (words read correctly per minute) doubled from 11 to 22, on average. Reading with comprehension increased overall from baseline to endline, and this result was driven mainly by improvement among boys. At endline, 4% of students could read with comprehension, a statistically significant increase over baseline (1%) at the 1% level. 32 There are also potentially unintended consequences of LEARN programming on children who do not know letters to begin with. Specifically, we find that the combined package does better for reading comprehension scores than the base package. One possibility is that knowledge of letters is an indicator that is relevant for readers and non-readers alike. The combined package may benefit more children to become readers but may not be geared toward helping the average Grade 2 student who does not know letters to become more literate in terms of knowledge of letters. Furthermore, when additional program activities are added in a combined package (e.g., school gardens or literacy boost training for teachers), teachers end up focusing more on reading and helping the readers or help with school gardens but may give less time to help the average student improve their knowledge of letters. Alderman et al. (2012) find evidence of unintended negative effects for school meal programs on at least one learning outcome, which they attribute due to potential time related demands of some of the school meal activity programs. Therefore, this effect can be thought of as a substitution effect, where fewer resources are being devoted to children who don’t know letters to learn letters and more are being spent on those who already know letters or can read. 104 | AIR.ORG LEARN Endline Evaluation Report Interestingly, there was a significant decrease in the proportion of students who could correctly answer an evaluative question related to the text. This result decreased across all counties and may point to a loss in critical thinking skills. The impact evaluation showed that provision of combined packages (school meals plus literacy boost and school health and nutrition interventions) led to improvement in reading comprehension scores, particularly among boys. Boys in schools that receive the combined package are 6.3 percentage points more likely than boys in comparison schools to read with comprehension (p < 0.05). In contrast, we do not find any similar impact on reading comprehension skills for girls who receive the combined package. When we examine effects on any of the intervention schools (base or combined package), we find a 1 percentage point increase in reading comprehension for girls relative to the control group, but a 4 percentage point increase for boys. This effect was likely driven by the ability of combined packages to increase readers among boys; null effects were found on readership among girls. We also considered students who could name at least 90% of the 26 letters of the English alphabet as “letter knowledgeable.” We did not find any significant effects from either the base or combined intervention on knowledge of letters. Overall, the combined package may help more children become readers but not help the average Grade 2 student who does not know letters to become more literate or a reader. Encouragingly, home literacy activities generally improved from baseline to endline. There were large variations between counties, with the largest proportion of students in Grand Gedeh and River Gee Counties reporting literacy activities at home in each category. There was also an increase in the amount of reading materials that students had access to at home. The proportion of students reporting that they have no reading materials at home decreased by 6 percentage points to a low 9%. The proportion of students with access to holy books and textbooks at home also increased from baseline to endline. Changes were generally uniform across counties. Knowledge of a balanced diet did not change from baseline to endline. Only 3% of students stated that they knew the definition of a balanced diet at endline, and of those, only nine students could successfully identify all three components of a healthy diet. The impact evaluation did not show any significant change in knowledge of a balanced diet due to any of the interventions, perhaps due to the small sample sizes. Handwashing knowledge only slightly increased since baseline, with knowledge levels varying by county. Handwashing knowledge significantly increased in both Grand Bassa and Grand Gedeh and significantly decreased in River Gee and Rivercess between baseline and endline (p < 0.10). Overall, self-reported handwashing behavior improved—albeit marginally by 6 percentage 105 | AIR.ORG LEARN Endline Evaluation Report points—relative to baseline (p < 0.05). The data show statistically significant improvements in Grand Bassa and Grand Gedeh (p < 0.10) and decreases in River Gee and Rivercess. Grand Gedeh had the largest proportion of students with handwashing knowledge and recommended practices at endline, registering a 21 percentage point improvement in self-reported practices over the low 2% level at baseline. Students were largely aware that rules exist for how teachers should treat students at school, and this high level of awareness remained constant from baseline to endline. The county-level data showed large differences, with nearly all students in Rivercess and River Gee showing awareness of these rules and the proportion decreasing in Grand Bassa County by 7 percentage points (p < 0.01). The proportion of students who had improved knowledge of SRGBV issues and a willingness to report increased significantly across all counties besides Grand Bassa, and differences were negligible by sex. We also asked Grade 6 students whether they agreed or disagreed with a series of five statements to find out if they held less biased and better perceptions of gender norms (assessed by whether they disagreed with at least four of the five gender stereotypes statements). Sixty percent ofstudents reached the threshold at endline, compared to only 48% at baseline, showing significant improvement in students’ perceptions of gender norms. Nevertheless, the data showed significant variation in perceptions of gender norms by county: A much lower percentage of students in Grand Bassa disagreed with at least four out of five gender norm statements compared to the other counties, especially River Gee. 6.2 Key Findings with Respect to Evaluation Criteria Below, we present the qualitative findings at endline by five OECD-DAC criteria and by outcome area. These criteria include stakeholder satisfaction and project alignment with Liberian context and beneficiary/stakeholder needs (relevance), steps taken to maintain the efficiency of project operations amidst various external and internal factors, the degree to which project activities were effective and had a perceived impact on LEARN beneficiaries at endline, and the degree to which inputs and systems required for sustainability are present at endline. 6.2.1 Relevance Enhancing School Quality (School Feeding, Teacher Attendance, and Teacher Performance) to Increase Enrollment, Attendance, and Achievement. PTAs serve not only as an important mechanism for communication between parents and teachers but also as a way for parents to mobilize other parents and community members to support school initiatives. PTAs can help to supplement some of the services that are not always reliably provided by government and whose provision by activities like LEARN will not continue indefinitely. 106 | AIR.ORG LEARN Endline Evaluation Report  School feeding and take-home rations. Students and caregivers overwhelmingly agreed on the importance of education for both girls and boys. Students self-reported enjoying school and had high expectations for what they could achieve in the future if they were able to continue their education. Local stakeholders and beneficiaries agreed that the school feeding portion of the project aligned with the needs of their communities, where parents are often faced with the choice between keeping their children out of school so they could engage in livelihood strategies and sending them to school hungry. Take-home rations for girls were also appreciated, though it may be increasingly important to remind communities about the rationale behind only providing girls with rations. Gardens and larger farms would be a major asset in supporting school feeding interventions, in addition to providing PTAs with a source of reliable income that would alleviate their need to ask parents for money to support various school projects.  Teacher support and training. Students and caregivers agreed that the provision of a high quality of education depended on having teachers who attended regularly, were well trained, and were dedicated to the work. At the same time, teachers often faced challenges in their work, such as not being paid on time (or at all), feeling unsupported by local education officials, and not receiving adequate training. Consequently, LEARN’s work to enhance school feeding and teachers’ working conditions has been highly relevant to improving student enrollment, attendance, and achievement. Improving Student Literacy.  Out-of-school literacy support. Improvement in teachers’ ability to teach literacy was a welcome advance in overall school quality, especially given students’ general interest in and enthusiasm for education and literacy. However, LEARN’s goal of providing students with additional out-of-school support met a serious obstacle, primarily as a result of the challenge in convincing teachers to take on the role of Literacy Champion without additional compensation. As such, the strategy to identify volunteer Literacy Champions in order to provide out-of-school support and resources was highly relevant.  Home reading support. Supporting caregivers and others in the community to provide support to students at home also helped to meet a critical need, especially in the non-literacy￾boost communities that did not have access to reading clubs or camps. Providing illiterate parents with strategies to support reading at home, was important, as parents often lacked the confidence to support their children or believed it required more time than they had. While there was no knowledge in the communities visited for qualitative research about the “I Help My Child to Learn” tool, caregivers did note that they’d heard radio jingles encouraging them to actively support their children’s education. 107 | AIR.ORG LEARN Endline Evaluation Report Reducing School-Related Gender-Based Violence. Despite widespread knowledge about the existence of a school code of conduct, including what items were included in the code, SRGBV remained a problem in schools across Liberia, indicating that knowledge does not translate into agreement with the code or improved adherence. Also, students at endline often indicated fear of retribution for reporting a teacher’s infraction. The potential of such safety concerns to limit attendance and achievement is clear. Multiple interventions by partners and the government have tried to overcome these challenges without much success, highlighting the clear relevance of SC’s work here. Improving School Health and Nutrition. Students and teachers emphasized the importance of having a sanitary environment with functioning infrastructure, which was not always available at their school. SHN activities to promote healthy practices were evidently necessary given the relative lack of knowledge that students had on these topics, but in the context of school environments sometimes lacking necessities such as soap and water, healthy practices promotion became secondary in importance. Overlap With Other Interventions: While the topics and approaches covered by LEARN are relevant to the needs of students and stakeholders in the LEARN communities, it is important to consider what other related interventions could be implemented in these school communities to increase their relevance. For example, Bridge Academy was observed to be operating in at least one LEARN school. 6.2.2 Efficiency Enhancing School Quality (School Feeding and Teacher Attendance and Performance) to Increase Enrollment, Attendance, and Achievement.  PTAs. Because many PTAs were said to be inactive at midline, SC conducted community outreach, including reinforcing and clarifying PTA roles and responsibilities, sensitizing PTAs to the importance of engaging with increasing numbers and types of people, and requesting the MOE’s PTA Engagement Division to better support PTAs.  School feeding. Since midline, improvements in road conditions, food provision, and the partnership with the MOE meant that food provision was regular and relatively well organized. Challenges remained, including large- and small-scale food theft (e.g., within the school communities) that occasionally interrupted the supply of food, but steps were, such as building secure warehouses and reprimanding community members caught stealing food. Also, kitchens did not always receive an adequate supply of utensils, and there were reports of PTAs not providing adequate condiments and dishes. That said, by endline, LEARN had made progress in working with PTAs to refresh them on their specific roles under LEARN and to support them in getting more parents involved in carrying out projects and raising funds. 108 | AIR.ORG LEARN Endline Evaluation Report  Gardens. Relatedly, LEARN worked more purposefully with PTAs to help them establish the school gardens meant to support school feeding activities. This required convincing parents to devote time to creating and tending the gardens. The Ministry of Agriculture admitted to having trouble distributing gardening materials on time, with the result that around 25% of school communities missed the growing season in 2022. Improving Student Literacy  Literacy Champions. Acknowledging first the problem of teachers who were reluctant to take on the role of Literacy Champion without compensation, along with the problem of high turnover among those teachers serving as Literacy Champions, SC took appropriate steps, including being more supportive by identifying non-teacher volunteers to serve as Literacy Champions, and also providing additional training to Literacy Champions (teacher or volunteer) throughout the year, making regular monitoring visits to hear their requests, and ensuring regular provision of rations (the Literacy Champions’ form of compensation). SC also instituted other measures to encourage teachers in general to stay, which was particularly important in more rural and less well serviced areas, as teachers in these areas tended to try to relocate elsewhere. These measures included instituting teacher recognition activities and collaborating with local education government officials (DEOs and CEOs) to improve pay and working conditions for teachers. Importantly, SC was also able to implement the Summer Reading Clubs program in 2021, which had not been possible in 2020 because of COVID.  Books and learning materials. Through LEARN, SC and USDA aimed to leverage work done by MOE and USAID Read Liberia to provide literacy materials to all children and teachers in Grades 1 and 2, and training to all teachers. For example, LEARN provided all Literacy Champions with boxes of books and materials that they could use in their work. At endline, there were no reports of book banks being established, though Literacy Champions had some books that they could use during their work with students, but could rarely let students borrow them. Literacy champions also had been trained to develop their own local reading materials to address the limited book supply issue. Some students reported being asked to pay money to teachers to take books home. Improving School Health and Nutrition. LEARN faced challenges with SHN champions similar to those faced with Literacy Champions: They were sometimes transferred to a non-LEARN school after being trained. Also, SHCs appear to be less active than at midline and were focused on school cleaning activities and procuring access to clean water and soap and more than on nutrition. Reducing School-Related Gender-Based Violence. At endline, the school code of conduct was nearly finalized, though drafts of revisions were circulated and schools were re-introduced to them. SC utilized a social-behavior change approach in tasking community mobilizers with 109 | AIR.ORG LEARN Endline Evaluation Report sharing SC-developed Safe Schools Stories with teachers, parents, and students independently. These stories each had a lesson to teach about the context and rationale for parts of the TCOC, to raise awareness about the TCOC while also promoting appropriate behavior and response to violations. It is unclear whether these stories were rolled out only in Grand Bassa and Grand Gedeh or throughout the LEARN communities. Sharing the stories via radio programming was planned, though this had not been rolled out at endline and was to be considered for LEARN II. 6.2.3 Effectiveness and Perceived Impact Enhancing School Quality (School Feeding and Teacher Attendance and Performance) to Increase Enrollment, Attendance, and Achievement.  PTAs. While the midline found that PTAs were relatively inactive, interviews with PTA members and other teachers and caregivers across the 12 qualitative sites at endline suggest much improvement in the PTAs’ understanding of their role and the activities they were expected to perform. At endline, nearly all respondents across all communities said PTAs were active in carrying out their responsibilities for LEARN and also other non-LEARN activities (e.g., supporting payment of volunteers and large-scale school infrastructure projects). They continue to struggle getting more parents to be involved, often because parents do not see the benefit of joining a PTA.  School feeding. At midline, respondents of all types across communities were extremely positive about the impact of school feeding on attendance, learner focus and attentiveness, and family food security. At endline, some learners complained that the rations were becoming monotonous, and there were rare instances of the food supply being interrupted or condiments not being available. Take-home rations for girls were also viewed positively by many who recognized that girls faced unique challenges that made it difficult for them to stay in school. However, there were indications from boy students and other stakeholders that the focus on providing girls with take-home rations could lead to resentment, since many boys also face challenges.  School gardens. At endline, in all interviews, whether there was an established garden or not, parents and teachers recognized the potential benefits of gardens. To some extent, this can be attributed to SC’s work sensitizing PTAs and others about the important role that gardens could play in helping supplement school meals and helping PTAs raise money by selling produce. Eventually expanding gardens into farms could also help parents cover school￾related expenses for their children and discourage them from taking their children out of school to being working.  Parent engagement with teachers. At endline, many teachers continued to lament what they considered to be limited parent engagement. They did admit that there were indeed parents 110 | AIR.ORG LEARN Endline Evaluation Report who were very involved. Among the teachers commenting on who does engage with them, all but one said that it was generally the female caregivers who took the needed time. All caregivers, meanwhile, said that they engaged with their children’s teachers and felt free to do so, which is an improvement from baseline and midline, when some said they lacked time or confidence or otherwise felt uncomfortable talking with teachers. Improving Student Literacy.  Pedagogical approaches in the classroom. At endline, teachers reflected on multiple topics that they had learned about in recent trainings on teaching literacy, and students also recognized the different components of the instruction they received. This stands in contrast to midline, when there was little to distinguish literacy-boost-trained and non-trained teachers, project implementation was limited, and trained teachers were often transferred after receiving training. That said, it is important to recognize that the sample of students in FGDs may not have captured the types of students identified by the quantitative data as not having progressed since baseline (i.e., students who were not readers to begin with).  Parent support for literacy at home. At endline, there were clear indications that SC’s work with parents to encourage more engagement with their children’s education at home, even given their self-perceived limitations (e.g., lack of literacy), was effective. All caregiver groups reported that they tried to help their children as often as possible and regularly asked about how they were doing. Students also perceived themselves to be receiving good support at home. This stands in contrast to the midline evaluation, when many parents did not feel empowered to engage with their children’s literacy education because they were too busy or perceived themselves to have limited skills. Though there was a marked improvement, students nonetheless said they would benefit from additional support, in particular having their own “study class teacher” or “home teacher” (as a personal tutor).  Literacy Champions. The reading clubs, summer reading camps, and other out-of-school literacy activities provided by Literacy Champions were broadly welcomed by students and parents in the literacy boost communities. This is notable, given the challenges that SC had with maintaining Literacy Champions as of midline, when students perceived the reading support they were given as largely indistinguishable from the support in other communities. Literacy champions themselves were also generally positive about their ability to help children in K–Grade 2 to learn to read. However, Literacy Champions identified several challenges to carrying out their work. All eight of the Literacy Champions – teachers and volunteers – mentioned that they did not feel adequately compensated for their work, which was demoralizing and had the potential of leading to attrition.  Books. Access to books, as at midline, remained sporadic. All communities mentioned the need for more learning materials, particularly books. Literacy Champions had books that they could use to support their work with students, but they rarely reported having books that 111 | AIR.ORG LEARN Endline Evaluation Report they were able to lend to some students. This is not necessarily surprising given emphasis had not yet been placed on lending systems at the time of research. Recognizing the lack of resources, Literacy Champions described how SC trained them to create reading resources for them and students to use. Literacy champions explained that they developed their own materials to help teach children to read and also taught children how to make their own reading materials. Improving School Health and Nutrition. At endline, despite their increased work with school feeding and gardening activities, PTAs seemed to be neglecting WASH and other SHN activities beyond supporting occasional infrastructure and cleaning projects. SHCs, responsible for cleaning the schools, were not always active, and many students in SHN schools complained about the lack of adequate sanitation. In this context, where a clean environment is difficult to attain, it appears as though the attention of SHCs and SHN champions may have been focused less on hygiene behaviors such as handwashing and even less on nutrition (a finding supported by the quantitative data, which showed no improvement in these two areas). Indeed, in FGDs and KIIs, nutrition education was rarely discussed as it related to LEARN activities. Reducing School-Related Gender-Based Violence. Respondents across all groups reported being aware of the content of the school code of conduct, but, as at midline, there was an evident degree of misunderstanding or disregard from both students and teachers. As at midline, students generally understood that a process existed for reporting a teacher for a violation of the code of conduct, but more frequently than at midline, students noted that reporting teachers could result in repercussions and that therefore they feared making a report. Students in Grand Bassa and Rivercess reported use of a complaint box that helps them to “be brave” in reporting, but teachers were said not to have been punished for complaints made. 6.2.4 Sustainability Enhancing School Quality Through School Feeding and Teacher Attendance and Performance. Endline data suggest that schools would not be able to continue feeding students after the program concludes, even where there are school gardens, which currently are only large enough to supplement food that is provided to the schools. Also, in many school communities, school gardens are yet to be established at all. Commodities from other sources are essential, and SC is working to secure commitments from the government to try to achieve sustainable food provision. Meanwhile, PTAs continue struggling to provide cooking materials, and kitchens regularly lack basic equipment. With the enhanced role and work of PTAs, along with larger gardens and farming cooperatives, including production of Power Gari, as planned for LEARN II, 112 | AIR.ORG LEARN Endline Evaluation Report sustainability may be more likely.33 With respect to teacher quality, SC’s advocacy in favor of teachers may have had some impact on local MOE representatives, but the evidence is unclear as to whether this will lead to an improvement in the government’s attention to teachers’ working conditions. At the same time, teachers will continue to impart the knowledge that they have gained through LEARN trainings, though the importance of follow-up training has been made clear. Improving Student Literacy. The literacy component of LEARN has made huge gains in potential sustainability. First, Literacy Champions have made great progress and gained popularity while working as volunteers. However, they all indicate that working for food rations is not ideal. The stipends that some PTAs have collected may help convince some to continue their work, but their persistence is not guaranteed. More promising is the role that parents and others in the community have begun to play in supporting their children’s reading education, including identifying or making their own reading materials. This culture of literacy is likely to be maintained, especially given that it was built upon a strong foundation of valuing education and high student ambitions for success. Improving School Health and Nutrition. Stakeholders and beneficiaries agree that having a clean environment is critical, and much collaborative work is done to achieve this. However, midline findings indicating high SHC activity may have been mainly a reflection of the COVID-related school closures, when more stringent WASH-related measures were common across Liberia. SHNs and SHCs appear to have lost momentum at endline, suggesting their work may not endure past LEARN. Reducing School-Related Gender-Based Violence. The 2013 MOE school code of conduct is institutionalized knowledge; however, the degree to which the specifics of the rules and regulations are understood and respected is questionable given open admission of violations by principals and teachers. To be sustainable, such rules and regulations need to be not only fully understood but fully agreed upon by stakeholders who will continue to push for adherence to these protocols. There is clearly still work to be done in this regard. 33 However, with additional responsibilities for PTA members, some program activities could suffer. For instance, promoting knowledge of balance diet suffered in the current program and may continue in the future. 113 | AIR.ORG LEARN Endline Evaluation Report 7. Recommendations Below, AIR presents recommendations based on key project outcomes, limitations, and lessons learned from the endline evaluation. Though LEARN is ending and most of the recommendations cannot be implemented at this point, they may be relevant to LEARN II and similar programming in the future.  Enhance literacy among non-readers. Further explore which types of students work with Literacy Champions or engage in other literacy boost interventions (e.g., reading clubs) to determine whether those who are already readers tend to seek this support more often. If existing readers tend to seek this help and non-readers do not, this could help explain why students who are already readers tend to improve while non-reading students do not. It may be beneficial to target non-readers or facilitate access to non-readers to literacy boost activities. Alternatively, if non-readers are being supported with such activities but still do not improve, then providing customized instruction based on their skill level may better help these less advanced students to progress.  Closely monitor MOE-hired teacher trainers to learn more about what they are focusing on as it relates to literacy. This will help SC to both contextualize literacy outcome findings and provide insights to help intervene where it seems necessary to better improve the desired outcomes (e.g., curriculum reform).  Continue using the option of mobilizing volunteer Literacy Champions instead of tasking teachers with the role but clarify with the volunteers the reasons their position is not, and will not be, compensated. At endline, volunteer Literacy Champions were effective and motivated, but did express some concerns with payment in take-home rations only.  Produce innovative and locally made reading materials. Continue empowering students and parents to create their own reading materials when there is a lack of content to read. Literacy champions have provided good examples of how children can use locally made materials (e.g., flashcards and transcribed stories narrated by community members) to enhance literacy.  Continue to work with the government to better support and maintain teachers and other community volunteers supporting education initiatives (e.g., volunteer Literacy Champions). Advocacy in this regard would need to come from multiple partners regularly, for example as has been done already through the Education Sector Development Committee, but it is critical to continue to acknowledge to the government the degree to which teachers lament being underpaid and overworked and feel the government is not listening to their concerns. In the meantime, expanding on existing strategies to help 114 | AIR.ORG LEARN Endline Evaluation Report acknowledge teachers’ work and provide supplementary compensation (e.g., through PTAs, or the STAR teachers intervention) could further help enhance teacher morale, attendance, and performance. Also, there remains the need to address the issue of frequent transfer of teachers to other schools, particularly those who have already been trained as Literacy Champions or SHN Champions.  Strengthen PTAs to support schools in the longer term. PTAs have critical roles in schools beyond LEARN activities. LEARN refresher trainings and meetings with PTAs after midline was effective in re-activating some PTAs that had lost momentum following the COVID closures or had been inactive for years prior. Working with PTAs to ensure that they have their own system for making and carrying out plans and remaining active without outside encouragement such as through LEARN can be critical in helping schools sustain themselves in the face of limited or sporadic government support. PTAs could have a more systematic role in supporting teachers who are feeling forgotten by the government or boosting the morale of Literacy Champions who lament not being paid cash for their work.  Attract more parents into PTAs. Continue stressing to PTA leaders the importance of including multiple parents and community members and train the leaders on strategies to attract parents and community members. One strategy is to convince parents that work done in collaboration with the PTA will ultimately provide compensation in the form of school improvements or parental influence over which activities are chosen.  Emphasize the importance of parents’ engagement in their children’s education and facilitate dialogues between parents and teachers about the challenges parents face in engaging with their children’s education. Teachers and principals can emphasize to parents the critical and constructive role they can play in enhancing their children’s education even without being educated themselves. This will also enable teachers to better understand the challenges that parents face and the assistance they in their efforts to support their children. With teachers, develop realistic strategies that parents and caregivers can use to encourage their children going forward.  Enhance PTAs’ understanding of the role of school gardens. Continue sensitizing PTA members to the active role that school gardens can play beyond supplementing school feeding activities. Rather, school gardens can be viewed as an income generation opportunity. For example, a larger garden could generate more income for PTA activities or help individual PTA members cover their children’s educational expenses. Meanwhile, it is critical that children are not exploited: teachers and students should be made aware that student work in the school garden is not meant to be done as punishment or demanded as free labor. Rather, all students, parents, and teachers can be expected make small contributions to the garden. 115 | AIR.ORG LEARN Endline Evaluation Report  Reiterate to communities the rationale for providing girls with take-home rations (THRs): they are aimed at reducing the risk of sex for grades and grooming, demonstrating commitment to equality by giving girls a boost (critical given past and current evidence of boys performing better).  Ensure schools have adequate materials and infrastructure to maintain a healthy and safe environment, particularly in kitchens. While cooks and storekeepers demonstrate adequate understanding of food safety procedures, they lament lack of materials or poor infrastructure to ensure they can keep up to those standards.  Work with the government to get its commitment to support institutionalizing school feeding across Liberia schools. Not only is school feeding popular, it increases the attendance and performance of students while alleviating many caregivers’ concerns about the well￾being of their children. At the same time, a school garden and the PTA alone cannot sustain daily hot lunches; additional commodities are essential. Implementation of the LEARN II school feeding model will provide an important case study in how to effectively roll out and sustain school feeding.  Separate WASH and nutrition components rather than grouping them as SHN, and task different parties to manage each. SHCs demonstrated willingness and capacity to engage in school cleaning activities, and some were active in teaching fellow students about handwashing. However, improving nutrition was rarely mentioned, likely because of the already difficult task SHCs and SHN champions had in maintaining school cleanliness. Having separate individuals responsible for the nutrition component (e.g., dividing an SHCs into two “wings”) may help prevent the important issue of nutrition from being sidelined.  Respect and enforce the school code of conduct. With the revision of the school code of conduct will come opportunities for widespread sensitization around its content, including the opportunity to have dialogues with school personnel, caregivers, and students on their perspectives. This will help elucidate what is limiting enforcement of the code of conduct, such as misunderstanding of the content despite the ability to list items in the code, disagreement with some of the rules, and lack of alternative disciplinary strategies that are in accord with the code (i.e., strategies that could replace corporal punishment).  Follow-up with schools on the status of their TCOC complaints mechanism to ensure it allows for children’s anonymity and protection, and that school leaders act on complaints made (or justify rationale for inaction) so that the system remains both safe and effective.  Track fidelity of implementation and contextualize findings and recommendations based on what has happened. Throughout project implementation, conduct regular assessments to identify gaps in implementation and work to fill those gaps appropriately. A robust 116 | AIR.ORG LEARN Endline Evaluation Report monitoring system to closely track fidelity of implementation may benefit the project and lead to a more refined evaluation of the project’s impacts at endline. This will also be critical during the scaling-up and expansion to occur as part of LEARN II. Such a system could also aim to identify other implementers working on similar projects within the project’s catchment area, allowing for collaboration and the avoidance of complications. 117 | AIR.ORG LEARN Endline Evaluation Report Annexes A. References B. Results Framework C. LEARN Evaluation Questions D. McGovern-Dole Performance Indicators E. Additional Tables and Complementary Outcomes F. Other Subtests of Reading Assessment G. Regression Analysis H. Additional Impact Evaluation Exhibits I. Inter-Rater Reliability J. Survey Instruments K. Qualitative Protocols L. Endline Evaluation SOW and TOR A1 | AIR.ORG LEARN Endline Evaluation Report Annex A. References Ahn, T., & Vigdor, J. (2010). The impact of incentives on effort: Teacher bonuses in North Carolina. Program on education policy and governance working papers series (PEPG 10-06). Cambridge, MA: Program on Education Policy and Governance. Harvard University, Kennedy School of Government. Alderman, H., Gilligan, D. O., & Lehrer, K. (2012). The impact of food for education programs on school participation in northern Uganda. Economic Development and Cultural Change, 61(1), 187-218. Angrist, N., de Barros, A., Bhula, R., Chakera, S., Cummiskey, C., DeStefano, J., ... & Stern, J. (2021). Building back better to avert a learning catastrophe: Estimating learning loss from COVID￾19 school shutdowns in Africa and facilitating short-term and long-term learning recovery. International Journal of Educational Development, 84, 102397. Better Criteria for Better Evaluation, Revised Evaluation Criteria, Definitions and Principles for Use, OECD/DAC Network on Development Evaluation. https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf Dowd, A.J., Pisani, L. & Borisava, I. (2016). “Evaluating Early Learning from Age 3 to Grade 3” in Understanding What Works in Oral Reading Assessments. Montreal: UNESCO Institute for Statistics (UIS). Feldman, H. A., & McKinlay, S. M. (1994). Cohort versus cross-sectional design in large field trials: Precision, sample size, and a unifying model. Statistics in Medicine, 13, 61–78. Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 613-619. Hansen, Christian, B. Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects. Journal of Econometrics, 140(2):670–694, 2007. Hanushek, Eric A. & Woessmann, Ludger, 2009. “Schooling, cognitive skills, and the Latin American growth buzzle.” Natural Bureau of Economic Research, Working Paper 1506. A2 | AIR.ORG LEARN Endline Evaluation Report Hess, R. D., & Holloway, S. D. (1984). Family and school as educational institutions. Review of child development research, 7, 179-222. Kim, Y. S. (2009). The relationship between home literacy practices and developmental trajectories of emergent literacy and conventional literacy skills for Korean children. Reading and Writing, 22(1), 57-84. Liberia Ministry of Education, 2016. Liberian Education Sector Analysis Liberia Ministry of Education, 2016. “Getting to Best, Education Sector Plan 2017-2020” Marcotte, D., & Hemelt, S. (2008). Unscheduled school closings and student performance. Education Finance and Policy, 3(3), 316-338. Medie, A. P. (2013). Fighting gender-based violence: The women's movement and the enforcement of rape law in Liberia, African Affairs, 112(448), 377–397, https://doi.org/10.1093/afraf/adt040 Miller, R. (2012). Teacher Absence as a Leading Indicator of Student Achievement: New National Data Offer Opportunity to Examine Cost of Teacher Absence Relative to Learning Loss. Center for American Progress. Parkes, J. (2016). The Evolution of Policy Enactment on Gender-based Violence in School. Prospects, 46, 93-107. https://doi.org/10.1007/s11125-016-9382-5 Postmus, J., Hoge, G., Davis, R., Johnson, L., Koechlein, E., & Winter, S. (2015) Examining gender based violence and abuse among Liberian school students in four counties: An exploratory study. Child Abuse and Neglect, 44, 76-86. https://doi.org/10.1016/j.chiabu.2014.11.012 RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. Stark, L., Warner, A., Lehmann, H., Boothby, N., & Ager, A. (2013). Measuring the incidence and reporting of violence against women and girls in Liberia using the 'neighborhood method'. Conflict and Health, 7(20). https://doi.org/10.1186/1752-1505-7-20 USAID. 2016. Food For Peace Liberia Desk Review. https://www.usaid.gov/sites/default/files/documents/1866/FFP-Desk-Review-Liberia￾Feb2016v2.pdf A3 | AIR.ORG LEARN Endline Evaluation Report Woods, R. (1990). The effect of teacher attendance on student achievement in two selected school districts. World Food Programme. 2012. Liberia Country Programme (2013–2018) Operation Document World Food Programme. 2017. Country Programme – Liberia (2013–2017), Standard Project Report B1 | AIR.ORG LEARN Endline Evaluation Report Annex B. LEARN Results Framework B2 | AIR.ORG LEARN Endline Evaluation Report C1 | AIR.ORG LEARN Endline Evaluation Report Annex C. LEARN Evaluation Questions Evaluation Questions Summary Finding Page Reference Relevance 1. Do program stakeholders (students, teachers, PTAs, parents, and local officials) feel the LEARN program is meeting their needs? Yes, in general stakeholders find the program relevant in every regard. 69–73 2. Are the school in-meals and take-home rations culturally appropriate? Yes, they are culturally appropriate but are becoming monotonous. 6, 109 3. Are the take home rations adequately meeting household needs? This is not conclusive given the study was not asking about households; however, stakeholders did mention that take-home rations were appreciated. Some volunteers (LCs, cooks, etc.) lamented that they were paid only in take-home rations. 5–6, 100, 104 4. Are book bank titles perceived as culturally appropriate and age-appropriate for primary school students, including over￾age learners? This was difficult to assess given respondents generally were not aware of book banks and, more generally, lacking access to books. Students in general appeared to appreciate any book that could be accessed but did say they would appreciate more storybooks. The study did not consider the perspectives of over-age learners. 91–92, 107 5. Did stakeholders feel that their voices were heard, and their needs considered throughout the program? In general stakeholders felt well-supported by LEARN staff, in particular the Literacy Champions and teachers who had increased attention from LEARN through more frequent trainings and monitoring. 5–6, 77–78, 86– 88, 90–91 6. Have activities to support literacy and improved nutrition been integrated in culturally appropriate ways in the target communities? We did not explore this question systematically. NA Are there any indications that activities contributed to community-level or individual-level resilience in terms of (a) ability to absorb and adapt to stressors in general (e.g., enhanced food security; health); (b) lessen the education loss; (c) make it easier for students to return to school at the reopening phase and continue to learn? The study was not able to answer this question though it is likely that the school meals and take-home rations in particular contributed to a household’s food security and / or increased savings that would help them to respond to shocks or stressors. NA Effectiveness 7. To what extent has the program achieved its output and outcome targets? Not able to respond about the outputs but many of the performance indicators measuring outcomes were short of their targets by endline (Annex D) D1 C2 | AIR.ORG LEARN Endline Evaluation Report Evaluation Questions Summary Finding Page Reference 8. What factors have inhibited or facilitated the achievement of program goals, objectives, and expected results? COVID-19 closures surely had a large impact on outcomes. 77–78, 87–88 Efficiency 9. Have intervention components been delivered within their planned timeline? School gardens were rolled out more slowly than anticipated; delivery of books was slower than anticipated. 75, 90–91 10. Are commodities being delivered within their planned timeline? Yes, as of endline. 73–74 11. How often are schools using produce from their school gardens to supplement USDA donated food? It is unclear how frequently they use produce, but the endline suggested that it was not common – many reported that meals were monotonous and lacked vegetables. 75, 108 12. Which commodity management strategies were most efficient for quick delivery and reduction of waste and theft? Provision of secured warehouses or similar storage spaces were effective in reducing theft. A no-tolerance policy of food theft (e.g., by teachers or cooks) was said to be effective. The evaluation was unable to assess adequately any further findings related to commodity management. 74, 106 13. Did school gardens produce enough food to supplement school meals adequately? No. 75, 110 Sustainability 14. What additional inputs are necessary to achieve sustainability? School gardens in particular need to be scaled up. Additional strategies to encourage LCs and other volunteers to continue their work uncompensated could be explored. 99–100, 110– 111 15. What are the current barriers to achieving sustainability? As above. 97, 99–100 16. Do schools have the necessary infrastructure and food management plans in place to continue feeding after the program concludes? Yes; the main component that is lacking is the commodity itself. 99–100 17. Do schools/communities have the necessary systems in place to recruit and maintain volunteers for reading camps? Yes, though as mentioned above, there should be more work on helping to convince volunteers of the importance of their role despite being uncompensated. 100–101 18. What are the necessary components for successful school handover of activities to the government and local community, as modeled by this program? The components articulated in LEARN II plans are excellent. 98–99 19. Is there evidence that LEARN program activities and benefits are likely to continue or to scale up after the project ends? Unlikely, thus the LEARN II components are critical. 101 Impact C3 | AIR.ORG LEARN Endline Evaluation Report Evaluation Questions Summary Finding Page Reference 20. Have the literacy skills of school-age children generally improved in the LEARN program area? We have mixed findings. 3, 41–26 21. Has LEARN contributed to increases in enrollment of school-age children? Yes. But see qualitative discussions for nuances. 58, 68–70, 104 22. Have nutrition, dietary, and food safety practices in schools improved in the LEARN program area? Health and nutrition practices for school feeding are established but inadequate. 94, 46–48, 62– 64 23. Are PTAs meeting on a regular basis and contributing effectively to the schools? Yes, though they could do more as it relates to SHN (especially nutrition). 108 24. Have there been any positive or negative impacts in the target areas, besides the realization of the strategic objective-level results? Some of the literacy markers (e.g., letter knowledge) decreased over time, probably due to COVID-19. 3, 41–43 25. How do the literacy and health KAP outcomes compare across the three treatment groups in Grand Gedeh County? Is there evidence of a positive impact of LEARN on literacy and health KAP outcomes? Reading comprehension improved, especially for boys. Boys were also more likely to become readers. There was improvement in handwashing behaviors among girls. No effects were found on knowledge of balanced diet (nutrition). Impacts seemed to be driven by the combined package rather than the base package. 59–65 26. How have the SRGBV activities affected knowledge and practices among students and teachers? Respondents across all groups reported being aware of the content of the school code of conduct, but, as at midline, there was an evident degree of disregard or misunderstanding from both students and teachers. 49–55 27. Has LEARN improved access to and the quality of early grade reading materials in Liberia? There was improvement in access to reading materials and books, yet access to books remained sporadic. 35–27, 90–91 28. How has the home literacy environment in target communities changed in the LEARN program area? Home literacy environment improved. 37–39, 109 29. How have the variety of distance education options not related to the project, but which were delivered across project areas affected outcomes? This is difficult to assess, but findings from midline suggest that the Home Learning materials were in general not helpful for students who lacked supplementary support in interpreting the materials (and encouragement in using them). NA D1 | AIR.ORG LEARN Endline Evaluation Report Annex D. McGovern-Dole Performance Indicators McGovern-Dole Indicators Data Collection Methods Data Source Baseline (Percentage or Number) Midline (Percentage or Number) Endline (Percentage or Number) Life of Project Target MGD 26: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Evaluation LBRA Boys: 1% Boys: 4% Boys: 5% 30% Girls: 1% Girls: 7% Girls: 4% 25% Overall: 1% Overall: 6% Overall: 4.5% 28% Custom: Percent of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters. Evaluation LBRA Boys: 65% Boys: 71% Boys: 54% 80% Girls: 59% Girls: 59% Girls: 44% 70% Overall: 62% Overall: 65% Overall: 49% 75% MGD 27: Number of individuals benefiting directly from USDA￾funded interventions SC/Monitoring SC 0 54,932 Boys: 27,090 Girls: 26,714 298,862 Overall: 53,804 MGD 28: Number of individuals benefiting indirectly from USDA￾funded interventions SC/Monitoring SC 0 174,760 165,672 20,361 MGD 1: Number of students regularly (80%) attending USDA supported classrooms/schools SC/Monitoring SC 22,766 35,589 48,132 D2 | AIR.ORG LEARN Endline Evaluation Report McGovern-Dole Indicators Data Collection Methods Data Source Baseline (Percentage or Number) Midline (Percentage or Number) Endline (Percentage or Number) Life of Project Target MGD 19: Number of individuals who demonstrate use of new child health and nutrition practices as a result of USDA assistance SC/Monitoring SC 0 857 438 179 MGD 21: Number of individuals who demonstrate use of new safe food preparation and storage practices as a result of USDA assistance SC/Monitoring SC 0 425 631 3,344 Custom: Percentage of teachers in target schools who attend and teach at least 90% of the scheduled school days SC/Monitoring SC 86% 75% 86% 90% MGD 2: Number of textbooks and other teaching and learning materials provided as a result of USDA assistance SC/Monitoring SC 0 9,959 4,521 7,847 MGD 5: Number of teachers/educators in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance SC/Monitoring SC 0 N/A 171 151 MGD 6: Number of teachers/educators/teaching assistants trained or certified as a result of USDA assistance SC/Monitoring SC 0 N/A 114 246 MGD 15: Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance SC/Monitoring SC 0 3,458,471 27,721,134 9,020,800 D3 | AIR.ORG LEARN Endline Evaluation Report McGovern-Dole Indicators Data Collection Methods Data Source Baseline (Percentage or Number) Midline (Percentage or Number) Endline (Percentage or Number) Life of Project Target MGD 16: Number of school-age children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance SC/Monitoring SC 0 44,122 Boys: 20,439 Girls: 20,439 45,104 Overall: 41,976 MGD 13: Number of take-home rations provided as a result of USDA assistance SC/Monitoring SC 0 24,073 232,939 MGD 14: Number of individuals receiving take-home rations as a result of USDA assistance SC/Monitoring SC 0 48,588 Boys: 670 Girls: 5,729 57,228 Overall: 6,399 MGD 17: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance SC/Monitoring SC 0 46579 Boys: 32,893 Girls: 31,795 290,342 Overall: 64,688 Custom: Number of daily school meals provided that include fruits, vegetables and/or animal￾sourced proteins in addition to USDA commodities SC/Monitoring SC 0 0 783,521 1,471,400 Custom: Number of schools with a strengthened support structure for a code of conduct policy SC/Monitoring SC 0 220 124 220 Custom: Percentage of children in target schools who demonstrate improved knowledge and practices toward SRGBV prevention and response Evaluation Student survey Boys: 67% Boys: 68% Boys: 72% 85% Girls: 68% Girls: 65% Girls: 72% 80% Overall: 67% Overall: 66% Overall: 72% 83% D4 | AIR.ORG LEARN Endline Evaluation Report McGovern-Dole Indicators Data Collection Methods Data Source Baseline (Percentage or Number) Midline (Percentage or Number) Endline (Percentage or Number) Life of Project Target MGD 12: Number of educational policies, regulations and/or administrative procedures in each of the following stages of development as a result of USDA assistance. SC/Monitoring SC 0 N/A 0 2 MGD 24: Number of students receiving deworming medication(s) SC/Monitoring SC 45,154 4230 39,156 45,154 Custom: Number of energy￾saving stoves provided as a result of USDA assistance SC/Monitoring SC 0 220 0 220 MGD 7: Number of educational facilities (i.e., school buildings, classrooms, and latrines) rehabilitated/ constructed as a result of USDA assistance SC/Monitoring SC 0 220 Warehouses/ storerooms: 51 220 Latrines: 35 Kitchens, cook areas: 66 Overall: 152 Custom: Number of primary school-age children in targeted communities who participated in a reading camp in the past year SC/Monitoring SC 0 5450 Boys: 1,611 7,080 Girls: 1,544 In school: 3,036 Out of school: 119 Overall: 3,155 Custom: Number of government officials trained in commodity management practices SC/Monitoring SC 0 39 0 28 MGD 10: Number of public￾private partnerships formed as a result of USDA assistance SC/Monitoring SC 0 220 0 220 D5 | AIR.ORG LEARN Endline Evaluation Report McGovern-Dole Indicators Data Collection Methods Data Source Baseline (Percentage or Number) Midline (Percentage or Number) Endline (Percentage or Number) Life of Project Target MGD 9: Number of PTAs or similar school governance structures supported as a result of USDA assistance SC/Monitoring SC 0 220 217 220 Custom: Percentage of Grades 2 and 6 students in target schools who can identify the components of a healthy diet Evaluation Student survey Boys: 0% Boys: 0% Boys: 0% 55% Girls: 0% Girls: 0% Girls: 1% 55% Overall: 0% Overall: 0% Overall: 1% 60% MGD 20: Number of individuals trained in safe food preparation and storage as a result of USDA assistance SC/Monitoring SC 0 847 Males: 0 Females: 537 880 Overall: 537 Custom: Number of schools equipped with food preparation and storage materials SC/Monitoring SC 100 220 217 220 MGD 11: Value of new public and private sector investments leveraged as a result of USDA assistance SC/Monitoring SC 0 12,355.70 19,189.20 4,400 MGD 18: Number of individuals trained in child health and nutrition as a result of USDA assistance SC/Monitoring SC 0 459 0 128 MGD 23: Number of schools with improved sanitation facilities SC/Monitoring SC 100 220 217 220 E1 | AIR.ORG LEARN Endline Evaluation Report Annex E. Additional Tables Exhibit E1. Overview of Topics Covered in Qualitative Protocols Topics Types of Questions (asked to groups/individuals) as relevant; see protocols in Annex D Background information  Questions about background/role in project  Any changes in role/location at onset of COVID closures Access to and value of education  Access to education in the community; specific barriers to access and full engagement (who is excluded)  Gender-equity of access  Parental involvement, etc.  Did COVID closures change any of these perceptions and, if so, how? School feeding/ nutrition  Existence of and quality of kitchen, gardens  Perceived effectiveness of feeding program; successes and areas for improvement  Status of training of MOE school feeding division officials  Input on progress related to MOE’s desire to move to a Home-Grown School Feeding approach to school meals under the National School Feeding Policy  Knowledge of and agreement to ground rules on gardening activities; challenges to date  Effectiveness of THRs for students and volunteers (in summer/during school closures; how the stoppage of THRs upon school re-openings is perceived SHCs/water, sanitation, and hygiene (WASH)/ nutrition  Perceived effectiveness of SHN champions and SHCs on improving nutrition and WASH practices in schools  WASH status in schools  Perception of WASH grants scheme (PTA)  Effectiveness of annual de-worming campaign  Progress on the development of SHCs and manuals  Progress on SC collaboration with CEOs and DEOs to provide training to the SHN Champions  Effectiveness of community mobilizers  How COVID messaging impacted any of this School literacy environment  How much students are exposed to literacy activities within the school environment (e.g., presence of library, teacher reading exercises)  Resources and encouragement provided to students to read outside of school (e.g., can take home library books, working with parents/PTAs to encourage reading at home), highlighting in particular whether they had anything to read during closures.  Feasibility of teachers with added load as Literacy Champions; related events Home/ community literacy  How much students are exposed to literacy activities within the home (e.g., presence of books or other reading materials)  Whether literacy is valued in the home (e.g., if reading and doing homework is encouraged) E2 | AIR.ORG LEARN Endline Evaluation Report Topics Types of Questions (asked to groups/individuals) as relevant; see protocols in Annex D environment/ reading clubs  Existence of/quality of community-based reading activities and resources (e.g., book banks, reading clubs, reading festivals [not yet started]), ease of accessibility to materials within them  Degree to which students actively pursue/take part in home/community reading activities  Reflections on the efficacy of the school year and summer reading clubs  Difference between in-school/out-of-school uptake in Summer Reading Clubs  Adequacy of training received to be Literacy Champion  How access to at-home resources, including the home learning packet, changed during COVID closures with distribution of various distance education programs SRGBV  Information around the extent to which students, parents, and teachers know about whether they are protected in the school by (a) a code of conduct that restricts SRGBV behaviors and (b) effective referral and reporting mechanisms to report such behaviors if they do occur.  Positive discipline strategies (as alternative to corporal punishment) in place, and their effectiveness or limitations  Existence of/effectiveness of reporting mechanisms for students/teachers to report violations of school code of conduct  What students like and dislike about their school and teachers (probing on issues specifically around school climate and safety)  Development and revision of MOE Code of Conduct; mechanisms for rollout and successes/challenges  Work on supporting reporting mechanisms at school and district levels; procedures for responding to reports against teachers and other staff  Perceived increases in violence at home during COVID closures Parent-Teacher Associations  Existence and activities of PTAs; specific successes and areas for improvement to enhance collaboration and effectiveness.  Perception on WASH grants scheme (PTA)  Degree to which parents in PTAs collaborate with teachers/principals  Effectiveness of parent engagement messages on literacy (engagement not yet started; awareness session held with communities)  PTA activities during school closures Exhibit E1. Baseline and Endline Levels for Key Project Indicators Sex Overall Grand Bassa Grand Gedeh Rivercess River Gee Base End Base End Base End Base End Base End Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand grade level texta Girls 1% 4% 1% 7% 0% 0% 1% 7% 1% 3% Boys 1% 5% 1% 5% 0% 3% 0% 4% 1% 6% Percentage of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters Girls 59% 44% 72% 58% 46% 33% 67% 51% 43% 28% E3 | AIR.ORG LEARN Endline Evaluation Report Sex Overall Grand Bassa Grand Gedeh Rivercess River Gee Base End Base End Base End Base End Base End Boys 65% 54% 68% 67% 57% 38% 77% 53% 56% 50% Percentage of children in target schools who demonstrate improved knowledge and practices toward SRGBV prevention and response Girls 64% 72% 66% 61% 52% 70% 74% 86% 67% 86% Boys 64% 72% 69% 64% 51% 75% 69% 81% 63% 79% Percentage of Grades 2 and 6 students in target schools who can identify the components of a healthy diet Grade 2 Girls 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Boys 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Grade 6 Girls 0% 2% 0% 0% 0% 5% 0% 0% 1% 0% Boys 1% 1% 1% 0% 0% 3% 0% 3% 0% 0% Exhibit E2. Student Sex Distribution Grade Percent Male Percent Female Baseline Endline Baseline Endline Grade 2 54% 53% 46% 47% Grade 6 56% 47% 44% 53% Total 55% 51% 45% 49% Source: Student survey. Authors’ calculations. Exhibit E3. Socio-economic status, by County Indicator Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End English is the main language 54% 67% 71% 60% 76% 79% 92% 95% 69% 72% Total number of household assets 1.75 1.88 1.69 2.24 1.83 1.97 1.77 1.57 1.76 1.93 Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Baseline: N = 958 for Grand Bassa, 198 for Grand Gedeh, 438 for Rivercess, 427 for River Gee; Endline: N = 630 for Grand Bassa, 467 for Grand Gedeh, 208 for Rivercess, 350 for River Gee E4 | AIR.ORG LEARN Endline Evaluation Report Exhibit E4. Availability of reading materials in the home, by County Does your home have…? Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Holy book 68% 73% 72% 78% 64% 86% 64% 79% 66% 77% Textbooks/schoolbooks 48% 59% 53% 59% 49% 59% 52% 74% 49% 62% Storybooks/comics 32% 47% 42% 37% 38% 22% 35% 21% 35% 35% Coloring and drawing books 22% 23% 32% 21% 15% 8% 15% 13% 20% 18% Newspapers 8% 6% 11% 4% 5% 1% 5% 3% 7% 4% None of the above 17% 8% 9% 14% 14% 6% 12% 4% 15% 9% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Baseline: N = 958 for Grand Bassa, 198 for Grand Gedeh, 438 for Rivercess, 427 for River Gee; Endline: N = 649 for Grand Bassa, 467 for Grand Gedeh, 208 for Rivercess, 350 for River Gee Exhibit E5. Access to Non-Textbook Reading Materials in School, by County Response Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End No 61% 53% 56% 36% 50% 69% 54% 32% 56% 46% Yes, but can't take off campus 16% 5% 7% 26% 20% 15% 12% 60% 15% 23% Yes, and can take home for free 21% 41% 36% 38% 25% 17% 33% 8% 26% 30% Yes, and can take home for a cost 1% 0% 1% 0% 5% 0% 2% 0% 2% 0% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Does not include students who responded “Did not know” or refused to respond. Bold denotes significance at the 10% level. Baseline: N = 658 for Grand Bassa, 134 for Grand Gedeh, 303 for Rivercess, 263 for River Gee; Endline: N = 412 for Grand Bassa, 259 for Grand Gedeh, 131 for Rivercess, 220 for River Gee Exhibit E6. Frequency with which Students Borrowed Non-Textbook Materials to Take Home Frequency Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Every day 6% 3% 6% 6% 8% 5% 9% 6% 7% 4% A few times during the week 29% 25% 8% 37% 12% 82% 21% 39% 18% 34% Once during the week 42% 49% 15% 29% 36% 14% 38% 50% 36% 40% Never 23% 23% 71% 28% 43% 0% 32% 6% 36% 22% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Does not include students who answered “Did not know” or refused to respond. Bold denotes significance at the 10% level. Baseline: N = 147 for Grand Bassa, 48 for Grand Gedeh, 89 for Rivercess, 90 for River Gee; Endline: N = 169 for Grand Bassa, 100 for Grand Gedeh, 22 for Rivercess, 18 for River Gee E5 | AIR.ORG LEARN Endline Evaluation Report Exhibit E7. Household Literacy Activities in the Past Week Activity Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Saw someone reading 39% 53% 49% 66% 55% 30% 55% 67% 47% 56% Helped with studies 53% 62% 71% 72% 68% 51% 68% 76% 61% 66% Read to student 44% 62% 59% 71% 56% 27% 54% 64% 50% 60% Told student a story 33% 36% 57% 46% 30% 35% 34% 47% 35% 40 Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Does not include students who responded “Did not know” or refused to respond. Baseline: N = 660 for Grand Bassa, 136 for Grand Gedeh, 305 for Rivercess, 266 for River Gee; Endline: N = 412 for Grand Bassa, 261 for Grand Gedeh, 132 for Rivercess, 221 for River Gee Exhibit E8. Teacher Attendance Frequency Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Every day (5 days) 84% 83% 73% 84% 93% 91% 87% 93% 86% 86% A few times (2-4 days) 10% 11% 14% 13% 5% 8% 7% 4% 9% 10% Once during the week 1% 1% 4% 0% 0% 0% 3% 0% 1% 0% Never 5% 5% 8% 2% 2% 1% 4% 3% 4% 3% Source: Student survey. Authors’ calculations. Bold denotes significance at the 10% level. Does not include students who responded “Did not know” or refused to respond. Baseline: N = 596 for Grand Bassa, 132 for Grand Gedeh, 294 for Rivercess, and 257 for River Gee. Endline = N = 305 for Grand Bassa, 174 for Grand Gedeh, 119 for Rivercess, and 104 for River Gee Exhibit E9. Teacher Tardiness Frequency Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Every day (5 days) 6% 9% 3% 5% 1% 0% 2% 0% 4% 5% A few times (2-4 days) 11% 8% 17% 11% 5% 1% 10% 6% 10% 7% Once during the week 9% 6% 14% 7% 10% 9% 14% 18% 11% 8% Never 74% 77% 66% 77% 84% 91% 75% 76% 75% 79% Source: Student survey. Authors’ calculations. Bold denotes significance at the 10% level. Does not include students who responded “Did not know” or refused to respond. Baseline: N = 550 for Grand Bassa, 123 for Grand Gedeh, 282 for Rivercess, and 248 for River Gee. Endline = N = 291 for Grand Bassa, 171 for Grand Gedeh, 117 for Rivercess, and 100 for River Gee E6 | AIR.ORG LEARN Endline Evaluation Report Exhibit E10. Second Grade Students Literacy Skills, by County Outcome Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Foundational Literacy Skills Letter knowledge (% correct) 91% 89% 87% 78% 93% 87% 86% 85% 90% 85% Word recognition (% correct) 46% 30% 31% 14% 45% 25% 28% 17% 38% 23% Invented word recognition (% correct) 6% 2% 4% 2% 5% 2% 3% 1% 4% 2% Reading Skills Students classified as readers (5+ words correct in 30 seconds) 14% 16% 22% 8% 17% 11% 17% 7% 17% 11% Accuracy (% words correct in passage), readers only 13% 54% 7% 31% 8% 53% 10% 63% 10% 51% Fluency (words correct per minute), readers only 11 24 11 15 12 16 11 25 11 22 Comprehension Skills Reading comprehensions questions correct (%), readers only 55% 54% 56% 52% 55% 52% 49% 55% 54% 53% Listening comprehension passed (%), non￾readers only 18% 14% 28% 17% 19% 13% 17% 14% 20% 15% Listening comprehension passed (%), readers only 35% 12% 36% 35% 19% 0% 12% 7% 25% 14% Source: Student survey. Authors’ calculations. Bold denotes significance at the 10% level. Does not include students who responded “Did not know” or refused to respond. Baseline: N = 240 for Grand Bassa, 137 for Grand Gedeh, 184 for Rivercess, and 197 for River Gee; Endline: N = 414 for Grand Bassa, 262 for Grand Gedeh, 132 for Rivercess, and 221 for River Gee. There were 68 readers in Grand Bassa, 20 in Grand Gedeh, 14 in Rivercess, and 15 in River Gee at endline; 33 readers in Grand Bassa, 30 in Grand Gedeh, 31 in Rivercess, and 33 in River Gee at baseline. E7 | AIR.ORG LEARN Endline Evaluation Report Exhibit E11. Student knowledge vs practice of critical handwashing moments, by County Indicator Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Handwashing knowledge 16% 22% 3% 27% 26% 14% 27% 21% 19% 20% Handwashing self-reported behavior 11% 17% 2% 23% 8% 3% 11% 10% 9% 15% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Baseline: N = 958 for Grand Bassa, 198 for Grand Gedeh, 438 for Rivercess, 427 for River Gee; Endline: N = 630 for Grand Bassa, 267 for Grand Gedeh, 208 for Rivercess, 350 for River Gee Exhibit E12. Comprehension Subtests Response Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Summary 14% 5% 20% 21% 11% 6% 7% 7% 13% 10% Literal 35% 41% 44% 36% 34% 37% 26% 34% 34% 37% Inferential 78% 74% 78% 67% 75% 80% 69% 77% 75% 73% Evaluative 73% 56% 81% 49% 69% 59% 68% 56% 72% 55% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Baseline: N = 240 for Grand Bassa, 137 for Grand Gedeh, 184 for Rivercess, and 197 for River Gee; Endline: N = 414 for Grand Bassa, 262 for Grand Gedeh, 132 for Rivercess, and 221 for River Gee. E8 | AIR.ORG LEARN Endline Evaluation Report Exhibit E15. Effect Size of LEARN on Key Learning Outcomes, by Treatment and Gender Outcomes Combined Package vs Control Group Base Package vs Control Group Combined And Base Package vs Control Group Combined vs Base Package Reading comprehension .34** (.016) .12 (.009) .24** (.011) .21 (.018) Knowledge of letters -.106 (.036) .122 (.111) -.032 (.098) -.28* (.077) Reader .565 (.094) .24 (.086) .435 (.079) .315 (.096) Observations 945 801 1402 1058 Girls Reading comprehension .12 (.006) .36 (.012) .2* (.006) -.26 (.015) Knowledge of letters -.134 (.13) .208 (.133) .002 (.124) -.334* (.089) Reader .2 (0.127) .3 (0.121) .255 (0.112) -.6 (0.111) Observations 448 368 656 496 Boys Reading comprehension .63** (.028) .04 (.008) .4** (.019) .48* (.025) Knowledge of letters -.156 (.113) .058 (.13) -.66 (.11) -.242 (.099) Reader .91* (0.100) .145(0.083) .595 (0.083) .645 (0.090) Observations 497 433 746 562 Source: Student survey; authors’ calculations; * p < 0.1 ** p < 0.05 *** p < 0.01; Standard errors shown in parentheses are clustered at the school level. Only key DID (Treatment X Post) key impact estimate are shown for relevant treatment and comparison/control groups and relevant outcomes. We first show estimates for overall sample, and then we disaggregate results by gender. We control for dummies for time (endline vs baseline), relevant treatment groups as well as age, age squared, gender, total number of assets and English as main spoken language at home. Effect sizes are shown that adjust unstandardized DiD impacts by standard deviation (SD) of the control groups in each group. 34The one exception is reading comprehension for girls, where a “pooled standard deviation” that averages SDs across control group and base group treatment is used since SD for control group was 0. 34 We follow J-PAL’s definitions for effect size: https://www.povertyactionlab.org/sites/default/files/research￾resources/ExerciseC_Participant.powercalc.TA_.pdf E9 | AIR.ORG LEARN Endline Evaluation Report Exhibit E16. Effect size of LEARN on Key Nutrition and WASH Outcomes, by Treatment Group and Gender Outcomes Combined Package vs Control Group Base Package vs Control Group Combined And Base Package vs Control Group Combined vs Base Package Handwashing behavior .285 (.074) .195 (.072) .243 (.067) .1 (.054) Knowledge of handwashing .018 (.056) -.06 (.066) -.02 (.053) .083 (.059) Knowledge of balanced diet -.04 (.007) .07 (.011) .01 (.007) -.1 (.009) Observations 945 801 1402 1058 Girls Handwashing behavior .423* (.088) .175 (.092) .318 (.08) .21 (.083) Knowledge of handwashing .054 (.085) -.036 (.098) .018 (.079) .085 (.087) Observations 448 368 656 496 Boys Handwashing behavior .188 (.079) .22 (.077) .2 (.073) -.125 (.056) Knowledge of handwashing -.01 (.069) -.095 (.077) -.043 (.066) .068 (.063) Knowledge of balanced diet -.06 (.013) .05 (.015) -.01 (.013) -.1 (.012) Observations 497 433 746 562 Source: Student survey; authors’ calculations; p < 0.1 ** p < 0.05 *** p < 0.01; Standard errors shown in parentheses are clustered at the school level. Sample is restricted to girls. Only key DID (Treatment X Post) key impact estimate are shown for relevant treatment and comparison/control groups and relevant outcomes. We first show estimates for overall sample, and then we disaggregate results by gender. We control for dummies for time (endline vs baseline), relevant treatment groups as well as age, age squared, total number of assets and English as main spoken language at home. Effect sizes are shown that adjust unstandardized DiD impacts by standard deviation (SD) of the control groups in each group. As knowledge of balanced diet had very few observations for girls, we do not show effect sizes for this outcome for girls. F1 | AIR.ORG LEARN Endline Evaluation Report Annex F. Other Subtests of Reading Assessment As mentioned in Section 3.2.1.5, we also tested students on other literacy skills including word recognition, and invented word recognition. This annex presents the outcomes of these subtests to shed more lights on children’s literacy outcomes. Word Recognition To assess children’s word recognition skill, students were given a chart of 20 words that we developed based on the most frequently used words from their textbooks. Exhibit F1 shows the ability of second graders to read these words. In comparison to their ability to identify letters, students struggled to read full words. Additionally, there were some large disparities between counties as more students struggled to read the words in Grand Gedeh and River Gee compared to Grand Bassa and Rivercess – a trend seen in both baseline and endline. As seen in Exhibit F1, overall, students were only able to identify 38% of the 20 words at baseline and 23% at endline. We see an increase in % of students identifying zero words, notably in Grand Gedeh where it increased from 11% to 29%. Generally, the outcomes for word recognition worsened from baseline to endline across all counties. Exhibit F1. Most Recognized Word, by County Indicator Grand Bassa Grand Gedeh Rivercess River Gee Overall Base End Base End Base End Base End Base End Total number of correctly read words 9.2 6.1 6.2 2.9 8.9 4.9 5.5 3.3 7.6 4.5 % of words read correctly 46% 30% 31% 14% 45% 25% 28% 17% 38% 23% % identified zero words 8% 17% 11% 29% 3% 14% 23% 26% 11% 21% Source: Student survey. Authors’ calculations. Note: Students were told to select all that apply, and therefore the total of the percentages do not add up to 100%. Bold denotes significance at the 10% level. Baseline: N = 240 for Grand Bassa, 137 for Grand Gedeh, 184 for Rivercess, and 197 for River Gee; Endline: N = 414 for Grand Bassa, 262 for Grand Gedeh, 132 for Rivercess, and 221 for River Gee. Exhibit F2 also shows that the overall distribution of the number of words identified has a downward trend with the plurality of students naming just 1-5 words correctly. F2 | AIR.ORG LEARN Endline Evaluation Report Exhibit F2. Distribution of Most Used Words Identified, by County 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Grand Bassa Baseline Endline 0% 5% 10% 15% 20% 25% 30% 35% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Grand Gedeh Baseline Endline F3 | AIR.ORG LEARN Endline Evaluation Report Source: Student survey. Authors’ calculations. Baseline: N = 240 for Grand Bassa, 137 for Grand Gedeh, 184 for Rivercess, and 197 for River Gee; Endline: N = 414 for Grand Bassa, 262 for Grand Gedeh, 132 for Rivercess, and 221 for River Gee. 0% 2% 4% 6% 8% 10% 12% 14% 16% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Rivercess Baseline Endline 0% 5% 10% 15% 20% 25% 30% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 River Gee Baseline Endline F4 | AIR.ORG LEARN Endline Evaluation Report Decoding (Invented Word Recognition) We also included a decodable word test in the LBRA to measure the ability of students in recognizing the basic sounds and phonemes. We rearranged the 20 most common words (from the word recognition test) to form “pseudo words” and asked students to decode. Students especially struggled with this task as they only identified less than one word correctly on average, both at baseline and endline. Exhibit F3 shows that 77% of the sample could not read even one word at baseline. This figure was 88% at endline. Exhibit F3. Invented Word Recognition Indicator Mean Observations Baseline Endline Baseline Endline Total number of correctly read invented words 0.87 0.34 758 1029 % of invented words read correctly 4% 2% 758 1029 % identified zero invented words 77% 88% 758 1029 Source: Student survey. Authors’ calculations. G1 | AIR.ORG LEARN Endline Evaluation Report Annex G. Inter-Rater Reliability Reading Assessment To measure the reliability and level of homogeneity of enumerators’ scores on children’s literacy skills, 9% of the overall endline sample (139 out of 1,488) of Grade 2 students were assessed by two different enumerators simultaneously. Long one-way Analysis of Variance techniques, which is used to determine whether the mean of a dependent variable is the same in two or more unrelated and independent groups, were used to calculate the intra-class correlation within pairs of assessors for a measure of inter-rater reliability. Adapted from Fleiss et al. (1973), we interpreted the intra-class correlations as it follows:  Less than .40 – Poor  Between .40 and .75 – Good or fair  Greater than .75 – Excellent Exhibit G1 shows the percent of agreement between the raters, as well as inter-rater reliability ratings for the project evaluation sample. Overall, the inter-rater reliability (IRR) across the project evaluation sample was excellent for all of the literacy skills measures showing high internal validity of the scores. For reading comprehension, there were no variations in the proportion of children who were able to answer at least 80% of comprehension questions. Therefore, the ANOVA test could not calculate the IRR. Given that this means the results were identical, it is a positive result. Exhibit G1. IRR by Literacy Skill Subtests for Performance Sample Literacy Skill Sub-test IRR Rating Letter Knowledge 92% Excellent Word Recognition 99% Excellent Reader 94% Excellent Fluency 100% Excellent Accuracy (out of the whole passage) 100% Excellent Accuracy (out of the words attempted) 99% Excellent Reading Comprehension n/a n/a Listening Comprehension 94% Excellent Source: Student survey. Authors’ calculations. N = 102 Grade 2 students Exhibit G2 shows the IRR results for the impact sample. The enumerators conducted paired interviews for 9% of the school feeding group, 8% of the school feeding, literacy boost, and school health and nutrition group, and 9% of the comparison group. Similar to the project evaluation G2 | AIR.ORG LEARN Endline Evaluation Report sample, the IRR was excellent for most measures. Again, there was no variation in the reading comprehension measure. Exhibit G2. RR by Literacy Skill Subtests for Impact Sample Literacy Skill Sub-test IRR Rating Letter Knowledge 97% Excellent Word Recognition 99% Excellent Reader 83% Excellent Fluency 100% Excellent Accuracy (out of the whole passage) 100% Excellent Accuracy (out of the words attempted) 100% Excellent Reading Comprehension n/a n/a Listening Comprehension 96% Excellent Source: Student survey. Authors’ calculations. N = 66 Grade 2 students H1 | AIR.ORG LEARN Endline Evaluation Report Annex H. Survey Instruments Student survey & LBRA ENDLINE AND BASELINE DATA COLLECTION FOR USDA FOOD FOR EDUCATION (LEARN AND LEARN II) IN LIBERIA IMPACT AND PROJECT EVALUATION Student Survey Start Time Date INTRODUCTION This section is for enumerators to fill County 1. Grand Bassa 2. Grand Gedeh 3. Rivercess 4. River Gee Districts Enter the name of the district ----------------------------- School Name Enter the school name ------------------ School ID Enter the school’s ID number (EMIS) _____________ Enum Enter your name -------------------------------- Enum_fem What is the gender of the enumerator? 0. Male 1. Female Consent Has the parent given consent for the child to participate in this survey? 0. No thank them and terminate the survey and select the next child on your list. 1. Yes stcode1 I__I Please get the student code from the team leader. It is very important to use the correct student code, so please enter the code twice. If you are unsure, please check again with the team leader stcode1 Please enter the student code CAREFULLY------------ stcode2 Please enter the student code CAREFULLY again ------------ Reliab Is this an individual assessment or a pair assessment? 0. Individual  “nickname” 1. Pair assessment ”reliabtype” I__I H2 | AIR.ORG LEARN Endline Evaluation Report Reliabtype Talking enumerator or observing enumerator? 0. Observing 1. Talking I__I Dear student: Hello my name is ___, and I am with Center for Action Research and Training. I am here asking some questions from children like you to understand more about a reading program. Your answers will help us make Liberia's education system better. Your parents, your classmates and your teachers will not know your answers to the questions. Everything you say will be kept a secret. There aren’t any right or wrong answers. I want you to answer honestly and as best as you can. It will take only 35 minutes. Do you have any questions for me? You can interrupt me to ask a question at any time. Also, if you don't know the answer to a question or don't want to answer it, just let me know and we can skip it. I will just start with a few questions to know you better, and then we will play a reading game. Are you ready to begin? Ask students’ assent from everyone assent Do you agree to answer the questions I have? 0. No, thank him/her, terminate the survey, and proceed to the next child on your list. 1. Yes, continue to the background section. I__I If child says No, thank him/her, terminate the survey, and proceed to the next child on your list. Background information [DON’T READ TO THE CHILD] Ask this section from students in both grades (Second and Sixth) Fname What is your first name? Lastname What is your last name? Caregivername What is the name of the person that takes care of you at home most of the time? Caregiver Who is (caregivername)’s to you? 1. Mother 2. Father 3. Older sister 4. Older brother 5. Grandmother H3 | AIR.ORG LEARN Endline Evaluation Report 6. Grandfather 7. Other female relative 8. Other male relative 9. Female non-relative 10. Male non-relative 11. Other (Specify) 888. Don’t know Caregiverschool Did (caregivername) go to school when she/he was small? 0. No 1. Yes 888. Don’t know/No response *Select only one option female 0. Male 1. Female I__I *Ask only if necessary age How old are you? …… *RECORD AGE >=5 & <25 *Mark 888 if no response/don’t know Only ask Newsch1 to Newsch1 to newsch3b from Grade 2 students in Grand Gedeh Newsch1 Did you move to this school in the last three years? 0. No  newsch4 1. Yes 888. Don’t know I__I *Select only one option *Probe to ensure the kid understands the sense of time *Ask only from Grade 2 in Grand Gedeh Newsch2 When did you start? 1. Before 2018 2. 2018 3. 2019 4. 2020 5. 2021 I__I *Select only one option *You can probe with asking students which semester they start Newsch3a Which school did you attend before the current school? School name: School ID: Write “Not listed” if it is not on the list, and add 888 as the school ID and pass to Newsch4. I__I NOTE: Find the school mentioned by child on your own tablet and select. If a paper survey, find the school on your school list, and write the full school name and ID on the survey. Newsch3b For how many years did you study in “newsch3a” school? …. *Add a number from 1 to 6 *Enter 888 if no response/don’t know H4 | AIR.ORG LEARN Endline Evaluation Report Newsch4 When you started at this school, which grade were you in? 1. Preschool/ABC 2. KG 3. Grade 1 4. Grade 2 5. Grade 3 6. Grade 4 7. Grade 5 8. Grade 6 888. Don’t know I__I *Select only one option *This is regarding the present school that they are enrolled. *It has to be asked from all students across four counties. grade Which grade/class are you in? 1. Grade 2 2. Grade 6 3. Other  Thanks the child and terminate the survey I__I *Select only one option everrpt Did you repeat any grades? 0. No  studattend 1. Yes  everrpt_b 888. Don't know/ No response  studattend I__I *Select only one option Everrpt_b Which grades have you repeated? 0. KGEverrpt_kg 1. Grade 1 Everrpt_c1 2. Grade 2 Everrpt_c2 3. Grade 3 Everrpt_c3 4. Grade 4 Everrpt_c4 5. Grade 5 Everrpt_c5 6. Grade 6 Everrpt_c6 888. Refuse to answer I__I *Select all that apply Everrpt_c0 How many times did you repeat Kindergarten? … *Enter the frequency *Select if everrpt_b=0 Everrpt_c1 How many times did you repeat Grade 1? … *Enter the frequency *Select if everrpt_b=1 H5 | AIR.ORG LEARN Endline Evaluation Report Everrpt_c2 How many times did you repeat Grade 2? … *Enter the frequency *Select if everrpt_b=2 Everrpt_c3 How many times did you repeat Grade 3? … *Enter the frequency *Select if everrpt_b=3 Everrpt_c4 How many times did you repeat Grade 4? … *Enter the frequency *Select if everrpt_b=4 Everrpt_c5 How many times did you repeat Grade 5? … *Enter the frequency *Select if everrpt_b=5 Everrpt_c6 How many times did you repeat Grade 6? … *Enter the frequency *Select if everrpt_b=6 studattend During the last week of school, how many days did you attend school? …. *Make sure there was a normal week without a test or a holiday or a cultural ceremony. *Record attendance >=0 & <5 for one week *Mark 888 if the child does not know the answer/refuse to answer *If Grand Bassa, make sure that count Friday as working is part of their school activity. mainlang What language do you speak at home most often? 1. English 2. Kpelle 3. Grebo 4. Krahn 5. Bassa 6. Kru 7. Lorma I__I *Do not read options *Select only one option H6 | AIR.ORG LEARN Endline Evaluation Report 8. Belleh 9. Sapo 10. Other, specify______ 888. Don’t Know otherlang At home, do you speak any other languages? 1. English 2. Kpelle 3. Grebo 4. Krahn 5. Bassa 6. Kru 7. Lorma 8. Belleh 9. Sapo 10. Other specify _______ 11. No 888. Don’t Know I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I *Select all that apply *Do not read the options ses In your home, do you have any of the following items that I will read to you? 1. CELL PHONE 2. CURRENT/LIGHT/GENERATOR/SO LAR PANEL/POWER BANK 3. ICE BOX 4. BICYCLE 5. TV 6. MOTORBIKE/PEMPEM 7. CAR 8. KEHKEH 9. RADIO  SES2 10. None 888. Don't know I__I I__I I__I I__I I__I I__I I__I I__I I__I *Please read all the options to the child and select all that apply *Define home for the child as their own family that they spend most of their time with and not other households living with them in one place. Radio Is the radio functional? 0. No 1. Yes 888. Don’t know I__I *Do not read options *Select only one option H7 | AIR.ORG LEARN Endline Evaluation Report book At home do you have: 1. TEXTBOOKS/SCHOOLBOOKS 2. NEWSPAPERS 3. STORYBOOKS/COMICS 4. COLORING AND DRAWING BOOKS 5. HOLY BOOK (BIBLE OR KORAN) 6. None 888. Don't know I__I I__I I__I I__I I__I I__I I__I *Please read all the options to the child and select all that apply WASH [DON’T READ TO THE CHILD] Okay, now I have some questions about hygiene. Ask this section from students in both grades (Second and Sixth) Hand1 Did you wash your hands at all yesterday? 0. No hand4 1. Yes 888. Don’t know I__I *Select only one option Hand2 At what point did you wash your hands yesterday? 1. After using the toilet (poo poo) 2. After using the toilet (pee pee) 3. Before eating food 4. When they were dirty 5. After eating 6. After playing 7. Before preparing food 8. After helping someone else use the toilet 9. Right after coming home 10. After coughing or sneezing 11. Other, specify_________ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I I__I * Probe if the child refers to the time s/he washed he/his hands, ask them why they washed their hands at that time *Do not read the options to the child. *Select all that apply. Hand3 What did you use to wash your hands yesterday? 1. Water only 2. Water and soap 3. Ash 4. Hand sanitizer 5. Other, specify _____ 888. Don't know 999. Refuse to answer I__I *Do not read the options to the child. *Select only one option H8 | AIR.ORG LEARN Endline Evaluation Report Hand4 At what point should you wash your hands? 1. After using the toilet (poo poo) 2. After using the toilet (pee pee) 3. Before eating food 4. When they were dirty 5. After eating 6. After playing 7. Before preparing food 8. After helping someone else use the toilet 9. Right after coming home 10. After coughing or sneezing 11. Other, specify__________ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I *Do not read the options to the child. *Select all that apply. Hand5 When schools were closed because of COVID￾19, did anyone teach you about washing your hands? 0. No 1. Yes  eatfreq 2. Knew before closures 888. Don’t know I__I *Select only one option Hand7 When schools were closed because of COVID￾19/Coronavirus, where did you learn more about handwashing? 1. Through SMSs/phone text messages received from the SC LEARN team 2. Teaching by radio-based messages 3. My parents 4. Learned them from LEARN/Save the Children when I (or my parents) went to collect my take￾home rations 5. Volunteers from the community 6. Teachers (school health and nutrition champions) 7. Knew before closures 8. Other, specify_______ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I *Do not read the options to the child. *Select all that apply. H9 | AIR.ORG LEARN Endline Evaluation Report Food Security [DON’T READ TO THE CHILD] Thank you! Now, I would like to ask you some questions about food. Ask this section from students in both grades (Second and Sixth) eatfreq How many times do you eat per day? 1. More than three times per day 2. Three times per day 3. Twice per day 4. Sometimes two times, sometimes one time 5. Once per day 6. I eat once a day and sometimes not eat at all 888. Don't know 999. Refuse to answer I__I *Select only one option diet1 Do you know what does a "balanced diet" mean? 0. Nodiet3 1. Yesdiet2 888. Refuse to answer  diet3 I__I *Do NOT probe if the child does not understand *Select only one option diet2 Can you explain to me what a balanced diet is? 1. Eating foods that give us energy to play, work, learn (Go) 2. Eating foods that help us grow (Grow) 3. Eating foods that protect us from disease (Glow) 4. None of the above 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I *Probe if needed but do NOT read the options to the child *Select all that apply *For programming purpose - restrict selection of None of the above and 888 with other options. diet3 Can you name foods that give you energy to play and learn? 1. Grains like maize (corn), rice, fufu, bulgur, or pasta 2. Sweet foods like sugarcane, sugar, or honey 3. Roots like potato, yam, cassavas, eddos, or sweet potato 4. Fats like margarine (butter), or oils 5. Other (Specify) I__I I__I I__I I__I I__I I__I *Probe if needed but do NOT read the options to the child *Select all that apply *For programming purpose - restrict selection of None of the above and 888 with other options. H10 | AIR.ORG LEARN Endline Evaluation Report 888. Don't know 999. Refuse to answer diet4 Can you name foods that help your body grow? 1. Dairy products like milk, yogurt, and cheese 2. Red meat 3. Poultry (chicken) 4. Fish 5. Eggs 6. Beans, peas, legumes/pulses like seeds and nuts 7. Other (specify) 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I *Probe if needed but do NOT read the options to the child *Select all that apply *For programming purpose - restrict selection of None of the above and 888 with other options. diet5 Can you name foods that protect your body from disease? 1. Green leafy vegetables like potato greens, spinach, collard green, cassava greens, watergreens 2. Fruits like mango, banana, pawpaw, oranges, pineapple, watermelon, or cucumber 3. Okra 4. Cauliflower 5. Pumpkin 6. Other (specify) 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I *Do NOT read the options to the child *Select all that apply *For programming purpose - restrict selection of None of the above and 888 with other options. diet6 How do you think the food should be divided between boys and girls? 1. Boys should get more 2. Girls should get more 3. Boys and girls should get equal amounts 888. Don't know 999. Refuse to answer I__I *Select only one option *Probe if necessary but do not lead them to an answer *Do not read the options to them canteen1 Did you eat a meal that was prepared at school for free yesterday? 0. No 1. Yes 2. No food was prepared I__I *Select only one option *Probe if necessary *If the interview is on Monday, ask the child about Friday or the last time the child was at H11 | AIR.ORG LEARN Endline Evaluation Report 888. Don't know 999. Refuse to answer school. If the child was absent yesterday, ask about the last time the child was at school. SCHOOL ENVIRONMENT AND PARTICIPATION [DON’T READ TO THE CHILD] That's great! You did a good job! Now I want to ask you a couple of questions about your school. The following questions are only for Grade 2 students. Enviro0 Do you like coming to school? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 888. Don’t know 999. Refuse to answer I__I *Select only one option enviro1 What do you like best about your class and school? 1. Like teacher 2. Learning new things/enjoy lessons 3. Participate in classroom games and activities 4. Playing a sport at school 5. Access to water 6. Access to clean toilet 7. Food is provided 8. Being with my friends 9. Other (specify)_______ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I I__I I__I *Select all that apply. *Do not read the options to the child. H12 | AIR.ORG LEARN Endline Evaluation Report enviro2 What do you not like about your class and school? 1. Teacher is mean to me/other students 2. S/he punishes me/ hits me/other students 3. Teacher asks for money 4. Lessons difficult to understand/learn 5. Not learning much at school 6. Poor toilet conditions/lack of toilets 7. No access to water 8. No food is provided/the food is bad 9. Other students tease me/fight with me/other students 10. I don’t feel safe at school 11. Lack of uniform 12. Lack of learning materials 13. Lessons are boring 14. Other (specify) 888. Don't Know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I * Do not read the options to the child * Select all that apply *Note to enumerators: Mean can be yelling, laughing at students, or humiliating them, etc. Enviro2a Do you feel valued and respected at school by teachers? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 888. Don’t know 999. Refuse to answer I__I *Select only one option enviro3 How many times in the last week did your teacher come to class? 1. Every day (5 days) 2. A few times during the week (2-4 days) 3. Once during the week 4. Never  enviro5 888. Don’t know 999. Refuse to answer I__I *Read the list to the respondent, but don't read 'don't know' *Select only one *Make sure there was a normal week without a test or a holiday or a cultural ceremony. *Don’t ask if the child did not attend school for the full week last week H13 | AIR.ORG LEARN Endline Evaluation Report enviro4 How many times in the last week did your teacher come late or miss a portion of the class? 1. EVERY DAY (5 DAYS) 2. A FEW TIMES DURING THE WEEK (2- 4 DAYS) 3. ONCE DURING THE WEEK 4. NEVER 888. Don’t know 999. Refuse to answer I__I *Read the list to the respondent, but don’t read “don’t know” *Select only one *Don’t ask if the child did not attend school for the full week last week enviro5 Does your school have books other than textbooks/schoolbooks for you to borrow? If yes, is it free, or do you have to pay money? 0. No  nhhold 1. Yes, we can take books, but not off campus  nhhold 2. Yes, we can take books home and it is free  enviro5a 3. Yes, we can take books home, but it costs money enviro5a 888. Don't know 999. Refuse to answer I__I *Select only one option enviro5a How many times in the last week did you borrow books other than textbooks/school books from school to take home to read? 1. EVERY DAY 2. A FEW TIMES DURING THE WEEK; 3. ONCE DURING THE WEEK; 4. NEVER 888. Don't know 999. Refuse to answer I__I *Read the list to the respondent, but don't read 'don't know' *Select only one Household Environment [DON’T READ TO THE CHILD] We are almost done! We have a few more questions about your home. The following questions are only for Grade 2 students. Nhhold How many people are there in your household, including yourself? ……. *Define the household for the child as a place where its members live with each other, eat out of the same pot *Record the number > 0 & < 40 Nhhold 3 Among all these people in your household, how many are able to read and write? …… *Enter 0 if they have none in any of the categories H14 | AIR.ORG LEARN Endline Evaluation Report *Enter 888 if do not know *Record the number >= 0 Nhhold2 Can you tell me the total number of sisters and brothers who live with you in the same house? 1. Older sisters 2. Younger sisters 3. Older brothers 4. Younger brothers …… …… ……. …… *Enter 0 if they have none in any of the categories *Enter 888 if do not know *Record the number >= 0 hh1 In the last week, did you see anyone in your house reading? 0. No  hh2 1. Yes  hh1a 888. Don't know 999. Refuse to answer I__I *Select only one option hh1a Who did you see reading last week? 1. Mother 2. Father 3. Older sister 4. Younger sister 5. Older brother 6. Younger brother 7. Grandmother 8. Grandfather 9. Other female relative 10. Other male relative 11. Female non-relative 12. Male none-relative 888. Don't know I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I *Select all that apply *Do not read the options to them hh2 In the past week, did anyone in your household help you with your studies/school work? 0. No  hh3 1. Yes  hh2a 888. Don't know 999. Refuse to answer I__I *Select only one option hh2a Who helped you study? 1. Mother 2. Father 3. Older sister 4. Younger sister I__I I__I I__I I__I I__I *Select all that apply *Do not read the options to them H15 | AIR.ORG LEARN Endline Evaluation Report 5. Older brother 6. Younger brother 7. Grandmother 8. Grandfather 9. Other female relative 10. Other male relative 11. Female non-relative 12. Male none-relative 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I hh3 In the past week, did anyone in your house read to you? 0. No  hh4 1. Yes  hh3a 888. Don't know I__I *Select only one option. hh3a Who read to you? 1. Mother 2. Father 3. Older sister 4. Younger sister 5. Older brother 6. Younger brother 7. Grandmother 8. Grandfather 9. Other female relative 10. Other male relative 11. Female non-relative 12. Male none-relative 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I *Select all that apply *Do not read the options to them hh4 In the past week, did anyone in your house tell you a story? 0. No readout1 1. Yes  hh4a 888. Don't know 999. Refuse to answer I__I *Select only one option. hh4a Who told you a story? 1. Mother 2. Father 3. Older sister 4. Younger sister 5. Older brother I__I I__I I__I I__I I__I I__I *Select all that apply *Do not read the options to them H16 | AIR.ORG LEARN Endline Evaluation Report 6. Younger brother 7. Grandmother 8. Grandfather 9. Other female relative 10. Other male relative 11. Female non-relative 12. Male none-relative 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I readout1 During the last week, did you read books other than textbooks/schoolbooks outside of school? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option readout2 Outside of your school or home, where else can you go to read or borrow books (other than textbooks)? 1. Community library 2. Church/Mosque or any other religious building 3. Reading clubs 4. Friends or relatives 5. Other 888. Don't know  readout2_enum 999. Refuse to answer  readout2_enum I__I I__I I__I I__I I__I *Select all that apply *Do not read the options to them readout2_enum FOR ENUMERATORS ONLY [DO NOT ASK THE CHILD] Why did you choose “888”or “999” in the previous question? 1. The child did not know the answer/refusedreadenjoy1 2. The child mentioned s/he has nowhere to go outside of school for reading  readenjoy1 3. Other (Specify)  readenjoy1 I__I *Select only one option readout3 Did you read books (other than textbooks) in any of those places you mentioned before [readout2 option]? 0. No 1. Yes I__I *Select only one option H17 | AIR.ORG LEARN Endline Evaluation Report 888. Don't know 999. Refuse to answer readenjoy1 Do you enjoy reading? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 888. Don’t know 999. Refuse to answer I__I *Select only one option Readenjoy2 Do you consider yourself to be a good reader? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 888. Don’t know 999. Refuse to answer I__I *Select only one option Readenjoy3 Have you used the ‘I help my child to learn’ tool with your parent/caregiver? 0. No 1. Yes 888. Don't know I__I *Select only one option Readenjoy4 Do you feel supported by your parents/caregivers in your learning and well￾being? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 888. Don’t know 999. Refuse to answer I__I *Select only one option Sexual and Gender-based Violence [DON’T READ TO THE CHILD] Thank you! Now, I would like to ask your opinion about something. There is no right or wrong answer. Ask this section from students in both grades (Second and Sixth) Conduct1 Have you ever heard of a teacher lying to get something they want or to get out of trouble? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option H18 | AIR.ORG LEARN Endline Evaluation Report Conduct1a Have you ever heard of a teacher stealing things from school? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct2 Have you ever heard of a teacher offering money to get something they want, or taking money from someone to give them what they want? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct3 Have you ever heard a teacher make a comment about a student’s body, or their in-front part, or behind part, or their chest part? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct4 Have you ever heard about a teacher touching a child on their behind part, chest part, or their in front part? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct5 Did you hear of any teachers coming to school drunk or high on drugs last week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct6 Did you hear of any teachers teasing/calling children names in the last week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct6_a In the last week, how many times did you hear about this happening to boys? …. Ask if conduct6 = Yes H19 | AIR.ORG LEARN Endline Evaluation Report Conduct6_b In the last week, how many times did you hear about this happening to girls? …. Ask if conduct6 = Yes Conduct7 Did you see a teacher treating one student better than any of the other students last week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct8 Last week, did any teacher fail to show up at school? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct8_a In the last week, how many teachers in your school were absent, including your own teacher? …. *Enter a number and add 888 if the students do not know the answer Conduct9 Did you see a teacher use corporal punishment last week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct9_a In the last week, how many times did you see this happening to boys? …. Ask if Conduct9=Yes Conduct9_b In the last week, how many times did you see this happening to girls? …. Ask if Conduct9=Yes Conduct10 If a teacher or school administrator acted violently towards you, would you tell anyone? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Conduct11 Did you witness any violence in the classroom in the past week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option *Explain that violence can include hitting, verbal abuse, humiliation, sexual comments sgbv1 Are there rules for the ways that teachers should treat students in school? 0. No -> sgbv3 I__I Probe if needed H20 | AIR.ORG LEARN Endline Evaluation Report 1. Yes -> sgbv2 888. Don't know sgbv2 What are they? 1. Teachers are not allowed to be in a relationship with students 2. Teachers are not allowed to beat students 3. Teachers are not allowed to use humiliating language on students 4. Teachers are not allowed to ask students for money 5. Teachers should not favor one student over the other 6. Teachers are not allowed to make a comment about students’ body, or their private parts (sexual harassment). 7. Teachers are not allowed to touch a student on their private parts (sexual abuse). 8. Teachers are not allowed to force students to work on their teacher’s farm as a punishment 9. Other (specify)________ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I * Do not read the options to the child * Select all that apply *Note that this is an illustrative list and their answers do not need to follow the exact wording. For example, if a child respond teachers should not love students, this can go under “Teachers are not allowed to be in a relationship with students”. Sgbv2_c Are there any other general rules for teachers in school? 0. No -> sgbv2_b 1. Yes -> sgbv2_d 888. Don't know I__I *Select only one option Srgbv2_d What are they? 1. Teachers are not allowed to come to school drunk or high on drugs 2. Teachers should not steal from school 3. Teachers are not allowed to arrive late or leave school early with no excuse 4. Teachers are not allowed to fail to show up at school unexpectedly 5. Other specify_______ 888. Don’t know *Select all that apply *Do not read out the options H21 | AIR.ORG LEARN Endline Evaluation Report 999. Refuse to answer Sgbv2_b How did you learn about the rules? 1. Rules posted in the school 2. Head teacher/principal 3. Your teacher 4. Parents 5. Other students 6. Other (Specify) 888. Don't know 999. Refuse to answer I__I *Select only one option sgbv3_boys How do teachers discipline boys at school? 1. Give extra work/assignments 2. Dismiss students from class 3. Physical violence (hitting students) 4. Humiliating language 5. Made to clean or work at the school 6. Other (specify) 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I * Probe if needed *Do not read the options to the child * Select all that apply Sgbv3_b_boys In your opinion, are boys afraid to go to school for fear of punishment? 0. Never 1. Rarely 2. Some of the time 3. Always 888. Don't know 999. Refuse to answer I__I *Select only one option sgbv3_girls How do teachers discipline girls at school? 1. Give extra work/assignments 2. Dismiss students from class 3. Physical violence (hitting students) 4. Humiliating language 5. Made to clean or work at the school I__I I__I I__I I__I I__I * Probe if needed *Do not read the options to the child * Select all that apply H22 | AIR.ORG LEARN Endline Evaluation Report 6. Other (specify) 888. Don't know 999. Refuse to answer Sgbv3_b_girls In your opinion, are girls afraid to go to school for fear of punishment? 1. Never 2. Rarely 3. Some of the time 4. Always 888. Don't know 999. Refuse to answer I__I *Select only one option sgbv4 If children are teased or touched in a way they don't like at school, what do they do? 1. Tell their teacher 2. Tell the principal or registrar 3. Tell their parents 4. Tell Management Committee 5. Tell the Police 6. Tell the Community leader (Village chief leader) 7. Tell Child services NGO (UN hotline, WONGOSOL, or LEARN Orange hotline) 8. Nothing 9. Other (specify) 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I * Probe if needed *Do not read the options to the child * Select all that apply Sgvb5 Do teachers or school officials take action when students report violence? 0. Never 1. Rarely 2. Some of the time 3. Always 888. Don't know 999. Refuse to answer I__I *Select only one option *It could be any violence that may happen in school (gender based or physical or any other types) Sgbv6 Have you listened to at least 2 safe school stories on the radio in the past week? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option Gender norms H23 | AIR.ORG LEARN Endline Evaluation Report I'm going to read you things that some children agree with and some children disagree with. After I read each one, please tell me if yes you agree or no you disagree. The following questions are only for Grade 6 students. gender1 If a boy touches a girl at school, it’s because the girl did something to attract him 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option gender2 There are times when a boy needs to beat his girlfriend/female friend 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option gender3 Girls like to be teased by boys 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option gender4 When girls wear short skirts, they are telling boys or men to touch them 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option gender5 For girls to get good grades, they sometimes have to let their teachers touch them or love them 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option Gender6 Women can lead community meetings and make important decisions 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option Gender7 Men and boys can help prepare and cook food 1. Disagree 2. Agree 888. No response/Not sure I__I *Select only one option Gender8 Who should help the family the most with housework? 0. Boys 1. Girls 2. Both boys and girls I__I *Select only one option H24 | AIR.ORG LEARN Endline Evaluation Report 888. Don't know Gender9 Who should help the family the most with farm work? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option Gender10 For whom is it more important to go to school? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option Gender11 Who should help more in carrying out school chores such as cleaning classrooms and toilets? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option Gender12 Who receives more negative comments and insults from teachers? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option Gender13 Who receives more positive comments from teachers? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option Gender14 Whom do teachers choose to answer questions most frequently? 0. Boys 1. Girls 2. Both boys and girls 888. Don't know 999. Refuse to answer I__I *Select only one option H25 | AIR.ORG LEARN Endline Evaluation Report Disability [DON’T READ TO THE CHILD] Thank you! You are doing a great job! Ask this section from students in both grades (Second and Sixth) dis1 Do you have difficulty seeing? For example, is it difficult to see the chalkboard when you are at school, even if you sit near the front of the classroom, or when you wearing your glasses (mention this example if they wear glasses)? What about when you sit at the back of the classroom? 0. No – no difficulty 1. Yes – some difficulty 2. Yes – a lot of difficulty 3. Cannot do at all 888. Don't know I__I *Select only one option ***Make sure difficulty is not because students are blocked by taller students in front of them dis2 Do you have difficulty hearing? For example, if you were in the main room of your house, could you hear someone talking in a normal voice on the other side of the room, or even when you wearing your hearing aid (only ask if you see they have hearing aid)? 0. No – no difficulty 1. Yes – some difficulty 2. Yes – a lot of difficulty 3. Cannot do at all 888. Don't know I__I *Select only one option dis3 Do you have difficulty walking or climbing steps? For example, is it difficult to move around in your home? 0. No – no difficulty 1. Yes – some difficulty 2. Yes – a lot of difficulty 3. Cannot do at all 888. Don't know I__I *Select only one option Diarrhea Disease Recall [DON’T READ TO THE CHILD] You are doing a great job! We are almost done! Ask this section from students in both grades (Second and Sixth) diar1 Have you had diarrhea in the last 2 weeks? 0. No -> cult1 1. Yes 888. Don't know I__I *Select only one option H26 | AIR.ORG LEARN Endline Evaluation Report 999. Refuse to answer diar2 When you had diarrhea, did you eat as much as usual, less than usual, more than usual, or nothing? 1. Much less 2. Somewhat less 3. About the same 4. More 5. Stopped food 6. Never ate food 888. Don't know 999. Refuse to answer I__I *Select only one option **If less, probe: less than usual or somewhat less? diar3 Did you seek advice or treatment for the diarrhea from any source? 0. No -> diar5 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option diar4 Where did you seek advice or treatment? 1. Government hospital/health center/health post 2. Community health worker 3. Private hospital/clinic/doctor/pharmacy 4. NGO hospital/clinic 5. Traditional practitioner/shop/market 6. Other specify __________ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I *Select all that apply diar5 Were you given any of the following at any time since you started having diarrhea? 0. No treatment was given 1. Increased fluids 2. ORS packet/pre-packaged ORS liquid 3. Zinc tablets or syrup 4. Government recommended homemade fluids (RHF) 5. Antibiotics 6. Home remedy 7. Other specify_________ 888. Don't know 999. Refuse to answer I__I I__I I__I I__I I__I I__I *Select all that apply **If “0” is chosen, no other option can be chosen H27 | AIR.ORG LEARN Endline Evaluation Report Cultural appropriateness of school meals [DON’T READ TO THE CHILD] Thank you! You are doing a great job! We are almost done! Then we can play the reading game! Ask this section from students in both grades (Second and Sixth) cult1 How similar is the school meal you receive to what you eat at home? 1. Not similar at all 2. Quite similar 3. Exactly identical 888. Don't know 999. Refuse to answer I__I *Select only one option cult2 Are you served different types of foods in your school meals? 0. No 1. Yes 888. Don't know 999. Refuse to answer I__I *Select only one option cult3 Are you served anything at school which your parents tell you to not eat for cultural/religious reasons? 0. No  cult5 1. Yes 888. Don't knowcult5 I__I *Select only one option *Examples include: catfish, honey, pumpkin, palm kernel, oil, duck cult4 Can you name these items? ----- *Enter 888 if don’t know Cult5 Do you enjoy the meals that you are served at school? 0. No 1. Yes 2. I like some of the food and dislike some of the food 3. 888. Don't know 4. 999. Refuse to answer I__I *Select only one option Food intake recall [DON’T READ TO THE CHILD] Thank you! You are doing a great job! We are almost done! Then we can play the reading game! Ask this section from students in both grades (Second and Sixth) foodintake1 How many meals do you usually have each day on a school day (i.e., weekday during the school year)? ___ *Enter 888 if don’t know H28 | AIR.ORG LEARN Endline Evaluation Report foodintake3 Do you eat any of your meals at school? 0. No 1. Yes I__I *Select only one option foodintake2 How many meals do you usually have each day when you are not in school (i.e., weekend, school holiday)? ___ *Enter 888 if don’t know foodintake4 Which meals do you eat at school? 1. Breakfast 2. Lunch 3. Dinner 4. Snack 888. Don't know 999. Refuse to answer I__I I__I I__I I__I *Ask if foodintake3=Yes *Select all that apply foodintake5 Who provides the meals you eat at school? 1. School 2. Caregiver foodintake5b 3. Self 4. Other, specify 888. Don't know I__I I__I I__I I__I *Ask if foodintake3=Yes *Select all that apply Foodintake5b Is this person a man or a woman? 1. Man 2. Woman 888. Don't know 999. Refuse to answer I__I *Select only one option Ask only if foodintake5 = “Caregiver” foodintake6 Which of the following food and drink did you consume for breakfast yesterday? 1. Nothing was consumed 2. Rice 3. Soup 4. Fufu 5. Dumboy 6. Mango 7. Pawpaw 8. Banana 9. Plantain 10. orange 11. Breadfruit 12. Butter pear 13. Yam 14. Eddo 15. Cassava 16. Watermelon *Select all that apply *Read out options to students H29 | AIR.ORG LEARN Endline Evaluation Report 17. Pumpkin 18. Peanut soup 19. Goat meat soup 20. Potato greens 21. Palava sauce 22. Fish 23. Rice bread 24. Chicken gravy 25. Torborgee 26. Kanyah 27. Palm butter soup 28. Pepper soup 29. Water 30. Juice 31. Milk 32. Other, specify 888. Don't know 999. Refuse to answer foodintake7 Which of the following food and drink did you consume for a snack yesterday? 1. Nothing was consumed 2. Rice 3. Soup 4. Fufu 5. Dumboy 6. Mango 7. Pawpaw 8. Banana 9. Plantain 10. orange 11. Breadfruit 12. Butter pear 13. Yam 14. Eddo 15. Cassava 16. Watermelon 17. Pumpkin 18. Peanut soup 19. Goat meat soup 20. Potato greens 21. Palava sauce 22. Fish 23. Rice bread *Select all that apply *Explain that snacks are foods that are eaten in between meals, breakfast, lunch, and dinner *Read out the options H30 | AIR.ORG LEARN Endline Evaluation Report 24. Chicken gravy 25. Torborgee 26. Kanyah 27. Palm butter soup 28. Pepper soup 29. Water 30. Juice 31. Milk 32. Other, specify 888. Don't know 999. Refuse to answer foodintake8 Which of the following food and drink did you consume for lunch yesterday? 1. Nothing was consumed 2. Rice 3. Soup 4. Fufu 5. Dumboy 6. Mango 7. Pawpaw 8. Banana 9. Plantain 10. orange 11. Breadfruit 12. Butter pear 13. Yam 14. Eddo 15. Cassava 16. Watermelon 17. Pumpkin 18. Peanut soup 19. Goat meat soup 20. Potato greens 21. Palava sauce 22. Fish 23. Rice bread 24. Chicken gravy 25. Torborgee 26. Kanyah 27. Palm butter soup 28. Pepper soup 29. Water 30. Juice *Select all that apply *Read out the options H31 | AIR.ORG LEARN Endline Evaluation Report 31. Milk 32. Other, specify 888. Don't know 999. Refuse to answer foodintake9 Which of the following food and drink did you consume for dinner yesterday? 1. Nothing was consumed 2. Rice 3. Soup 4. Fufu 5. Dumboy 6. Mango 7. Pawpaw 8. Banana 9. Plantain 10. orange 11. Breadfruit 12. Butter pear 13. Yam 14. Eddo 15. Cassava 16. Watermelon 17. Pumpkin 18. Peanut soup 19. Goat meat soup 20. Potato greens 21. Palava sauce 22. Fish 23. Rice bread 24. Chicken gravy 25. Torborgee 26. Kanyah 27. Palm butter soup 28. Pepper soup 29. Water 30. Juice 31. Milk 32. Other, specify 888. Don't know 999. Refuse to answer *Select all that apply *Read out the options foodintake10 Was yesterday a typical day in terms of the types of food you consumed? 0. No I__I *Select only one option H32 | AIR.ORG LEARN Endline Evaluation Report 1. Yes 888 Don’t know 999. Refuse to answer H33 | AIR.ORG LEARN Endline Evaluation Report The literacy assessment is only for Grade 2 students. Observational Questions This Question is for the enumerator – DO NOT ASK THIS QUESTION FROM THE CHILD If a Grade two student, check this question at the end of the literacy assessment. Canteen1_obs DOES THE SCHOOL HAVE A CANTEEN? 0. No, there is no canteen available in the school 1. Yes, there is an inactive canteen 2. Yes, there is an active canteen 3. Other (Specify) --------------- I__I *An inactive canteen means it is no longer functional not that it is temporarily closed because it is not meal time at the moment H34 | AIR.ORG LEARN Endline Evaluation Report LITERACY BOOST ASSESSMENT Understanding Letters 1. Give the child the list of letters and say to the child: 2. Say: Let’s look at some letters. Can you start here (point to first letter) and tell me what these letters are moving in this direction? (indicate left to right direction) Do you understand? Ok, you can begin. 3. Mark the letters correct or incorrect as the child reads. 4. Correct letters are:  the letter name in the home language or language of instruction  any sound that is acceptable for in the home or instructional language  a response which says “It begins like…” giving a word for which the letter is the initial letter 5. If the child read the letters out of order, then remember to bring his/her attention to the ones they might have skipped. 6. Make sure you marked all of the letters 7. Move to the Most Used Words section. What to do if a student is struggling:  If the student is struggling, and hesitates at any letter for five seconds, ask to follow up questions: Do you know its name? What sound does it make? Do you know a word that starts with this letter?  If the student still hesitates for five seconds, ask: Can you tell me any of these letters?  If the student still hesitates for five seconds, then stop and thank him/her for trying his/her best.  Mark letters not identified or not attempted as incorrect.  Move to the Most Used Words section. x v s o a k g c f b p l h d z t q m i e w u r n j y H35 | AIR.ORG LEARN Endline Evaluation Report H36 | AIR.ORG LEARN Endline Evaluation Report Most Used Words 1. Give the pupil the laminated copy of the "Most Used Words" list. 2. Say: I would like you to read some words to me. They are words from your textbook. Please point to and say each of these words starting here (point to first word) and moving across each line like this (indicate left to right direction). Do you understand? Ok, you can begin. 3. Mark the words correct or incorrect as the child reads 4. Remember that pronunciations of words with local accent are acceptable. 5. If the child read the words out of order, then remember to bring his/her attention to the ones they might have skipped. 6. Make sure you marked all of the words. 7. Move to the Decoding Section. What to do if a student is struggling:  If the student is struggling, and hesitates at any words for five seconds ask the child, Are there any words on the list that you know? Tell me or say the words you know. Repeat the request to encourage the child to continue.  If the student still hesitates for five seconds, then stop and thank him/her for trying his/her best.  Mark words not identified or not attempted as incorrect. your his uncle we school girls want help and said story room go she will ask not was mother did H37 | AIR.ORG LEARN Endline Evaluation Report Invented words 1. Give the pupil the laminated copy of the "Invented Words" list. 2. Say: I would like you to read another list of words to me. These words are not real words, rather they are words that we made up ourselves. But they can still be read. Please point to and say each of these words starting here (point to first word) and moving across each line like this (indicate left to right direction). Do you understand? Ok, you can begin. 3. Mark the words correct or incorrect as the child reads. 4. Remember that pronunciations of words with local accents are acceptable. 5. If the child read the words out of order, then remember to bring his/her attention to the ones they might have skipped. 6. Make sure you marked all of the incorrect words. 7. Move to the Reading Passage section. What to do if a student is struggling:  If the child hesitates at any word for five seconds, ask the child, Are there any words on the list that you know? Tell me or say the words you know. Repeat the request to encourage the child to continue.  If the student still hesitates for five seconds, then stop and thank him/her for trying his/her best.  Mark words not identified or attempted as incorrect.  Move to the Reading Passage section. jour mir undle ne sprood kirls vakt gelb alt baid flory koom vo phe yill asb dok sar rothem thu H38 | AIR.ORG LEARN Endline Evaluation Report COMPREHENSION PASSAGES AND QUESTIONS 1. Give the pupil the reading passage. 2. Say: I am going to give you a reading passage to read. When I say 'begin,' start reading aloud from the title on this page. Try to read each word. If you come to a word you don't know, I'll tell it to you. Be sure to try to do your best reading. Do you understand what I want you to do? 1. Say: 'Begin' and when the pupil begins to say the first word of the title press START. 2. As the pupil reads, follow along on your screen. Click on words read incorrectly (they will turn with a line through them). 3. If the pupil stops reading before the end of the passage, encourage the pupil to keep reading. Show the pupil where he/she stopped, if necessary. Follow along on your copy. If the child does not want to or cannot read anymore, stop the timer and select the last word the child read. Thank the child for reading it and read it out to him/her. 4. After 30 seconds, a message will flash, “Please mark the item being attempted.” Mark the word that the child was reading when the message came, and a blue box will appear around it. 5. When the screen flashes at the end of 30 seconds, do a quick count of the correct words.  If the pupil has read less than 5 words correctly, then: o Politely stop the child and Press “Finish” box to stop the timer. Say: Thank you. o Read the passage to them. o On the next page, mark NONREADER o And ask them comprehension questions.  If the pupil has read 5 or more words correctly, then: o Select the box under the word being read/attempted by the child at 30 seconds. o Allow the pupil to finish the passage. o Continue marking which words are read incorrectly by clicking on them. o As soon as the pupil finishes the last word of the passage, click the FINISH button. Say: Thank you. o On the next page, for the question, ‘Was the student a reader or nonreader?’ mark READER. o Move to the Reading Comprehension questions What to do if a student is struggling:  If the pupil is struggling and fails to correctly pronounce a word within five seconds, tell him/her the word and mark it as an error by clicking on it (the word should appear with a line through it). H39 | AIR.ORG LEARN Endline Evaluation Report The Lone Star Kite! One hot day, all the children were outside playing. Many were flying kites high in the sky. Moses looked at the kite that his older sister Mary made for him. It had red and white stripes and a blue lone star at the top. It looked great. Moses was proud of his kite. He ran up the hill. Moses ran so fast that he fell down and broke his kite. Moses began to cry. Mary came down from the hill. “Why are you crying?” she asked. “My kite is broken,” said Moses. “I will fix it,” said Mary. Moses trusted his sister. Mary fixed the kite with glue. She handed it to Moses. “Try it now!” Moses ran and the wind carried the kite in the air. All the children came running to look at the beautiful Lone Star kite. Moses was right – his big sister always knew what to do. Question to enumerator – DO NOT ASK THIS QUESTION FROM THE CHILD reader Is child a reader or a non-reader? 0. A non-reader read fewer than 5 words accurately 30 seconds) à reader_confirm 1. A reader (read correctly 5 per 30 seconds) à nonreader_confirm I__I Select only one option Reader_ comfirm What kind of reader did you survey? 1. A perfect reader who finished the passage in less than 6 minutes on her/his own 2. A reader who was not able to finish the passage in 6 minutes, and I read the remainder of the passage to her/him after 6 minutes I__I Select only one option H40 | AIR.ORG LEARN Endline Evaluation Report 3. A reader who could not read the entire passage or gave up in the middle and I read the remainder of the passage to her/him nonreader_ confirm What kind of non-reader did you survey? 1. A non-reader who was not able to read at all and I read the passage to her/him after 30 second 2. A non-reader who was only able to read 1-4 words and I read the passage to her/him after 30 second I__I Select only one option Comprehension Questions Comp1 What happened in the story? 1. Moses wants to fly the kite that his sister made 2. Moses falls and breaks his kite 3. Moses‘s sister fixes the kite 4. Moses is able to fly the kite 5. None I__I mark every main point mentioned by the child Comp2 Who made the kite for Moses? (His older sister, Mary) 0. False 1. True I__I Don’t read the answer to them Comp3 What did the kite look like? (Lone Star/red and white stripes with blue star) 0. False 1. True I__I Don’t read the answer to them Comp4 How did the kite break? (Moses tripped and dropped it) 0. False 1. True I__I Don’t read the answer to them Comp5 Who fixed Moses’s kite? (his sister, Mary) 0. False 1. True I__I Don’t read the answer to them Comp6 How did Mary fix the kite? (with glue) 0. False 1. True I__I Don’t read the answer to them Comp7 Does the kite fly at the end of the story? (yes) 0. False 1. True I__I Don’t read the answer to them H41 | AIR.ORG LEARN Endline Evaluation Report Comp8 Why was Moses proud of his kite? (his sister made it for him/it was a Lone Star kite) 1. Student could explain their answer with information from the story 2. Student could NOT explain their answer with information from the story I__I Don’t read the answer to them Comp9 How did Moses feel after he broke his kite? (Sad or depressed) 0. False 1. True I__I Don’t read the answer to them Comp10 Why do you think Mary was a good sister? 0. False 1. True True if student can support opinion with details from story Thank you very much for answering my questions. End time ………………….. Comment …………………… H1 | AIR.ORG LEARN Endline Evaluation Report School Assessment School Assessment for USDA FOOD FOR EDUCATION (LEARN) IN LIBERIA Start time End time Date County District School name gps GPS coordinates Variable Item Response Options Instructions Enrollment Information Enrollment Please see the principal for the enrollment list for 2021-2022 enrollABCb What is the number of boys enrolled in ABC? *Use registration lists to populate enrollABCg What is the number of girls enrolled in ABC? *Use registration lists to populate enrollKGb What is the number of boys enrolled in KG? *Use registration lists to populate enrollKGg What is the number of girls enrolled in KG? *Use registration lists to populate enroll1b What is the number of boys enrolled in 1st grade? *Use registration lists to populate enroll1g What is the number of girls enrolled in 1st grade? *Use registration lists to populate enroll2b What is the number of boys enrolled in 2nd grade? *Use registration lists to populate H2 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions enroll2g What is the number of girls enrolled in 2nd grade? *Use registration lists to populate enroll3b What is the number of boys enrolled in 3rd grade? *Use registration lists to populate enroll3g What is the number of girls enrolled in 3rd grade? *Use registration lists to populate enroll4b What is the number of boys enrolled in 4th grade? *Use registration lists to populate enroll4g What is the number of girls enrolled in 4th grade? *Use registration lists to populate enroll5b What is the number of boys enrolled in 5th grade? *Use registration lists to populate enroll5g What is the number of girls enrolled in 5th grade? *Use registration lists to populate enroll6b What is the number of boys enrolled in 6th grade? *Use attendance lists to populate enroll6g What is the number of girls enrolled in 6th grade? *Use registration lists to populate Note Insert your comment if any, especially if the enrollment list is not available grade6_comp_b Number of boys successfully completing Grade 6 last year (2020-2021) H3 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions grade6_comp_g Number of girls successfully completing Grade 6 last year (2020-2021) Dropout information for students in 2019-2020 dropout_abc_b_19 Number of boys who dropped out of ABC during the last academic year (2019- 2020) dropout_abc_g_19 Number of girls who dropped out of ABC during the last academic year (2019- 2020) dropout_kg_b_19 Number of boys who dropped out of KG during the last academic year (2019- 2020) dropout_kg_g_19 Number of girls who dropped out of KG during the last academic year (2019- 2020) dropout_g1_b_19 Number of boys who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g1_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g2_b_19 Number of boys who dropped out of Grade 2 during the last academic year (2019- 2020) H4 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions dropout_g2_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g3_b_19 Number of boys who dropped out of Grade 3 during the last academic year (2019- 2020) dropout_g3_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g4_b_19 Number of boys who dropped out of Grade 4 during the last academic year (2019- 2020) dropout_g4_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g5_b_19 Number of boys who dropped out of Grade 5 during the last academic year (2019- 2020) dropout_g5_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) dropout_g6_b_19 Number of boys who dropped out of Grade 6 during the last academic year (2019- 2020) H5 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions dropout_g6_g_19 Number of girls who dropped out of Grade 1 during the last academic year (2019- 2020) Note Insert your comment if any, especially if the dropout list is not available Dropout information for Teachers for 2019-2020 school year dropout_abc_t Did the teacher in ABC drop-out? 0. No 1. Yes *Select only one option dropout_g1_t Did the teacher in Grade 1 drop-out? 0. No 1. Yes *Select only one option dropout_g2_t Did the teacher in Grade 2 drop-out? 0. No 1. Yes *Select only one option dropout_g3_t Did the teacher in Grade 3 drop-out? 0. No 1. Yes *Select only one option dropout_g4_t Did the teacher in Grade 4 drop-out? 0. No 1. Yes *Select only one option dropout_g5_t Did the teacher in Grade 5 drop-out? 0. No 1. Yes *Select only one option Dropout information for students in 2020-2021 dropout_abc_b Number of boys who dropped out of ABC during the last academic year (2020- 2021) dropout_abc_g Number of girls who dropped out of ABC during the last academic year (2020- 2021) dropout_kg_b Number of boys who dropped out of KG during the last academic year (2020- 2021) H6 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions dropout_kg_g Number of girls who dropped out of KG during the last academic year (2020- 2021) dropout_g1_b Number of boys who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g1_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g2_b Number of boys who dropped out of Grade 2 during the last academic year (2020- 2021) dropout_g2_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g3_b Number of boys who dropped out of Grade 3 during the last academic year (2020- 2021) dropout_g3_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g4_b Number of boys who dropped out of Grade 4 during the last academic year (2020- 2021) H7 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions dropout_g4_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g5_b Number of boys who dropped out of Grade 5 during the last academic year (2020- 2021) dropout_g5_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) dropout_g6_b Number of boys who dropped out of Grade 6 during the last academic year (2020- 2021) dropout_g6_g Number of girls who dropped out of Grade 1 during the last academic year (2020- 2021) Note Insert your comment if any, especially if the dropout list is not available Dropout information for Teachers in 2020-2021 school year dropout_abc_t_20 Did the teacher in ABC drop-out? 0. No 1. Yes *Select only one option dropout_g1_t_20 Did the teacher in Grade 1 drop-out? 0. No 1. Yes *Select only one option dropout_g2_t_20 Did the teacher in Grade 2 drop-out? 0. No 1. Yes *Select only one option dropout_g3_t_20 Did the teacher in Grade 3 drop-out? 0. No 1. Yes *Select only one option H8 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions dropout_g4_t_20 Did the teacher in Grade 4 drop-out? 0. No 1. Yes *Select only one option dropout_g5_t_20 Did the teacher in Grade 5 drop-out? 0. No 1. Yes *Select only one option Attendance Information Attendance Please ask the Principal first if they have the attendance if not talk to the teacher in each grade attend1b How many boys in 1st grade attended school last Thursday? *Use attendance lists to populate attend1g How many girls in 1st grade attended school last Thursday? *Use attendance lists to populate attend2b How many boys in 2nd grade attended school last Thursday? *Use attendance lists to populate attend2g How many girls in 2nd grade attended school last Thursday? *Use attendance lists to populate attend3b How many boys in 3rd grade attended school last Thursday? *Use attendance lists to populate attend3g How many girls in 3rd grade attended school last Thursday? *Use attendance lists to populate attend4b How many boys in 4th grade attended school last Thursday? *Use attendance lists to populate attend4g How many girls in 4th grade attended school last Thursday? *Use attendance lists to populate attend5b How many boys in 5th grade attended school last Thursday? *Use attendance lists to populate attend5g How many girls in 5th grade attended school last Thursday? *Use attendance lists to populate H9 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions attend6b How many boys in 6th grade attended school last Thursday? *Use attendance lists to populate attend6g How many girls in 6th grade attended school last Thursday? *Use attendance lists to populate Note Insert your comment if any, especially if the attendance list is not available Additional Interventions intervention1 Are there other similar (education, health, or nutrition) programs operating in this school? 0. No Enrollment Yes *Select only one option Intervention3 Please list the names of the other interventions and the year the program started in this school Name1_________ Year1________ Name2_________ Year2________ Name3_________ Year3________ Name4_________ Year4________ Name5_________ Year5________ Name6_________ Year6________ Name7_________ Year7________ *Add responses based on intervention2 H10 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions Note Insert your note if there are more programs or there are any other important information regarding these programs OBSERVATIONS structure Is this school a permanent or temporary structure? 1. Permanent 2. Temporary *Select only one option type What type of structure is the school? 1. Concrete/block 2. Mud 3. Mat 4. Open air 5. Other (specify) *Select only one option clean1 Are the school grounds free from standing water? 0. No 1. Yes *Select only one option Clean2 Are the school grounds free from trash and feces? 0. No 1. Yes *Select only one option Clean3 Are the school grounds free from sharp objects? 0. No 1. Yes *Select only one option Clean4 Is the grass within the school grounds kept short? 0. No 1. Yes *Select only one option expansion1 Has there been any expansion of existing buildings in the past year (since February, 2021)? 0. No expansion2 1. Yes *Select only one option * Confirm the expansion with the principal expansion1b Is this school a permanent or temporary structure? 1. Permanent 2. Temporary *Select only one option H11 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions expansion1c What type of structure is the school? 1. Concrete/block 2. Mud 3. Mat 4. Open air 5. Other (specify) *Select only one option Expansion2 Has there been any addition of new buildings in the past year (since February, 2021) 0. No toilet1 1. Yes *Select only one option *Confirm the addition with the principal expansion2b Is this school a permanent or temporary structure? 1. Permanent 2. Temporary *Select only one option expansion2c What type of structure is the school? 1. Concrete/block 2. Mud 3. Mat 4. Open air 5. Other (specify) *Select only one option Sanitation Information toilet1 How many toilets are there? toilet2-toilet5 is repeated for each toilet in school (up to 6 toilets) …… * Record the number of toilets between 0 and 10 *If there are more than 10 toilets in the school just make a note at the end in the comment box toilet2-toilet5 has to be repeated for each toilet in school (up to 6 toilets) toilet2 Who is this toilet for? 1. Girls 2. Boys 3. Not designated by gender *Select only one option H12 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions toilet3 What type of toilet is this? 1. Flush or pour-flush 2. Pit latrine with slab 3. Composting toilet 4. Pit latrine (without slab) 5. Hanging latrine 6. Bucket latrine 7. Other (Describe…) *Select only one option toilet6 How would you rate the cleanliness of the inside of the latrine? 1. Very clean 2. Clean 3. Dirty 4. Very Dirty *Select only one option *Very clean=Completely free from feces outside the pit, completely free from used paper outside the pit/bin, recently washed Clean= Mostly free from feces outside the pit, mostly free from used paper outside the pit/bin, recently washed Dirty=Some feces outside the pit, some used paper outside the pit/bin, not recently washed Very dirty=Much feces outside the pit, much used paper outside the pit/bin, not recently washed" toilet4 Is the toilet accessible? 0. No 1. Yes *Select only one option *Doors are unlocked or key is available H13 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions toilet5 Is the toilet private? 0. No 1. Yes *Select only one option *Walls that protect the user from view - - may be a sheet of plastic in the form of an L that allows someone to walk in and not be seen toilet 7 Are there locks to close the toilets from the inside when in use and outside when not in use? 0. No 1. Yes *Select only one option *The locks can be a rope or a metal lock that does not allow anyone to walk in. Note Insert your comment if any, especially if the number of toilets are more than 10 Hygiene Information washstation1 Is there a handwashing station available near the toilets? (see the picture above for an example of what a handwashing station looks like) 0. No --> go to washstation4 1. Yes *Select only one option H14 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions washstation1_b Does this handwashing station have water to wash hands? 0. No 1. Yes *Select only one option washstation2 Is this wash station within 10 paces of a toilet? 0. No 1. Yes *Select only one option washstation3 Is there soap at this wash station? 0. No 1. Yes *Select only one option *only soap is yes -- ash or mud is no Washstation4 Is there a wash station before entering the school? 0. No water1 1. Yes *Select only one option Washstation5 Does this handwashing station have water to wash hands? 0. No 1. Yes *Select only one option Washstation6 Is there soap at this wash station 0. No 1. Yes *Select only one option *only soap is yes -- ash or mud is no water1 Is there water available for drinking? 0. No 1. Yes, but not treated (untreated surface water, tanker trucks) 2. Yes, treated water (bottled, chlorine, boiled, Water Guard) *Select only one option COVID safety protocols Covid1 When you are at the school are the following groups wearing mask? 1.Teachers 2.Other school personnel such as principal 3. Students *Add yes or no for each H15 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions Covid2 Are classrooms arranged with one￾meter distance in between desks? 0. No 1. Yes *Select only one option *Just do a spot check in a couple of classrooms and make a note in the comment box if not all of them doing it Covid3 How many thermometers are available in the school? …… *enter a number – add 0 if none *check this question with the principal Covid4 Are any of the following items available at the school? 1. Extra masks for students or staff in case they forget to bring theirs 2. A back sprayer 3. Cleaning supplies such as a bucket, towel, and floor mop 4. Reusable gloves 5. Rubber boots goggles Covid5 Does the school have a cleaning staff? 0. No 1. Yes *Select only one option Notes Assessor comments Canteen Information canteen1 Is there a place for food preparation at this school? 0. No --> go to library 1. Yes *Select only one option canteen1_b Is the canteen functional? 0. No 1. Yes *Select only one option Canteen1_c Is the canteen clean and/or disinfected? 0. No 1. Yes *Select only one option H16 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions canteen2 Do you see the following related to food preparation? 1. Food storeroom with lock 2. Food on pallets 3. Food securely closed in bags 4. Place for cook to wash hands 5. Place for cook to wash vegetables 6. Cooked food protected from flies 7. Leftover food stored at school *Check all that apply canteen3 How many cooking stations are open fire? 0. 0 1. 1 2. 2 3. 3 4. more than 3 *Select only one option canteen4 How many cooking stations are energy saving stoves? 0. 0 1. 1 2. 2 3. 3 4. more than 3 *Select only one option canteen5 Does the kitchen have a table for the stocking of clean dishes, spoons, and cooking utensils? 0. No 1. Yes *Select only one option Canteen6 Do students share the same cups and utensils for eating and drinking without adequate washing? 0. No 1. Yes *Select only one option *ONLY APPLICABLE IF THE OBSERVATION IS BEING CONDUCTED DURING LUNCH TIME Notes Assessor comments Learning material in class H17 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions For the following observations, ask permission from the principal to visit the classroom in Grade 1 and 2 and politely explain to the teachers that you want to check the availability of the learning materials in their classroom lets_read_g1 Are there Grade 1 “Let’s Read” books available in the classroom? 0. No 1. Yes *They should be located in plastic trunks/containers *Remember the Let’s read book is blue for Grade 1 and red for Grade 2 lets_read_g2 Are there Grade 2 “Let’s Read” books available in the classroom? 0. No 1. Yes *They should be located in plastic trunks/containers *Remember the Let’s read book is blue for Grade 1 and red for Grade 2 activity_book_g1 Is there a Grade 1 student activity book for each student in the classroom? 0. No 1. Yes *Select only one option *Remember the student activity book is blue for Grade 1 and red for Grade 2 H18 | AIR.ORG LEARN Endline Evaluation Report Variable Item Response Options Instructions activity_book_g2 Is there a Grade 2 student activity book for each student in the classroom? 0. No 1. Yes *Select only one option *Remember the student activity book is blue for Grade 1 and red for Grade 2 instruct_guide_g1 Is there a Grade 1 teacher instructional guide in the classroom? 0. No 1. Yes *Select only one option *Remember the instructional book is blue for Grade 1 and red for Grade 2 instruct_guide_g2 Is there a Grade 2 teacher instructional guide in the classroom? 0. No 1. Yes *Select only one option *Remember the instructional book is blue for Grade 1 and red for Grade 2 library_note Insert any comments about the library, if any Notes Assessor comments I1 | AIR.ORG LEARN Final Evaluation Annex I. Qualitative Protocols Detailed Overview of Topics Covered in Qualitative Protocols Students Topics Types of Questions Background information  Questions about student’s background Access to and value of education  Access to education in the community; specific barriers to access and full engagement (who is excluded)  Gender-equity of access  Parental involvement, etc.  How COVID-related closures may have affected access for some during re-openings School feeding / nutrition  Perceived effectiveness of feeding program; successes and areas for improvement School health clubs / WASH / nutrition  Perceived effectiveness of SHN champions and school health clubs on improving nutrition and WASH practices in schools  WASH status in school School literacy environment  How much students are exposed to literacy activities within the school environment (e.g., presence of library, teacher reading exercises)  Resources and encouragement provided to students to read outside of school (e.g., can take home library books, working with parents/PTAs to encourage reading at home) Home / community literacy environment / reading clubs  How much students are exposed to literacy activities within the home (e.g., presence of books or other reading materials) reading materials)  Whether literacy is valued in the home (e.g., if reading and doing homework is encouraged) – Existence of / quality of community-based reading activities and resources (e.g., book banks, reading clubs, reading festivals (not yet started)); ease of accessibility to materials within – Degree to which students actively pursue/take part in home/community reading activities  Literacy-related resources and support during COVID and how this may have affected present situation I2 | AIR.ORG LEARN Final Evaluation School-related gender-based violence  While the team will not ask any direct questions related students’ exposure to sexual and gender-based violence, related questions are included in order to capture more general information around the extent to which students know about whether they are protected in the school by (a) a code of conduct that restricts such behaviors and (b) an effective referral and reporting mechanisms to report such behaviors if they do occur.  Students will also be asked more generally about what they dislike and like about their school and teachers, (probing on issues specifically around school climate and safety)  New challenges and concerns since reopening / given situation during closures Teachers and principals (interviewed separately, but same questions; (include SHN Champions – principal and science teacher)) Topics Types of Questions Background information  Teachers’ tenure at the school Access to and value of education  Access to education in the community  Gender-equity of access  Parental involvement, etc.  How COVID-related closures may have affected access for some during re-openings School feeding / nutrition  Existence of and quality of kitchen, gardens  Perceived effectiveness of feeding program; successes and areas for improvement School health clubs / WASH / nutrition  Perceived effectiveness of SHN champions and school health clubs on improving nutrition and WASH practices in schools  Perception on WASH grants scheme  Adequacy of SHN training  Effectiveness of annual de-worming campaign School literacy environment  How much students are exposed to literacy activities within the school environment (e.g., presence of library, teacher reading exercises)  Resources and encouragement provided to students to read outside of school (e.g., can take home library books, working with parents/PTAs to encourage reading at home)  Literacy-related resources and support during COVID and how this may have affected present situation I3 | AIR.ORG LEARN Final Evaluation Reading Clubs (Literacy Champions Only)  Reflections on the efficacy of the school year and summer reading clubs  Difference between in school / out of school uptake in Summer Reading Clubs  Feasibility of teachers with added load as Literacy Champions; related events  Adequacy of training received to be Literacy Champion School-related gender-based violence  Perceived prevalence of SRGBV behaviors in the school (itemized by type of behavior e.g., bullying vs. Corporal punishment vs. Sexual abuse vs. Sexual harassment vs. Physical abuse)  Positive discipline strategies (as alternative to corporal punishment) in place, and their effectiveness or limitations  Existence of / effectiveness of school code of conduct to reduce SRGBV  Existence of / effectiveness of reporting mechanisms for students/teachers to use to report violations of school code of conduct  New challenges and concerns since reopening / given situation during closures Parent-Teacher Associations  Existence and activities of PTAs; specific successes and specific areas for improvement to enhance collaboration and effectiveness.  Degree to which teachers / principals collaborate with PTA member parents  Knowledge about future role of PTAs for engaging with parents  Perception on WASH grants scheme  Role during COVID-related closures and what has changed since reopening Parents (mixed gender; members of PTA to be included) Topics Types of Questions Background information  Questions about children’s background Access to and value of education  Access to education in the community; specific barriers to access and full engagement (who is excluded)  Gender-equity of access  Parental involvement, etc.  How COVID-related closures may have affected access for some during re-openings School feeding / nutrition  Perceived effectiveness of feeding program; successes and areas for improvement Home / community literacy environment / reading clubs  How much students are exposed to literacy activities within the home (e.g., presence of books or other reading materials) reading materials)  Whether literacy is valued in the home (e.g., if reading and doing homework is encouraged) I4 | AIR.ORG LEARN Final Evaluation  Existence of / quality of community-based reading activities and resources (e.g., book banks, reading clubs, reading festivals (not yet started); ease of accessibility to materials within  Degree to which students actively pursue/take part in home/community reading activities  Literacy-related resources and support during COVID and how this may have affected present situation School-related gender-based violence  Knowledge of SRGBV behaviors (sexual and physical violence and harassment; bullying; corporal punishment) and agreement that they are negative behaviors  While the team will not ask any direct questions related students’ exposure to sexual and gender-based violence, related questions are included in order to capture more general information around the extent to which parents know about whether their children are protected in the school by (a) a code of conduct that restricts such behaviors and (b) an effective referral and reporting mechanisms to report such behaviors if they do occur.  Parents will also be asked more generally about what their children dislike about their school and teachers, and what parents like and dislike about the school and teachers (probing on issues specifically around school climate and safety)  New challenges and concerns since reopening / given situation during closures Parent-Teacher Associations  Existence and activities of PTAs; specific successes and specific areas for improvement to enhance collaboration and effectiveness.  Degree to which teachers / principals collaborate with PTA member parents  Knowledge about future role of PTAs for engaging with parents  Perception on WASH grants scheme  Role during COVID-related closures and what has changed since reopening Government – County/District Education Officers, national-level ministries (Ministry of Education, Ministry of Health, and Ministry of Agriculture) Topics Types of Questions Background information  Role in government; relationship with project School feeding / nutrition  Status of training of MOE school feeding division officials  Input on progress related to MOE’s desire to move to a Home-Grown School Feeding (HGSF) approach to school meals under the National School Feeding Policy (NSFP)  knowledge of and agreement to ground rules on gardening activities; challenges to date School health clubs / WASH / nutrition  Opinions on the school health and nutrition manual used.  Progress on Save collaboration with CEOs and DEOs to provide training to the SHN Champions  Effectiveness of de-worming campaign School literacy environment  Feasibility of teachers with added load as Literacy Champions; related events I5 | AIR.ORG LEARN Final Evaluation School-related gender-based violence  Development and revision of MOE Code of Conduct; mechanisms for roll-out and successes / challenges  Work on supporting reporting mechanisms at school and district level; procedures for responding to reports against teachers and other staff Parent-Teacher Associations  Work supporting PTAs, particularly the emphasis on PTAs supporting literacy Community Members KII - storekeepers Topics Types of Questions Background information  Role in school/community; relationship with project School feeding / nutrition  Existence of and quality of kitchen  Perceived effectiveness of feeding program; successes and areas for improvement  Specific challenges managing storeroom and food management practices  Adequacy of in-kind payments in form of take-home rations  Effectiveness of take-home rations for students and volunteers (in summer) Community Members -KII cooks Topics Types of Questions Background information  Role in school/community; relationship with project School feeding / nutrition  Existence of and quality of kitchen  Perceived effectiveness of feeding program; successes and areas for improvement  Specific challenges providing meals  Adequacy of in-kind payments in form of take-home rations  Effectiveness of take-home rations for students and volunteers (in summer) Community Members KII - and non-teacher Literacy Champions (community volunteers) Topics Types of Questions Background information  Role in school/community; relationship with project School literacy environment  How much students are exposed to literacy activities within the school environment (e.g., presence of library, teacher reading exercises) I6 | AIR.ORG LEARN Final Evaluation  Resources and encouragement provided to students to read outside of school (e.g., can take home library books, working with parents/PTAs to encourage reading at home) Home / community literacy environment / reading clubs  Existence of / quality of community-based reading activities and resources (e.g., book banks, reading clubs, reading festivals (not yet started)); ease of accessibility to materials within  Degree to which students actively pursue/take part in home/community reading activities  Reflections on the efficacy of the school year and summer reading clubs  Difference between in school / out of school uptake in Summer Reading Clubs  Adequacy of training received to be Literacy Champion Project staff including community mobilizers Topics Types of Questions Update since closures  Detailed lines of inquiry to learn more about what specifically happened with regards to the work with / by government, roles of Literacy Champions, SHN champions, PTA, storekeepers, and cooks Background information  Role in school/community; relationship with project General  Perceived challenges and successes with particular role in project (customized to informant) Community mobilization  Quality of training received from Save  Approximate reach thus far; sensitization activities delivered (e.g., which trainings and to whom)  Challenges in training cooks, storekeepers  Experiences working with parents  Knowledge of / utility of SC community sustainability guide Discussion Guides Note that these discussion guides are written in Standard English, and the qualitative team is experienced in interpreting questions written in Standard English into Liberian English during interviews, or to otherwise rephrase the wording of the questions so as to help the participants understand the question being asked. For clarity, the discussion guides below will remain in Standard English and be used as-is for training the qualitative team, so they fully understand the content that we wish to obtain during interviews. FGDs will begin with an age-appropriate ice-breaker. I7 | AIR.ORG LEARN Final Evaluation Questions in green text are for intervention sites only; questions in black text will be asked of both comparison and intervention sites (with slight modification in how the question is asked to be specific to the group, as necessary). Table I1. FGD Students (include students who are involved in Reading Clubs and FMCs) # Activity Topic Specific Topic Discussion Question and Probes 0 Background Background information [Age, gender, current year in school, size of household] 1 LB Access to and Value of education Do you think children need to go to school? For how long? Are there any differences between boys and girls? [short icebreaker question] 2 LB Access to and Value of education We know that during the COVID school closures, a lot had changed in terms of being able to access learning. Now schools are reopened, and we want to talk about any new challenges that you see or have experienced in children being able to attend school on a regular basis. What are some of these challenges you see? Are these different from challenges before COVID, or the same? Are there some children who attend school more than others? What prevents some children in this community from going to school? Are there different reasons that prevent boys and girls from going to school? What usually happens when children are unable to attend regularly? 3 SF School feeding / nutrition We understand there had been an initiative in your school before the COVID closures that provided students with hot meals. We know this was interrupted during closures but is beginning again. Please describe what you think the effect of having these school meals has been for you personally both before the closures and after the closures. What about for the school as a whole? (for FMCs only: What is your role on the food management committee? What are some challenges you face in your work? What would help?) 4 SHN School health clubs / WASH / nutrition Are you aware of the school health clubs and SHN champions in the school? If so, what sorts of activities do you see them engaged in? Is it effective and helpful? If so, in what ways; if not, why not? Have they been involved in any de-worming activities, to your knowledge? If so, please describe what this looks like? 5 SHN School health clubs / WASH / nutrition What is the status of Water, Sanitation and Health (WASH) in this school now that you are back in school after the interruption? How does it compare to the status of WASH prior to the interruption? What further improvements are needed now and how is the school dealing with them? 6 LB School literacy environment Now that you are back in school, what activities does your teacher in class or the school as a whole do with you to help you learn to read? Which are your favorite activities and are they helping you to learn to read? Which are the activities that you do not enjoy or do not find useful? 7 LB Home / community literacy I want to talk about whether you are encouraged to read outside of school / at home? First let’s talk about before school was closed, did anyone encourage you to read outside of school? Who? I8 | AIR.ORG LEARN Final Evaluation environment / reading clubs Now I want to talk about during the school closures, what sort of encouragement to read did you receive? From who/ where (e.g., radio, pamphlets, adults in the community)? Did you appreciate the encouragement? Do you think it helped you to learn how to read better? Finally, I want to know about today, are you still getting this encouragement? From whom? Is it different now from before the closures? Thinking about today, in your community, do you think that reading with children is encouraged, or simply tolerated but not considered to be important? Is this a change from before COVID? 8 LB Home / community literacy environment / reading clubs Do you have any reading resources that you can use at home? Like what? Are there any sorts of book banks in the school or in the community that allow you to read anything outside of school hours? [If yes] how often do you read these materials outside of school? What is the process/rules for borrowing a book?; [if no] if these materials were available, do you think you would use them? 9 LB School-related gender-based violence Are you aware of a school code of conduct? What sorts of behaviors, according to this code of conduct, are not allowed? [only probe to clarify what is said, do not introduce behaviors to the group if they are not offered] Show cards with images and words depicting the types of behaviors NOT allowed. Let the participant select the card and ask them why they chose it. If they didn’t select a particular card, ask them why. Now, thinking about the lockdown time, how do you think we may need to reconsider the codes of conduct now that we’re back in school. 10 LB School-related gender-based violence In case a teacher or school administrator does something against this code of conduct, what should people do? Do people report, to whom? Can you give me an example of what might happen if a student reports that a teacher or school administrator was abusive or violent? What would happen? Would you feel comfortable reporting a teacher or school administrator if they did something wrong? Describe what it would be like if you reported a teacher or school administrator for being abusive or violent. [if a specific incident / behavior is mentioned in previous discussion question, refer to it again here to help students understand the question] 11 LB School-related gender-based violence What is your favorite thing about being at school? How do the teachers and school administrators treat the students here? Are you happy at school? Do you feel safe at school? Tell me about a time when you felt safe at school? What are some of the things that make you not feel safe and happy? Are your feelings about being at school different now since the closures are over, as compared to before the closures? In what way? I9 | AIR.ORG LEARN Final Evaluation Table I2. FGD Parents / Guardians (include PTA members and FMC members) # Activity Topic Specific Topic Discussion Question and Probes 0 Background Background information [Age, gender, # children and years in school, marital status, size of household] 1 LB Access to and Value of education Do you think children need to go to school? For how long? Are there any differences between boys and girls? [short icebreaker question] 2 LB Access to and Value of education We know that during the COVID school closures, a lot had changed in terms of being able to access learning. Now schools are reopened, and we want to talk about any new challenges that you see or have experienced in children being able to attend school on a regular basis. What are some of these challenges you see? Are these different from challenges before COVID, or the same? Are there some children who attend school more than others? Are there different reasons that prevent boys and girls from going to school? What usually happens when children are unable to attend regularly? 3 SF School feeding / nutrition We understand there had been an initiative in your child’s school before the COVID closures that provided students with hot meals. We know this was interrupted during closures but is beginning again. Please describe what you think the effect of having these school meals has been for you and your child personally both before the closures and after the closures. What about for the school as a whole? (for FMCs only: What is your role on the food management committee? What are some challenges you face in your work? What would help?) 4 SHN School health clubs / WASH / nutrition [PTA only] What is your impression of the grants scheme to improve WASH in your school? Has there been any work done yet to apply for such a grant? 5 LB Home / community literacy environment / reading clubs I want to talk about whether you encourage your child to read outside of school / at home? First let’s talk about before school was closed, did you encourage your child to read outside of school, or did you read to them, even if not specifically for schoolwork? Now I want to talk about during the school closures, what if anything did you do with your child? How did you help or encourage them? Do you think it helped you them to learn how to read better? Were there other resources provided to help your child to read? From who/ where (e.g., radio, pamphlets, adults in the community)? Finally, I want to know about today, will you be providing any encouragement or reading to your children? Is it different now from before the closures? Thinking about today, what do you think is typical in your community as it relates to parent involvement in children's reading. Is reading with children encouraged, or simply tolerated but not considered to be important? I10 | AIR.ORG LEARN Final Evaluation 6 LB Home / community literacy environment / reading clubs Not including resources in the school, are there any sorts of book banks in the school or in the community that allow your children to read anything outside of school hours? [If yes] how often do your children read these materials outside of school? What is the process/rules for borrowing a book?; [if no] if these materials were available, do you think they would use them outside of school? 7 LB School-related gender-based violence Are you aware of a code of conduct for teachers and school administrators at your child’s school? What sorts of behaviors, according to this code of conduct, are not allowed? [only probe to clarify what is said, do not introduce behaviors to the group if they are not offered]. Show cards with images and words depicting the types of behaviors NOT allowed. Let the participant select the card and ask them why they chose it. If they didn’t select a particular card ask them why. 8 LB School-related gender-based violence If a teacher or a school administrator is abusive or acts violently towards your child, what would you do? Do you have any rights as a parent? Could you report the incident? Describe what it would be like if you reported a teacher or school administrator for being abusive or violent [if a specific incident / behavior is mentioned in previous discussion question, refer to it again here to help parent understand the question 9 LB School-related gender-based violence Do you think your child feels happy at school? Do you think your child feels safe at their school? In general, do you see your child’s school as a positive, safe place for them to be? Thinking about the people there and the resources they have there, what are some of the areas that need to be improved at their school, especially related to your child’s safety and well-being? Are your feelings about your children being at school different now since the closures are over, as compared to before the closures? In what way? 10 SF Parent-Teacher Associations Do you engage with your child(ren)’s teacher about their education? If so, what is the nature of that engagement? Do you find it useful? If not, what might make such engagement easier? 11 SF Parent-Teacher Associations Please describe this school’s parent-teacher association. When was it established, who is involved (how many members), what sorts of activities do they work on? [probe: codes of conduct, meal provision, WASH infrastructure management; school meals). What are the PTAs’ main goals and mandate? What has been the role of mobilizers in establishing and strengthening PTAs? What are some of the major challenges the PTA faces in achieving its mandate? 12 SF Parent-Teacher Associations What sorts of messages have you heard relating to the PTAs role in engaging with parents on issues related to the children’s education? Do you think such engagement strategies will be effective? I11 | AIR.ORG LEARN Final Evaluation Table I3. FGD Teachers/KII Principals (include SHN Champions – principal and science teacher; FMC Member – Vice Principal) # Activity Topic Specific Topic Discussion Question and Probes 0 Background Background information [Age, gender, # years teaching in this school, # years teaching overall, certifications obtained] 1 LB Access to and Value of education Do you think children need to go to school? For how long? Are there any differences between boys and girls? [short icebreaker question] 2 LB Access to and Value of education We know that during the COVID school closures, a lot had changed in terms of being able to access learning. Now schools are reopened, and we want to talk about any new challenges that you see or have experienced in children being able to attend school on a regular basis. What are some of these challenges you see? Are these different from challenges before COVID, or the same? Are there some children who attend school more than others? What prevents some children in this community from going to school? Are there different reasons that prevent boys and girls from going to school? What usually happens when children are unable to attend regularly? 3 LB School literacy environment What activities do you do in class or the does the school as a whole do with learners to help them learn to read? Which are their favorite activities and are they helping you to learn to read? Which are the activities that they do not enjoy or do not find useful? Are the activities that you do now, after the closures, going to be any different from those you did before the closures? If so, please explain the differences and why you are approaching activities differently. 4 LB School literacy environment Thinking about before closures, during closures, and today now that schools are reopening, how have students been encouraged to read outside of school / at home? First let’s talk about before closures – was there encouragement? By whom? What happened during closures? Was there any support? From who/ where (e.g., radio, pamphlets, adults in the community)? Will any encouragement that took place during the closures endure? Why / why not? Do you think that students have appreciated the encouragement? In your community, do you think that reading with children is encouraged, or simply tolerated but not considered to be important? 5 Do students have any reading resources that they can use at home? Like what? Are there any sorts of book banks in the school or in the community that allow children to read anything outside of school hours? [If yes] Do they access them? What is the process/rules for borrowing a book?; [if no] if these materials were available, do you think they would use them outside of school? 6 LB School literacy environment How do you feel about your present workload as a teacher? Do you have sufficient time to complete all your tasks? What extra tasks have you been given in the last one year or so? Could you take on more? What would you need to be able to take on more responsibilities? How have the school closures affected your workload today? 7 LB Home / community literacy environment / reading clubs (Literacy Champions Only) Do you feel as though you have been adequately trained to be a Literacy Champion, leading these clubs? Please explain what training you have gotten, and what more you feel you would need to be more effective in your role? Please reflect specifically on the work you did during the school closures I12 | AIR.ORG LEARN Final Evaluation 8 SF Parent-Teacher Associations Do parents engage with you directly about their child(ren)’s education, s? Do all, most, or few parents engage with you? What is the nature of their engagement and do they/you find it useful? What type of engagement do you feel is most successful in supporting children? Are there certain types of parents who tend/tend not to engage with you? What might make engagement easier and more productive for both you and parents? Did parents attempt to engage during closures? Has this engagement changed as compared to their engagement before the closures? 9 SF Parent-Teacher Associations Please describe this school's parent-teacher association [probe: codes of conduct, meal provision, WASH infrastructure management; school meals). What is your involvement with the PTA? What was your role on the PTA during the school closures? What is it now and is this different from before the closures? What has been the role of mobilizers in establishing and strengthening PTAs? 10 SF Parent-Teacher Associations What sorts of messages did you hear during closures relating to the PTAs role in engaging with parents on issues related to the children’s education? Do you think such engagement strategies were effective? Have you heard of such engagement now that schools are reopened? 11 SF School feeding / nutrition We understand there had been an initiative in your school before the COVID closures that provided students with hot meals. We know this was interrupted during closures but is beginning again. Please describe what you think the effect of having these school meals has been for you personally both before the closures and after the closures. What about for the school as a whole? (for FMCs only: What is your role on the food management committee? What are some challenges you face in your work? What would help?) 12 SF School feeding / nutrition Is there a kitchen at this school? When was it built/opened? Does it have adequate materials and is well￾maintained? Who are the staff working in it? Is there a school garden? 13 SHN School health clubs / WASH / nutrition Are you aware of the school health clubs and SHN champions in the school? If so, what sorts of activities do you see them engaged in, both during closures and today? Is it effective and helpful? If so, in what ways; if not, why not? Have they been involved in any de-worming activities, to your knowledge? If so, please describe what this looks like? (SHN Champion only: What is your particular role as an SHN Champion? What are some challenges you face in this role; what would help? What sort of training did you receive as an SHN Champion? Was it adequate? What topics did you learn about? Did your training include guidance on participatory or child-to-child methodologies?) 14 SHN School health clubs / WASH / nutrition What is the status of WASH in this school now that you are back in school after the interruption? How does it compare to the status of WASH prior to the interruption? What further improvements are needed now and how is the school dealing with them? [PTA only] What is your impression of the grants scheme to improve WASH in your school? Has there been any work done yet to apply for such a grant? 15 LB School-related gender￾based violence What are some of the strategies you use to discipline students in your classroom? Are they effective? Have you been trained on any alternative discipline strategies? What do other teachers use, to your knowledge? Would you benefit from training or resources to help you maintain a controlled classroom? 16 LB School-related gender￾based violence Are you aware of a school code of conduct? What sorts of behaviors, according to this code of conduct, are not allowed? Show cards with images and words depicting the types of behaviors NOT allowed. Let the participant select the card and ask them why they chose it. If they didn’t select a particular card, ask them why. [Only probe to clarify what is said, do not introduce behaviors to the group if they are not offered] I13 | AIR.ORG LEARN Final Evaluation 17 LB School-related gender￾based violence If one of your colleagues violates something in this code of conduct, what would you do? Can you think of any examples where you would choose to not do anything? Is there a procedure in place where you could report the incident? To your knowledge, is this an effective mechanism to report incidents? Would you trust it enough to use it in case you needed to? If not, why not? 18 LB School-related gender￾based violence What do you do to make students feel safe at school? Do you feel as though there are any new challenges that you’ll need to consider given the time students spent away from school? Table I4. Project Staff KII – General Framing Topics to probe into # Activity Topic Specific Topic Discussion Question and Probes 0 Background Background information [Age, gender, role in school/community/project, education level] LB Home / community literacy environment / reading clubs What was the role of the Literacy Champions during closures? Please provide me a detailed example of a week for one of these people. What was their responsibility? What were they aiming to accomplish? What training / support did they get? How often did they work on the various activities associated with this position? How were they monitored?  In particular, please elaborate on what specifically was their role with the Home Learning Program and the associated packets, radio, SMS LB Parent-Teacher Associations As with Literacy Champions in the previous question, I’d like to know more about the PTA members. Please provide me a detailed example of a week for one of these people. What was their responsibility? What were they aiming to accomplish? What training / support did they get? How often did they work on the various activities associated with this position? How were they monitored? In particular, please elaborate on what specifically was their role with the Home Learning Program and the associated packets, radio, SMS SHN School health clubs / WASH / nutrition Now I’d like to know more about SHN Champions. Please provide me a detailed example of a week for one of these people. What was their responsibility? What were they aiming to accomplish? What training / support did they get? How often did they work on the various activities associated with this position? How were they monitored? SHN School health clubs / WASH / nutrition Please explain how, during the COVID closures and since, engagement with government may have changed in terms of the following activities which were expected to have been ongoing before those closures:  Progress of MOE moving to the 'Home Grown School Feeding' Approach to school meals  MOE school feeding officials who were meant to be trained on the school feeding project  DEOs who were meant to be training kitchen staff  Select training for some MOE staff to know about the ground rules for gardening activities that have been established?  SHN Health Education Manual and its adaptation to the Liberian context I14 | AIR.ORG LEARN Final Evaluation # Activity Topic Specific Topic Discussion Question and Probes  Training of SHN Champions and roll-out of work with school health clubs  Revision and roll-out of school codes of conduct  Work to support advocacy and enforcement of school codes of conduct  Work with PTAs 1 Gen Gen What are the biggest challenges you have faced in your role as [specify] in this community? [start with biggest issue the respondent sees, and ask all follow up questions. Then, continue onto the next biggest issue, and ask follow up questions. Try to cover at least the three biggest issues]. Is this an issue in other communities? What are some of the ways that you’ve tried to overcome this challenge? What would help? 2 Gen Gen What are some of the biggest successes you have had in your role as [specify] in this community? [As with previous question, start with biggest success the respondent sees, and ask all follow up questions. Then, continue onto the next biggest success, and ask follow up questions. Try to cover at least the three biggest successes]. How can you further build on this success and/or maintain it? Do you think other communities have been as successful? Why/why not? 3 Gen Community mobilization Describe the training you have received from Save on performing your role. Have you had training on participatory community mobilization? Was it effective? What about training on codes of conduct? Are you knowledgeable about the SC community sustainability guide? Please explain how it is utilized at the school / community level. 4 Gen Community mobilization What are the specific sensitization activities that you personally have delivered (e.g., which trainings and to whom)? Did you feel adequately prepared to deliver these? Were those who received your trainings receptive? What would help? 5 Gen Community mobilization Have you trained storekeepers and cooks? Do you feel adequately prepared? Do you feel as though the storekeepers and cooks that you trained are ready to perform their tasks? If no, what more do they need? 6 Gen Community mobilization Have you worked directly with parents yet? Doing what? What has been your experience? I1 | AIR.ORG LEARN Final Evaluation Annex J. Endline ToR Terms of Reference Baseline, Midline, Endline Evaluation Liberia Empowerment through Attendance, Reading, and Nutrition II (LEARN II) Program Donor: United States Department of Agriculture (USDA) Start Date: October 1, 2021 End Date: September 30, 2026 I. Introduction Save the Children’s work leading the USDA Liberia Empowerment Through Attendance, Reading, and Nutrition (LEARN) program, with partner Mercy Corps, has achieved significant successes in school feeding, literacy, school health and nutrition, and child health over the past four years. Save the Children will build on that success through a five-year, $25 million second phase, LEARN II, focused on reinforcing local capacity at every level to address Government of Liberia (GoL) and USDA objectives to provide nutritious school feeding and quality literacy education in a healthy, supportive environment. LEARN II will lead 265,830 direct beneficiaries along a sustainable path toward graduation, targeting 85,129 pre-primary and primary school children in 234 schools, along with educators, Parent-Teacher Associations (PTAs), government, the private sector, and communities. LEARN II will build on LEARN’s one-team approach, with Save the Children leading implementation in Grand Gedeh and River Gee and Mercy Corps leading implementation in Grand Bassa and River Cess, and technical support coming from both organizations. Kawadah Farms will serve as a partner on a new local and regional procurement activity in which women￾led farming cooperatives will be trained in improved techniques for cassava production and linked to local partner Kawadah Farms, that will purchase, process, and fortify their products into cassava-based Power Gari for preparing nutritious porridge. LEARN II schools in the four target counties are grouped into six cohorts according to both their starting point from LEARN participation and their programming pathway envisaged under LEARN II, to build capacity at all 234 target schools. As in the LEARN project, LEARN II will implement a different package of activities depending on the target county. See Annex A and Annex B for further details. I2 | AIR.ORG LEARN Final Evaluation Figure 1: LEARN & LEARN II Activities To achieve the program’s primary objective of carrying out school feeding to reduce hunger and improve literacy and primary education, LEARN II will implement 14 activities to achieve complementary and often mutually reinforcing results in the areas of school feeding, school health and nutrition (SHN), literacy, and capacity building (see Figure 1). Provide School Meals – Provide 5,030 MTs of U.S. commodities and 715 MTs of locally procured commodities to 234 schools in 4 counties. Mentor Food Management Committees (FMCs) and train county and district education officers (CEOs and DEOs). Strengthen Local Provision of Food for School Meals – Develop a new cassava processing and fortification facility in River Gee through public-private partnership, strengthening women’s cooperatives for sustainable supply and schools’ capacities to finance demand. Establish School Gardens – Train teachers and Parent-Teacher Associations (PTAs) to grow nutritious local produce in 95 school gardens, to augment U.S. donated and local and regional procurement (LRP) commodities as part of Ministry of Education (MoE) and Ministry of Agriculture (MoA) Home-Grown School Feeding (HGSF) priority. Training: Food Preparation and Storage Practices – Train, mentor, and encourage cooks to use safe food storage and preparation techniques. Building/Rehabilitation: Kitchens and Storerooms – Ensure adequate kitchens and storerooms are constructed to include 13 additional schools and encourage all school PTAs to make infrastructure repairs and upgrades to ensure minimum quality standards. Training: Good Health and Nutrition Practices – Support rollout of the MoE’s National Training Manual for School Health with a Training of Trainers (ToT) supporting School Health Club formation to 168 schools not trained in LEARN, also developing COVID-19 outreach materials and celebrating international days to promote healthy practices. Distribution: De-worming Medication, Vitamins, and Minerals – Facilitate coordination, mobilization, and data collection for the government’s annual de-worming campaign. Building/Rehabilitation: Latrines, Wells, and Water Stations/Systems – Construct latrines and water stations at 13 newly targeted schools. Support all PTAs and communities to I3 | AIR.ORG LEARN Final Evaluation make minor repairs on water, sanitation, and hygiene (WASH) infrastructure, ensuring access for children with disabilities and separate latrines for female and male students. Establish Activities to Promote Literacy – Continue USAID Read Liberia program in 57 LEARN schools and roll out to the remaining 174 target schools, including providing Read Liberia materials; complementary in-class, extracurricular, family, and radio-based literacy support through PTA-led reading clubs and events and supplies for the most vulnerable. Training: Teachers – Facilitate master trainers to train grade 1 and 2 teachers from 174 LEARN II schools and refresh teachers from 57 target schools in the Read Liberia approach for teaching reading effectively (including to students with disabilities), as well as modules supporting MoE policies such as the Teacher Code of Conduct and Child￾Centered Positive Pedagogy including child safeguarding, child rights, gender equality, inclusive education, positive discipline, and classroom management. Training: School Administrators – Train school principals in the same content as teachers, as well as in coaching and leadership to build a positive school culture. Training: Parent-Teacher Associations (PTAs) – Train PTAs on their roles under MoE policy, emphasizing teacher accountability, literacy, and action and resource planning. Raising Awareness on the Importance of Education – Participate in finalization of MoE policies and promotion through a radio-based social and behavior change (SBC) strategy. Capacity Building: Local, Regional, and National Levels – Facilitate MoE master trainers to train teacher trainers at the government’s Webbo Teacher Training Institute (TTI) for the Southeast and local MoE staff in the USAID Read Liberia approach, distance learning, student-centered teaching, and MoE policies. Build qualifications and capacity of female teachers and administrators through scholarships and practical experiences. I4 | AIR.ORG LEARN Final Evaluation II. Scope of Work for Evaluator Save the Children is seeking a consultant or research consulting firm to lead its external evaluation process from baseline to endline. Given the follow-on nature of LEARN II, the endline for LEARN will be conducted at the same time and by the same evaluator as the baseline for LEARN II. The midterm and final evaluation contracts will be dependent on satisfactory completion of the baseline. The midterm and final evaluations will be re-competed if the baseline does not meet quality standards. The methodology and sampling detailed below may require revision based on the results of the baseline and suggestions from the consulting entity. Evaluation Approach and Methodologies LEARN II will be evaluated through performance evaluation and an impact evaluation. For LEARN II’s performance evaluation baseline study in Year 1, the evaluation team will use a mixed-methods approach and will replicate the approach for the midterm and final evaluations in Year 3 and Year 5, respectively. The evaluation team will use quantitative and qualitative methods with repeated cross-sections of project stakeholders to establish baseline values and track progress for targeted performance indicators throughout the life of the project. Additionally, a subset of schools in the four counties will serve as the site for a quasi￾experimental impact evaluation to assess the impact of LRP and school gardens on children’s literacy, health, and nutrition outcomes; and cost-effectiveness, timeliness, and perceived cultural appropriateness of the food assistance. The impact evaluation baseline and final study will take place alongside the performance evaluation studies. Tools The evaluation will include the following quantitative data collection tools to establish baseline indicator values and measure outcomes at the midterm and final evaluation stages:  Literacy Boost Reading Assessment of students in grade 2 in all counties. The evaluation team will administer a one-on-one oral emergent literacy assessment to a sample of children. The assessment will assess students on five competencies: letter awareness, single word recognition, reading fluency and accuracy, and a set of comprehension questions linked to the passage. The team will use the same assessment used in LEARN. This tool will capture information about students’ home and school learning environment.  School-Related Gender Based Violence (SRGBV) questionnaire administered to children to assess their knowledge of SRGBV behaviors, perceived gender norms and safety in school, awareness of the teachers’ code of conduct, and reporting mechanisms.  A health and nutrition assessment of students in grades 2 and 6 in all counties. The project will assess all sampled students on height, weight, age, individual food intake I5 | AIR.ORG LEARN Final Evaluation recall, and incidence of diarrheal disease recall. A questionnaire to assess perceived cultural appropriateness of meals will also be administered to all sampled students.  A health and nutrition knowledge, attitudes, and practices (KAP) assessment of a sample of grade 2 and 6 students, teachers, and school meal providers in all four counties.  A school meal provider survey in which cost data will be collected retrospectively following an ingredients approach using a semi-structured questionnaire. The project will base the survey on a standardized costing framework capturing capital (fixed) and recurrent costs incurred at the provider level. The questionnaire will cover cash and in￾kind contributions to estimate financial costs. Financial costs capture actual expenditures for project implementation on an annual basis.  A school observation checklist, including WASH resources, safe food preparation, and classroom learning environment observation assessment. In addition to quantitative data collection, LEARN II will conduct focus group discussions with children, parents, teachers, cooks, and kitchen storekeepers as well as key informant interviews with County Education Officers (CEOs), District Education Officers (DEOs), principals, PTA members, cassava cooperative members, INGOs, partner organizations and county government officials to gain a better understanding of where the greatest health, literacy, and local producer needs remain in LEARN II communities. The evaluation team will submit the evaluation protocol (including all tools) to the University of Liberia Pacific Institute for Research and Evaluation Institutional Review Board to ensure that the proposed evaluation complies with the local research ethics standards. In addition, approval will be sought from Save the Children’s Ethics Review Committee that oversees all research and evaluations taking place at Save the Children and ensures that they comply with appropriate research ethics standards with a special focus on child safeguarding. The Ministry of Education Department of Planning, Research, and Development will be engaged to obtain an authorization letter to conduct data collection. Data Collection and Analysis For quantitative data collection, the evaluation team will collect the literacy, protection, health, and nutrition data on tablets using the electronic data collection software Kobo Toolbox (developed by the Harvard Humanitarian Initiative and Brigham and Women’s Hospital). Save the Children has extensive experience programming surveys in Kobo and training internal and external staff on their use. The evaluation team will clean and analyze quantitative data using Stata or similar software. The contracted external evaluator will produce summary statistics and indicator data according to a pre-specified analysis plan. For qualitative data collection, the focus group discussion and key informant interviews will be recorded on tablets. Following the I6 | AIR.ORG LEARN Final Evaluation recording, the qualitative data will be transcribed by transcribing assistants and subsequently analyzed by the evaluation team using Atlast.ti or similar software. All data collection will take place in gender-safe spaces. The data collection team will consist of equal number of men and women. The data collection will take place separately with girls, boys, women, and men. All respondents will also have the option to be assessed/interviewed by an enumerator of the same sex and the data collection will take place in a location and time that is most convenient for the respondent, taking into account the gendered preferences that respondents may have. The data collection team will also follow all COVID-19 guidelines. Data collectors will receive SC Child Safeguarding and Safe Programming policy training. Research Design and Sampling The quantitative portion of LEARN II’s performance evaluation approach allows for tracking trends over time in children’s literacy skills, health and nutrition status, and KAP about health, hygiene, and nutrition, as well as in school meal provider (i.e., cooks) KAP about meal preparation. The impact evaluation provides an estimate of the differential impact of different intervention modalities on children’s health and education outcomes. The research design and sampling for the performance evaluation and impact evaluation are explained separately below: Performance Evaluation. For the broader performance evaluation, the evaluation team will collect literacy, health and nutrition status, and KAP data from children in schools across the four counties in order to track changes in indicators over time. Save the Children will use a two￾stage clustered sampling approach to select a cross-section of grade 2 students for the literacy, health and nutrition status, and KAP assessments. For the health, hygiene, and nutrition KAP assessment, the evaluation team will also select a sample of grade 6 students. The evaluation team will randomly select a sample of project schools from each county proportional to the total number of project schools participating in LEARN II (see Table 1 below). Subsequently, the team will randomly choose 10 students each (five girls and five boys) from one randomly chosen grade 2 and grade 6 classroom each. The sample size for the literacy, health and nutrition, and KAP assessments was derived using the recommendations from the USAID EGRA35 Toolkit to confirm the sample size of 1,140 children (570 children in grades two and four each). The sample size was calculated using the following formula: 35 RTI International. 2016. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. https://pdf.usaid.gov/pdf_docs/PA00M4TN.pdf I7 | AIR.ORG LEARN Final Evaluation Where CLtvalue is the critical value corresponding to a 95% confidence level (set to 1.96), DEFT is the square root of [1 + (𝑘-1)*ICC], 𝑘 stands for the number of children in the cluster (10 per school), and ICC is the inter-class correlation (set to 0.5 based on previous EGRA studies). SD is the estimated standard deviation (set to 26 based on previous EGRA studies), and CIwidth is the width of the confidence interval (set to 10). The formula yields a desired sample size (n) of 571, which has been adjusted downward to 570 children to allow the school sample size in each county to be proportionate to the number of project schools in the counties. Table 1: Sample Sizes from Each County for the Project Evaluation County Number of Schools Number of schools selected for performance evaluation Total Grade 2 students Total Grade 6 students Grand Gedeh 56 12 120 120 River Gee 39 11 110 110 Grand Bassa 95 23 230 230 River Cess 44 11 110 110 Total 234 57 570 570 Impact Evaluation: To estimate the impact of the local procurement of commodities and school garden produce on children’s literacy, health and nutrition outcomes and cost-effectiveness, perceived cultural appropriateness, and timeliness of food assistance, LEARN II will employ a quasi-experimental impact evaluation alongside the performance evaluation. The impact evaluation will follow a quasi-experimental design and take place in 70 schools across the four counties. The evaluation team will assign 70 schools into two treatment arms:  Arm 1 (Local Procurement of Commodities and School Gardens): 35 schools that receive school meals prepared with locally procured commodities and school garden produce in addition to meals prepared with U.S. food commodities in River Gee and Grand Gedeh counties. 35 out of the 95 schools in River Gee and Grand Gedeh will be randomly selected for the impact evaluation.  Arm 2 (U.S. commodities only): A matched sample of 35 comparison schools, that do not receive local commodities (LRP or school gardens) for preparing meals, will be drawn from the remaining 2 counties (Grand Bassa and River Cess). Detailed information about the impact evaluation sample can be found in Table 2: Table 2: Impact Evaluation Treatment Groups Treatment Arm Sample Mode of Selection Counties LEARN II Intervention Intervention 35 schools Random selection within River Gee and Grand Gedeh LEARN II schools River Gee, Grand Gedeh, School Meals (USG + LRP + School Garden commodities) Comparison 35 schools Matched LEARN II schools Grand Bassa, River Cess School Meals (USG commodities only) Total 70 schools I8 | AIR.ORG LEARN Final Evaluation The evaluation team will use propensity score matching, or similar matching technique, to identify the comparison schools that most closely match the set of treatment schools. This will minimize threats to internal validity such as the presence of external factors that result in schools in River Gee and Grand Gedeh receiving local commodities to prepare school meals. The evaluation team will use a difference-in-difference approach to estimate the difference in the literacy, health, nutrition, and food assistance outcomes between the intervention and comparison groups. Data collection for the impact evaluation will coincide with the performance evaluation data collection. The evaluation team will randomly select 20 Grade 2 students (10 girls and 10 boys) from each of the 70 schools to participate in the literacy, health and nutrition, and KAP assessments. This yields a total sample size of 1,400 grade 2 students. This sample size was calculated using the user-written “clustersampsi” command in Stata 15.1, assuming a power level of 80% and an intra-cluster correlation of 0.25. The team will assess the same students in the midterm and endline evaluations. The evaluation team will administer the school meal provider KAP and cost survey to all school meal providers for the schools sampled as part of the student-level assessments. A questionnaire to assess perceived cultural appropriateness of school meals will also be administered to all sampled students for the impact evaluation. Key Informants The key informants for the evaluation are students, teachers, school meal providers (school cooks and storekeepers), PTA members, CEOs and DEOs, school principals, district/county government officials, and cassava cooperative members. Students will be assessed on their reading skills; health and nutrition status; and health, hygiene, and nutrition KAP. School meal providers’ KAP will also be captured through the baseline, midline, and endline studies. PTA members, school principals, district/county government officials, and cooperative members will participate in focus group discussions and key informant interviews to provide deep insight about the status of the school meal, learning, and school health systems. Key Audiences and Stakeholders Save the Children will consult key stakeholders in both the design and results dissemination phases of performance evaluations and the impact evaluation. Save the Children will plan the evaluation in collaboration with the project implementing and technical partner (Mercy Corps), Kawadah Farms, local and national government partners (MoE, MOH and MoA), research institutions such as local universities (e.g., Cuttington University), and USDA. Stakeholder groups to be consulted as key audiences include students, parents, teachers, PTAs, school meal providers, school administrators, government county/district officials, and community leaders. I9 | AIR.ORG LEARN Final Evaluation LEARN II will keep all stakeholders, including children and parents, informed of the evaluation findings in appropriate ways. Alignment with the McGovern-Dole Learning Agenda LEARN II’s evaluation plan and questions are closely aligned with the McGovern-Dole Learning Agenda. The evaluation results will deepen the evidence base about school meals, literacy, and health and nutrition interventions by contributing responses to the following key questions from the Learning Agenda: How do the impacts of local procurement models and other community and nationally sourced models compare with those that rely on international food sources? LEARN II will compare the cost-efficiency of school meals between schools that receive both locally procured and international food commodities and schools that only receive international food commodities to prepare school meals. The quantitative evaluation is embedded in a mixed-method evaluation that includes key informant interviews and focus-group discussions with school cooks and storekeepers. The qualitative inquiry will allow the evaluation team to understand how school cooks’ and storekeepers’ experiences vary depending on whether they work in schools that receive both locally procured and international food commodities and schools that only receive international food commodities to prepare school meals. The evaluation will also allow the evaluation team to compare operational efficiency and kitchen staff’s experiences between the two procurement models. How are nutritional outcomes affected by different food sourcing modalities of school meal programs? Outcomes to consider may include iron deficiency, body mass, and other measurements or behavior changes related to nutritional intake and dietary diversity. Through a quasi-experimental impact evaluation, the evaluation team will assess whether health and nutrition outcomes differ for children receiving school meals through different sourcing modalities. In this two-arm impact evaluation, children in arm 1 will receive school meals prepared with both locally procured commodities and U.S. commodities. In arm 2, children will only receive school meals prepared with U.S. commodities. The primary outcomes of interest are children’s KAP towards health and nutrition, perceived cultural appropriateness of the meals, and nutritional diversity of children’s diets. I10 | AIR.ORG LEARN Final Evaluation LEARN Endline Evaluation *Note LEARN Endline and LEARN II Baseline will be conducted in parallel A. Methodology The LEARN endline evaluation will follow the same sampling methodology as at midterm for both the project and impact evaluations. The same data collection tools and protocols will be used with some revisions to accommodate the research questions for the LEARN II project and impact evaluations. B. Key Evaluation Questions Like the midterm evaluation, the final evaluation will focus on questions of relevance, effectiveness, efficiency, impact, and sustainability. Key evaluation questions for the final evaluation include the following: Relevance  Did stakeholders feel that their voices were heard and their needs considered throughout the program?  Have activities to support literacy and improved nutrition been integrated in culturally appropriate ways in the target communities? Effectiveness  To what extent has the program achieved its output and outcome targets?  What factors have inhibited or facilitated the achievement of program goals, objectives, and expected results? Efficiency  Were intervention components delivered within their planned timeline?  Which commodity management strategies were most efficient for quick delivery and reduction of waste and theft?  Did school gardens produce enough food to supplement school meals adequately? Sustainability  Do schools have the necessary infrastructure and food management plans in place to continue feeding after the program concludes?  Do school-communities have the necessary systems in place to recruit and maintain volunteers for reading camps?  What are the necessary components for successful school handover of activities to the government and local community, as modeled by this program? What were the lessons learned?  Is there evidence that LEARN program activities and benefits are likely to continue or to scale up after the project ends? Impact  Has LEARN improved access to and quality of early grade reading materials in Liberia? I11 | AIR.ORG LEARN Final Evaluation  How has the home literacy environment in target communities changed in the LEARN program areas?  Have literacy skills of school-age children improved in the LEARN program area?  Has LEARN contributed to increased enrollment of school-age children and attendance of children and teachers?  Have nutrition, dietary, and food safety practices in schools improved in the LEARN program area?  Are PTAs meeting on a regular basis and contributing effectively to schools?  Have there been any positive or negative impacts in the target areas, besides the realization of the strategic objective-level results?  How do literacy and health KAP outcomes compare across the two treatment groups and one comparison group in Grand Gedeh County? Is there evidence of a positive impact of LEARN on literacy and health KAP outcomes? LEARN II Baseline Study Purpose and Scope Baseline data will be collected for four purposes:  To measure pre-implementation values for performance indicators;  To confirm estimated indicator targets;  To determine the baseline values for the performance evaluation and confirm the comparability of the intervention and comparison group schools for the impact evaluation; and  To confirm project design assumptions and identify potential threats to project implementation.  To conduct gender and power analysis study to inform more gender-sensitive programming LEARN II project staff will use values obtained from the baseline data collection to update initial indicator targets before the start of any project activities. Methodology, Tools, Research Design, and Sampling To ensure comparability of the baseline, midterm, and final evaluation findings, the baseline evaluation will use the same data collection tools and sampling methodology described in the evaluation approach and methodologies section above. In addition, a gender and power analysis study will be completed at baseline to inform greater inclusion of gender-sensitive approaches to programming. Key Audience and Stakeholders The project will plan the baseline evaluation in collaboration with Mercy Corps, Kawadah Farms, local, and national government partners, and USDA. The key audience for the baseline I12 | AIR.ORG LEARN Final Evaluation study will consist of the same stakeholder groups as described in the evaluation approach and methodologies section above. Timeline The baseline study will take place in Year 1 in the middle of the school year, in February-March 2022, prior to the start of project activities. USDA will receive the final baseline report within six months of finalization of the performance monitoring plan and evaluation plan. Since the LEARN endline and LEARN II baseline performance evaluation will coincide, and given the large (94%) overlap between LEARN and LEARN II schools, LEARN II will use the data collected in the LEARN endline performance evaluation to establish its baseline indicator values. As a result, LEARN II’s baseline performance evaluation will be staggered into two stages: Stage 1 will leverage quantitative data collection from the LEARN endline performance evaluation to inform the baseline performance indicator values for LEARN II. Stage 2 will consist of qualitative data collection for LEARN II. Qualitative data collection will succeed the quantitative data collection, and results from quantitative data collection will inform the design of qualitative data collection tools. Table 3: Timeline of Activities for the LEARN Endline/LEARN II Baseline Baseline Assessment Activity Month and Year Finalize Performance Monitoring Plan with USDA November 2021 Finalize Terms of Reference (ToR) for the evaluator with USDA November 2021 Advertise the LEARN II evaluator contract for the baseline, midline, and endline (including impact evaluation) November 2021 Recruit consultant and finalize contract Nov. – Dec. 2021 Refine Evaluation Methodology and Data Collection Tools January 2022 Data Collection  Stage 1 (Quantitative Data Collection)  Stage 2 (Qualitative Data Collection) February 2022 March 2022 Data Analysis April 2022 Conduct Stakeholder Meetings to share initial evaluation findings May 2022 Finalize and submit draft LEARN endline report and LEARN II baseline report to USDA (within 60 days of evaluation fieldwork and within 15 days of report completion) May 2022 Submit final endline and baseline report and established targets to USDA June 2022 Discuss actions to address findings and recommendations with USDA July 2022 Report on implementation of follow-up activities September 2022 I13 | AIR.ORG LEARN Final Evaluation Midterm Evaluation Purpose and Scope The purpose of the midterm performance evaluation is to assess the progress of LEARN II’s implementation, assess the relevance and early effectiveness of the interventions, determine whether the project is on track to meet its objectives, summarize the lessons learned to date, and recommend any necessary changes to project components. The midterm evaluation will use the same data collection tools from the baseline assessment. The evaluation will also include key informant interviews and focus group discussions with students, parents, PTA members, teachers, principals, school meal cooks, storekeepers, local government officials, and cassava cooperative members. Evaluation Questions The midterm evaluation will assess the relevance, effectiveness, efficiency, sustainability, and impact of the key project interventions through the following evaluation questions. The results for all evaluation questions will be disaggregated by sex, age, and socio-economic status: Relevance  To what extent are the project education, school feeding, and health practices aligned with Government of Liberia’s priorities and policies? If they are different, what are the differences?  Are the in-school meals perceived as being culturally appropriate by students? Are meals prepared with local commodities perceived as more culturally appropriate than meals prepared with U.S. commodities?  Do students and teachers perceive educational and instructional materials as culturally￾and age-appropriate and empowering for girls and boys?  Do project stakeholders feel that LEARN II is meeting their needs? Why or why not? Effectiveness  To what extent has the project achieved its output and outcome targets? Did these outcomes vary for girls and boys? For children from families with different household income/wealth? For children with disabilities?  What factors have inhibited or facilitated the achievement of project goals, objectives, and expected results?  Have project interventions been effective in making the school environment safe and inclusive for all students and teachers? Is it different for girls, boys, men and women? Efficiency  Were intervention components delivered within the planned timeline?  How often and in what ways are schools utilizing produce from local producers to supplement and/or substitute donated food? Sustainability I14 | AIR.ORG LEARN Final Evaluation  Do schools and school meal providers have the necessary infrastructure and food management plans in place to continue feeding after the project concludes?  What are the current barriers to achieving sustainability according to the different stakeholders?  What are the necessary resources (financial, physical, knowledge, partnerships) for schools to successfully continue activities currently supported by the project, once project support ends? Impact  Have literacy skills of school-age children improved in the LEARN II project area? What factors are associated with children showing more or less improvement?  Have nutrition and health outcomes of school-age children improved in the LEARN II project area? What factors are associated with children showing more or less improvement?  Have nutrition, dietary, and food safety practices in schools improved in the LEARN II project area?  Have there been any positive or negative impacts of LEARN II intervention in the target areas, besides the realization of the strategic objective-level results of the project?  How do literacy, health and nutrition, and Knowledge, Attitudes, and Practices (KAP) outcomes compare across the intervention and comparison group? Methodology, Tools, Research Design, and Sampling To ensure comparability of the midterm evaluation findings with the baseline, the midterm evaluation will use the same data collections tools and sampling methodology described in the evaluation approach and methodologies section above. Key Audience and Stakeholders The project will plan the midterm evaluation in collaboration with Mercy Corps, Kawadah Farms, local and national government partners, and USDA. The key audience for the midterm evaluation will consist of the same stakeholder groups as described in the evaluation approach and methodologies section above. Timeline Preparation for the midterm evaluation will commence at the end of Year 2 in August 2023, with data collection in March 2024 and a final report submitted to USDA in June 2024. Table 4: Timeline of Activities for the Midterm Evaluation Midterm Evaluation Activity Month and Year Submit draft ToR for the midterm evaluation to USDA August 2023 Finalize midterm evaluation ToR with USDA and evaluator October 2023 Revise and finalize evaluator contract November 2023 I15 | AIR.ORG LEARN Final Evaluation Prepare for midterm evaluation  Finalize internal project evaluation team  Finalize evaluation design with firm and government Dec. 2023 – Feb. 2024 Data collection March 2024 Data analysis April 2024 Conduct stakeholder meetings to share initial evaluation findings April-May 2024 Finalize and submit draft midterm report to USDA (within 60 days of evaluation fieldwork and within 15 days of report completion) May 2024 Submit final midterm report and established targets to USDA June 2024 Discuss actions to address findings and recommendations with USDA July 2024 Result dissemination and learning event July 2024 Report on implementation of follow-up activities September 2024 I16 | AIR.ORG LEARN Final Evaluation Final Evaluation Purpose and Scope The purpose of the final evaluation is to assess whether the project achieved the results outlined in LEARN II’s results framework. In addition, the evaluation team will compare cost efficiency, and perceived cultural appropriateness between the different school meals. The team will use a quasi-experimental impact evaluation design to determine the impact of LRP school meals on children’s health and nutrition outcomes. The final evaluation will use the same quantitative and qualitative methods as the baseline and midterm evaluations to explore questions related to project design, implementation, management, lessons learned, sustainability, and impact. The evaluation team will explore the pathways through which local producers satisfy the food demand of schools with a focus on the sustainability and national￾level scaling up of the current feeding model. LEARN II will invite stakeholders to participatory workshops to review and reflect on project findings and to offer input for ensuring the continuity of LEARN II approaches. Key Evaluation Questions Like the midterm evaluation, the final evaluation will focus on questions of relevance, effectiveness, efficiency, impact, and sustainability. The results for all evaluation questions will be disaggregated by sex, age, and socio-economic status. The key evaluation questions for the final evaluation include: Relevance  Did stakeholders feel that their voices were heard and needs considered throughout the project?  Has the project integrated activities to support literacy and improved nutrition in culturally appropriate ways in the target communities? Effectiveness  To what extent has the project achieved its output and outcome targets?  What factors have inhibited or facilitated the achievement of goals, objectives, and expected results?  Has the classroom environment changed with respect to safety and inclusivity in LEARN II supported schools? If so, how?  Was the project’s Take Home Ration component effective in incentivizing reading club facilitators, school cooks, and storekeepers?  Are the Parent Teacher Associations more inclusive, empowered, and engaged in school management activities? Efficiency  Did the project deliver intervention components within their planned timelines? If not, why? I17 | AIR.ORG LEARN Final Evaluation  Which commodity management strategies were most efficient for quick delivery and reduction of waste?  Was the unit cost of school meals lower for schools where locally procured commodities were offered alongside the USG commodities?  Did school gardens produce enough food to supplement school meals? If yes, to what extent? Sustainability  Under which conditions are local producers able to satisfy food demand of schools for the feeding program? What are the pathways that lead to successful sale and delivery of food by local producers to processors and/or schools?  Do schools have the necessary infrastructure, food management plans, and systems in place to continue feeding and literacy activities after the project concludes?  What are the necessary components for successful handover of activities to schools, as modeled by this project? What were the lessons learned?  Was the local commodity procurement model successful in providing schools with nutritious and cost-effective school meals? What is required to scale this model to other schools in Liberia? Is this model sustainable after LEARN II ends? Which commodity procurement modality is more sustainable to deliver school meals in Liberia? Impact  Have literacy skills of school-age children improved in the LEARN II schools?  Have nutrition, dietary, and food safety practices improved in the LEARN II schools?  Have teachers’ knowledge, attitudes, and practices about teacher code of conduct, norms related to corporal punishment, and abuse changed?  Has parental engagement with children’s learning and development changed?  Has LEARN II contributed to regular attendance of school-age children?  Is there a significant impact of LRP and school gardens on children’s health and nutrition outcomes?  Does the LRP component of the project improve the cost-effectiveness, timely delivery, and cultural appropriateness of the school meals provided to children?  Have there been any positive or negative impacts in the target areas, besides the realization of the strategic objective-level results since the midline assessment? Methodology Tools, Research Design, and Sampling The LEARN II final evaluation will use the evaluation methodology detailed in the research design and methodologies section. Key Audience and Stakeholders The key audience and stakeholders for the final evaluation will be the same as the key audience and stakeholders described in the research design and methodologies section above. I18 | AIR.ORG LEARN Final Evaluation Timeline Preparation for the final evaluation will commence at the end of Year 4 in August 2025, with data collection in March 2026 and a final report submitted to USDA in June 2026. Table 5: Timeline of Activities for the Final Evaluation Final Evaluation Activity Month and Year Submit draft ToR for the final evaluation to USDA August 2025 Finalize final evaluation ToR with USDA and evaluator October 2025 Revise and finalize evaluator contract November 2025 Prepare for final evaluation  Finalize internal project evaluation team  Finalize evaluation design with consultant/firm and government Dec. 2025 – Feb. 2026 Data collection March 2026 Data analysis April 2026 Conduct stakeholder meetings to share initial evaluation findings April-May 2026 Results dissemination and learning event July 2026 Finalize and submit draft final evaluation report to USDA (within 60 days of evaluation fieldwork and within 15 days of report completion) May 2026 Submit finalized final evaluation report to USDA June 2026 I19 | AIR.ORG LEARN Final Evaluation III. EVALUATION MANAGEMENT A. Roles and Responsibilities LEARN II MEAL staff in Liberia will manage the monitoring and evaluation of the project. Save the Children’s U.S.-based education and MEAL technical advisory staff will provide technical input on the development of tools, sampling plan, electronic data collection instruments, assessor training, and piloting of tools. For the baseline, midterm, and final evaluations and the impact evaluation, Save the Children will contract an independent third-party consultant firm to collect baseline, midterm, and final evaluation data that is reliable, accurate, valid, and timely. Save the Children will support the independent consultant firm through review of the survey plan, survey instruments, sampling methods, and the development of a data analysis plan based on the project indicators. MEAL staff in Liberia will be responsible for managing the commodity monitoring system as well as maintaining the M&E monitoring system used for internal data collection and semi-annual reporting to USDA. MEAL staff will conduct monthly visits to project communities to monitor and collect data on project activities. The Senior MEAL Coordinator and technical team will review all data in the M&E monitoring system before submitting evaluation reports to USDA. See Table 13 below for additional details on roles and responsibilities. Table 6: Roles and Responsibilities Save the Children Senior MEAL Coordinator: Draft and revise baseline, midterm, and final evaluation ToR for external evaluator; support selection of external evaluator; review draft evaluation methodology and tools; review data in monitoring system; finalize and submit donor reports; coordinate dissemination events with regional MEAL staff and external evaluator. Chief of Party: Recruit and contract external evaluator; review ToR for baseline, midterm, and final evaluations; review draft evaluation methodology and tools; review evaluation reports. MEAL Officers: Obtain necessary permits for evaluation activities, collect and input data in monitoring system, contribute to and review donor reports, and coordinate dissemination events. Save the Children Research Team: Revise assessment tools as necessary, support Senior MEAL Coordinator and external evaluator to conduct data collection, review evaluation reports. External Evaluator Draft and finalize tools and methodology, train enumerators and field test tools for evaluations, manage data collection and ensure data quality, analyze evaluation data, facilitate reflection event for participatory analysis of preliminary results, co-facilitate evaluation dissemination I20 | AIR.ORG LEARN Final Evaluation events, prepare draft and final reports. The independent evaluator will be free to draw their conclusions devoid of organizational or political pressure. Local University/Research Partner The external evaluator consultant/team will be required to work with a local university/research partner or data collection agency based in Liberia. The local knowledge and data collection partner will play a critical role in supporting the external evaluation consultant/team to adapt the evaluation questions and data collection tools to the local context to ensure that the performance and impact evaluations are culturally relevant. Government Partners County and District Education Staff, MoH Staff, MoE Staff, MOA staff: Contribute to design of ToR and interpretation of evaluation findings, and participate in reflection and dissemination workshops. USDA Comment and approve evaluation ToRs and reports and participate in a stakeholder phone call with the third-party evaluator. B. Learning Save the Children will share the learnings from LEARN II evaluations with stakeholders, such as beneficiaries, local authorities, government agencies in Liberia, other local or regional organizations working in the education and nutrition sector, USDA, and other USG-funded education projects. LEARN II will use varied strategies and formats to disseminate the evaluation findings and facilitate learning with LEARN II stakeholders. The evaluation team and the LEARN II MEAL staff will share the results of the evaluation and performance monitoring with the LEARN II programming teams to adaptively manage the implementation of LEARN II activities. These findings will also be shared with GoL (including MoE, MoH, and MoA) and other in-country development partners to inform education, feeding and health priorities in Liberia. The project will also hold dissemination events after the baseline, midterm, and final evaluations to present findings to community members, including project beneficiaries such as students, parents, teachers, school principals, PTAs, local producers, and VSLA members. These opportunities will also be used to collect feedback from project beneficiaries to ensure that their feedback is incorporated in the project implementation. This will ensure that LEARN II stays accountable to project beneficiaries and is managed adaptively to continue to meet beneficiaries’ needs throughout its implementation. Save the Children will ensure that results are shared in appropriate formats (e.g., stakeholder workshops and on Save the Children’s external website) and at various venues, including government partnership meetings, internal Save the Children presentations and workshops, and externally facing conferences such as the Comparative International Education Society I21 | AIR.ORG LEARN Final Evaluation annual conference. Additionally, USDA will be notified and invited to attend key events where evaluation results will be presented. C. Evaluation Budget The LEARN II evaluation budget is $709,276. Additional details can be found in Table 7 below. Table 7: Evaluation Budget Budget Line Cost (in USD) Baseline $134,762154,935 Midline $276,618270,535.59 Endline $297,896283,805.41 Total $709,276 I22 | AIR.ORG LEARN Final Evaluation D. Deliverables The consultant should submit the following deliverables for each stage of the evaluation process (baseline36, midterm, endline) during the evaluation process: An inception report A research protocol that includes at a minimum: Principal Investigator, Country/Location, Objectives, Research Questions, Research Design, Sample, Data Collection Methods, Data Analysis Methods and Plan, Quality Assurance Plan, Data Handling and Confidentiality, Consent and Assent Forms, Translation Services (if needed), and Data Collection Tools. The research protocol will be submitted to Save the Children US Ethics Review Committee (ERC) and the consultant will incorporate ERC’s input. Data collection tools developed for primary data collection. A draft report A final report submitted in English that incorporates Save the Children’s feedback into the draft report (public and internal versions, where relevant) Raw data (both qualitative and quantitative) and appropriate data documentation including a data dictionary Cleaned datasets Presentation of key findings to be delivered at an evaluation stakeholders’ meeting (x2 - in-country and to USDA Washington) Standalone summary37 As noted, the deliverables will be reviewed and approved by the Save the Children team, which will include the SC/Liberia Chief of Party, the Senior MEAL Coordinator, SC/Washington technical advisors, as well as USDA/Washington. Save the Children expects that the final reports will include the following sections, at a minimum: Cover Page Acronym List Executive Summary Project Background Objectives of the Evaluation Key Evaluation Questions Evaluation Methodology Evaluation Results Conclusions (successes and challenges) Recommendations 36 A baseline report for LEARN II as well as an endline report for LEARN will be expected at the same time 37 A two to three-page stand-alone summary describing the evaluation design, key findings and lessons learned. This document will serve to inform any interested stakeholders of the final evaluation, and should be written in a language easy to understand by non-evaluators and with appropriate graphics and tables. I23 | AIR.ORG LEARN Final Evaluation Lessons Learned A minimum of two success stories (not relevant for baseline) Performance indicator tables including custom and standard indicators and updated values Attachments (photos, charts, graphs, regression analysis results)38 The final versions of the baseline, midterm, and final evaluation reports must be submitted in two hard copies and in electronic format. 38 Any photos, particularly of children, must be accompanied by a parental media consent form in the annexes. I24 | AIR.ORG LEARN Final Evaluation IV. Annex A. Project Map Annex B: LEARN II Project-Level Results Framework I25 | AIR.ORG LEARN Final Evaluation I26 | AIR.ORG LEARN Final Evaluation I27 | AIR.ORG LEARN Final Evaluation About the American Institutes for Research Established in 1946, with headquarters in Arlington, Virginia, the American Institutes for Research® (AIR®) is a nonpartisan, not-for-profit organization that conducts behavioral and social science research and delivers technical assistance to solve some of the most urgent challenges in the U.S. and around the world. We advance evidence in the areas of education, health, the workforce, human services, and international development to create a better, more equitable world. For more information, visit AIR.ORG. Save the Children 899 North Capitol St. NE, Suite 900 Washington, D.C., 20002 www.SavetheChildren.org AIR® Headquarters 1400 Crystal Drive, 10th Floor Arlington, VA 22202-3289 +1.202.403.5000 | AIR.ORG Notice of Trademark: “American Institutes for Research” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners. Copyright © 2022 American Institutes for Research® and California Department of Social Services. All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, website display, or other electronic or mechanical methods, without the prior written permission of the American Institutes for Research. For permission requests, please use the Contact Us form on AIR.ORG. 17934_06/22