This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by International Solutions Group. Foreign Agricultural Service, United States Department of Agriculture Student seated at a desk in a McGovern-Dole supported school in Laos. Photo courtesy of Catholic Relief Services Laos McGovern-Dole Project Midterm Evaluation February 2020 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. USDA McGovern-Dole International Food for Education and Child Nutrition Program – Learning and Engaging All in Primary School (LEAPS) II Project, Lao PDR The United States Department of Agriculture (USDA) McGovern-Dole International Food for Education and Child Nutrition (McGovern-Dole) project in Lao PDR is implemented by Catholic Relief Services (CRS) from 2016 to 2021 and is titled Learning and Engaging All in Primary School (LEAPS) II. The project aims primarily to improve the literacy skills of school-age children and increase the use of health and dietary practices in 350 schools across seven educationally disadvantaged districts of Savannakhet province. Agreement Number: FFE-439-2016/009-00 Project Duration: 2016-2021 Implemented by: Catholic Relief Services (CRS) Evaluation Authored by: IMPAQ International, LLC Elnaz Safarha Jessica Li Daniel Zaas Connor Harrison IMPAQ International, LLC Photo courtesy of Catholic Relief Services USDA McGovern-Dole International Food for Education and Child Nutrition Program – Learning and Engaging All in Primary School (LEAPS) II Project, Lao PDR Final Midterm Performance Evaluation Report February 2020 LEAPS II Midterm Performance Evaluation Report i  February 2020 SUBMITTED TO Catholic Relief Services (CRS) Savannakhet, Lao PDR www.crs.org ATTENTION CRS Savannakhet, Lao PDR SUBMITTED BY IMPAQ International, LLC 10420 Little Patuxent Parkway Suite 300 Columbia, MD 21044 (443)256-5500 www.impaqint.com PROJECT USDA McGovern-Dole International Food for Education and Child Nutrition – Learning and Engaging All in Primary School (LEAPS) II Project (2016-2021), Award Number FFE-439-2016/009-00, Lao PDR – Performance and Impact Evaluations TASK & DELIVERABLE Midterm Performance Evaluation Report AUTHORS Elnaz Safarha, IMPAQ International Jessica Li, IMPAQ International Daniel Zaas, IMPAQ International Connor Harrison, IMPAQ International ii IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report ACKNOWLEDGEMENTS IMPAQ thanks the U.S. Department of Agriculture (USDA) and Catholic Relief Services (CRS) for their financial support. The evaluation team extends our special thanks to Cornelia Sage, Patrick Williams, Inpeng Bounyavong, Vatvisa Keosalivong, and their colleagues at CRS for their support and advice. IMPAQ also thanks CRS’ remarkable field staff and community mobilizers in Lao PDR for facilitating the planning and rollout of the midterm data collection. Special thanks go to SKO, our data collection partner, and their team leader, Salika Khoonbarthao, who supervised a thorough and 18 amazing and professional enumerators. Without the support and hard work of the enumerators in the field, the evaluation team would not have been able to collect data with such high quality. IMPAQ appreciates Save the Children staff for being available to provide support and advice when needed. IMPAQ also thanks Maria DiFuccia, Research Associate at IMPAQ International, for her solid insights and technical support. The evaluation team also acknowledgesthe government of Lao PDR for providing IMPAQ permission to conduct fieldwork in Lao and assistance facilitating the logistics of our data collection. Finally, IMPAQ thanks all the local stakeholders and beneficiaries who freely shared their perspectives and enabled the development of in￾depth analysis for the report. The findings, interpretations, and conclusions expressed in this report are entirely those of the authors. They do not necessarily represent the views of IMPAQ International, CRS, or USDA. LEAPS II Midterm Performance Evaluation Report iii  February 2020 TABLE OF CONTENTS Acknowledgements....................................................................................................................................... ii Table of Contents......................................................................................................................................... iii Table of Exhibits........................................................................................................................................... vi List of Acronyms......................................................................................................................................... viii Executive Summary...................................................................................................................................... ix Methods............................................................................................................................................... ix Key Findings ......................................................................................................................................... ix Recommendations............................................................................................................................... xi Section 1. Introduction .................................................................................................................................1 1.1 Overview of Program Implementation...............................................................................................1 1.1.1 Provision of School Meals............................................................................................................2 1.1.2 Literacy Interventions..................................................................................................................2 1.1.3 Inclusive Education (IE)................................................................................................................3 1.1.4 Water, Sanitation, and Hygiene (WASH) .....................................................................................4 1.2 Overview of Evaluation Background...................................................................................................4 Section 2. Evaluation Approach....................................................................................................................6 2.1 Research Questions ............................................................................................................................6 2.2 Evaluation Methodology.....................................................................................................................6 2.2.1 Quantitative Sampling Design......................................................................................................6 2.2.2 Qualitative Sampling Design ........................................................................................................8 2.3 Data Sources .....................................................................................................................................10 2.3.1 Student Survey and LBRA...........................................................................................................10 2.3.2 Classroom Observations ............................................................................................................11 2.3.3 Qualitative Protocols..................................................................................................................11 2.4 Data Analysis.....................................................................................................................................11 2.4.1 Quantitative Data Analysis.........................................................................................................11 2.4.2 Qualitative Data Analysis...........................................................................................................12 Section 3. Fieldwork....................................................................................................................................13 3.1 Human Subject Protection................................................................................................................13 3.2 Preparation for Data Collection ........................................................................................................13 3.3 Field Activities...................................................................................................................................14 iv IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report 3.4 Field Challenges ................................................................................................................................14 Section 4. Evaluation Sample......................................................................................................................16 4.1 Schools..............................................................................................................................................16 4.2 Students............................................................................................................................................16 4.3 Household Characteristics ................................................................................................................18 4.3.1 Household Size...........................................................................................................................18 4.3.2 Language Spoken at Home ........................................................................................................18 4.3.3 Socioeconomic Status................................................................................................................19 Section 5. Quantitative Results...................................................................................................................21 5.1 School Environment..........................................................................................................................22 5.1.1 Student Attitudes toward Schooling..........................................................................................22 5.1.2 School Libraries..........................................................................................................................23 5.1.3 Classroom Activities...................................................................................................................24 5.2 Household Literacy Environment......................................................................................................25 5.2.1 Availability of Reading Material at Home ..................................................................................25 5.2.2 Home Literacy Practices.............................................................................................................26 5.3 Student Outcomes............................................................................................................................27 5.3.1 Health.........................................................................................................................................27 5.3.2 Food Security .............................................................................................................................29 5.3.3 Attentiveness.............................................................................................................................30 5.3.4 Literacy Assessment Outcomes.................................................................................................31 Section 6. Qualitative Results .....................................................................................................................38 6.1 Relevance..........................................................................................................................................39 6.1.1 Alignment with Government Objectives ...................................................................................39 6.1.2 Adequacy of Trainings to Address Knowledge Gaps..................................................................39 6.1.3 Acceptability and Appropriateness of School Meals .................................................................40 6.2 Effectiveness.....................................................................................................................................41 6.2.1 Effect of School Meals on Attendance and Attentiveness.........................................................42 6.2.2 WASH Knowledge, Attitudes, and Practices..............................................................................43 6.2.3 Increased Capacity after Trainings.............................................................................................45 6.2.4 Learning and Stakeholder Dialogue ...........................................................................................46 6.3 Efficiency ...........................................................................................................................................47 6.3.1 Management Structure..............................................................................................................47 LEAPS II Midterm Performance Evaluation Report  February 2020 v 6.3.2 Monitoring .................................................................................................................................47 6.4 Sustainability.....................................................................................................................................48 6.4.1 School Meals..............................................................................................................................48 6.4.2 Literacy Education......................................................................................................................50 6.4.3 WASH and School Infrastructure ...............................................................................................51 6.5 Perceived Impact ..............................................................................................................................52 6.5.1 Changes in Literacy Education ...................................................................................................52 6.5.2 Attendance and Drop-out Rates................................................................................................53 6.5.3 Unintended and Differentiated Impacts....................................................................................54 Section 7. Conclusion..................................................................................................................................56 7.1 Limitations.........................................................................................................................................56 7.2 Key Outcomes and Implications........................................................................................................57 7.3 Recommendations............................................................................................................................58 7.3.1 Project Recommendations.........................................................................................................58 7.3.2 Evaluation-Specific Recommendation .......................................................................................60 Appendices..................................................................................................................................................62 Appendix A. References..............................................................................................................................A1 Appendix B. LEAPS II Evaluation Questions and Conceptual Framework...................................................B1 Appendix C. McGovern Dole Indicators......................................................................................................C1 Appendix D. List of Stakeholders................................................................................................................D1 Appendix E. Additional Tables and Complementary Outcomes................................................................. E1 Appendix F. Other Subtests of Reading Assessment.................................................................................. F1 Appendix G. Regression Analysis............................................................................................................... G1 Appendix H. Inter-rater Reliability ............................................................................................................. H1 Appendix I. Survey Instruments...................................................................................................................I1 Appendix J. Qualitative Protocols................................................................................................................J1 Appendix K. Midterm Evaluation SOW and TOR ......................................................................................K19 vi IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report TABLE OF EXHIBITS Exhibit 1. LEAPS II Targeted Districts in Savannakhet Province....................................................................1 Exhibit 2. Sampling Strategy .........................................................................................................................7 Exhibit 3. Respondents by Type and Gender................................................................................................8 Exhibit 4. Categorization of School and Community Stakeholders...............................................................8 Exhibit 5. Determination of High-Performing and Low-Performing Schools................................................9 Exhibit 6. Distribution of Midterm Sample .................................................................................................16 Exhibit 7. Student Gender Distribution.......................................................................................................17 Exhibit 8. Age Distribution by Grade...........................................................................................................17 Exhibit 9. Grade Repetition by Grade .........................................................................................................18 Exhibit 10. Household Size..........................................................................................................................18 Exhibit 11. Main Language Spoken at Home ..............................................................................................19 Exhibit 12. Socioeconomic Status...............................................................................................................20 Exhibit 13. Midterm Levels for McGovern-Dole Performance Indicators ..................................................21 Exhibit 14. Borrowing Storybooks from School ..........................................................................................23 Exhibit 15. Frequency of Classroom Activities............................................................................................24 Exhibit 16. Reading Material Available in Students’ Homes.......................................................................26 Exhibit 17. Home Literacy Environment .....................................................................................................27 Exhibit 18. Health-Related Absences..........................................................................................................28 Exhibit 19. Most Common Illnesses among Students.................................................................................28 Exhibit 20. School Lunches and Student Hunger ........................................................................................29 Exhibit 21. Second-Grade Students’ Literacy Skills.....................................................................................32 Exhibit 22. Number of Letters Identified ....................................................................................................33 Exhibit 23. Letter Proficiency by Gender ....................................................................................................34 Exhibit 24. Proportion of Students Defined as Readers, by District ...........................................................35 Exhibit 25. Students Who Passed Each Literacy Section.............................................................................36 Exhibit 26. Reading Proficiency...................................................................................................................37 Exhibit 27. Summary of Qualitative Results................................................................................................38 Exhibit 28. Key Evaluation Questions..........................................................................................................B1 Exhibit 29. LEAPS II Conceptual Framework ...............................................................................................B3 Exhibit 30. McGovern Dole Indicators........................................................................................................C1 Exhibit 31. List of Stakeholder Interviews...................................................................................................D1 Exhibit 32. Other Languages Spoken at Home............................................................................................ E1 Exhibit 33. Total Number of SES Goods Owned by District ........................................................................ E1 Exhibit 34. Frequency of Teachers Asking Questions about Stories by District.......................................... E2 Exhibit 35. Frequency of Teachers Playing Alphabet or Reading Games by District.................................. E2 Exhibit 36. Frequency that Teachers Tell a Story Not from Textbooks by District ..................................... E3 Exhibit 37. Availability of Books at Home ................................................................................................... E3 Exhibit 38. What Students Like about School ............................................................................................. E4 Exhibit 39. Lunch was Served at School by District..................................................................................... E4 Exhibit 40. Amount of Time Spent on Each Student Activity...................................................................... E5 Exhibit 41. Amount of Time Spent on Each Teacher Activity...................................................................... E5 Exhibit 42. Distribution of Letter Identification by Language..................................................................... E6 Exhibit 43. Reasons for Liking School by Treatment................................................................................... E6 Exhibit 44. Frequency of Classroom Activities by SF+LB Schools................................................................ E7 Exhibit 45. Attentiveness by Student Activity............................................................................................. E7 LEAPS II Midterm Performance Evaluation Report vii  February 2020 Exhibit 46. Expressive Vocabulary by District ............................................................................................. F1 Exhibit 47. Word Pairs................................................................................................................................. F2 Exhibit 48. Most Common Used Words at Midterm .................................................................................. F3 Exhibit 49. Predictors of Literacy Skills at Midterm................................................................................... G2 Exhibit 50. Predictors of Literacy Skills at Midterm Detailed Regression.................................................. G2 Exhibit 51. IRR by Literacy Skill Subtests for Performance Sample............................................................ H1 viii IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report LIST OF ACRONYMS CBR Community-based Rehabilitation CI Confidence Interval CLV Community Literacy Volunteer COP Chief of Party CRS Catholic Relief Services CM Community Mobilizers DCOP Deputy Chief of Party DESB District Education and Sports Bureau ECD Early Childhood Development EMC Emerging Markets Consulting FFE Food For Education FGD Focus Group Discussion IE Inclusive Education IRB Institutional Review Board IRR Inter-rater Reliability KII Key Informant Interview LB Literacy Boost LBRA Literacy Boost Reading Assessment LEAPS Learning and Engaging All in Primary School LEARN Liberia Empowerment through Attendance, Reading, and Nutrition MGD McGovern-Dole MoES Ministry of Education and Sports NGO Non-governmental Organization OLS Ordinary Least Squares O&M Operations and Maintenance PA Pedagogical Advisor PESS Provincial Education and Sports Service PMP Performance Monitoring Plan SCI Save the Children International SES Socioeconomic Status SF School Feeding SILC Savings and Internal Lending Community SKO SKO, Co., Ltd. SOW Scope of Work THR Take Home Rations TOR Terms of Reference TOT Training of Teachers UO University of Oregon USAID United States Agency for International Development USDA United States Department of Agriculture VEDC Village Education Development Committee WASH Water, Sanitation, and Hygiene WFP World Food Programme LEAPS II Midterm Performance Evaluation Report  February 2020 ix EXECUTIVE SUMMARY Learning and Engaging All in Primary School (LEAPS) II is a five-year project (2016–2021) funded by the U.S. Department of Agriculture McGovern-Dole International Food for Education and Child Nutrition program. Catholic Relief Services (CRS) implements LEAPS II in Lao People’s Democratic Republic (PDR) with government partners, Save the Children International (SCI), and the University of Oregon (UO). LEAPS II aims primarily to improve the literacy skills of school-age children and increase the use of health and dietary practices in 350 schools across seven educationally disadvantaged districts of Savannakhet province. LEAPS II supports school meals; trains educators in Literacy Boost (LB) teaching methods and inclusive education (IE); and strives to improve water, sanitation, and hygiene (WASH) practices. CRS selected IMPAQ International to design and conduct the performance and impact evaluation of LEAPS II. This midterm performance evaluation report documents the progress of LEAPS II to date in achieving project outcomes and recommends mid-course corrections. Methods The midterm performance evaluation used a mixed-methods approach, collecting data from: ▪ Surveys of students in Grades 1–5 from a sample of 83 LEAPS II schools. For students in Grade 2, the survey included a reading assessment. ▪ Classroom observations to assess student attentiveness in all the LEAPS II evaluation schools. ▪ Focus group discussions (FGDs) and key informant interviews (KIIs) with local Ministry of Education and Sports (MoES) stakeholders and school and community-level individuals, including students, parents, teachers, cooks, principals, Village Education Development Committees (VEDCs), and community literacy volunteers (CLVs). Interviews with LEAPS II staff also provided valuable contextual information and insight on activity progress. IMPAQ followed similar sampling methods and instruments for the midterm evaluation (March 2019) as for the baseline (March 2017). The qualitative research included several new stakeholders, including students, a representative from the Provincial Education and Sports Service, District Education and Sports Bureau (DESB) officers, Pedagogical Advisors (PAs), community literacy volunteers (CLVs), and community mobilizers (CMs). Key Findings Quantitative Findings The evaluation found statistically significant improvements from baseline in four of the five key LEAPS II McGovern-Dole performance indicators: ▪ Health-related absences. Among students who reported that they were sick in the last week, a lower proportion at midterm (27 percent) than at baseline (33 percent) reported having missed school. These results were the same for girls and boys. ▪ Student hunger during afternoon class. Fewer students reported at midterm (4 percent) than at baseline (8 percent) that they felt hungry during the day, with no gender differences. ▪ Letter recognition in Grade 2. The proportion of students who could identify at least 75 percent of the letters increased significantly from baseline (47 percent) to midterm (73 percent). This improvement was much larger for girls (31 percentage points) compared to boys (22 percentage x IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report points) even though their level of proficiency was almost the same at baseline (46-47 percent, respectively). ▪ Basic reading comprehension in Grade 2. More students were able to read at their grade level (Grade 2) at midterm (11 percent) than at baseline (3 percent). Improvements in comprehension primarily drove this result. Exhibit 30 in Appendix C presents a full list of MGD indicators mapped to their project targets, as reported by IMPAQ and CRS. The one McGovern-Dole indicator in which LEAPS II schools did not show improvement was classroom attentiveness, measured by classroom observations. Perhaps teachers spending more time on class management than teaching, as the observations at midterm showed, could explain the low level of attentiveness at midterm. The qualitative notes also suggest several factors that could influence how well students pay attention in class, including unappealing teaching materials and ineffective classroom management of multi-grade classes. These findings suggest correlations between the observed changes in outcomes and LEAPS II, but not causation. Although the baseline and midterm studies used the same methods and instruments, the sample of students varied. Furthermore, the analysis cannot account for confounding factors outside of LEAPS II. Though the results show that LEAPS II students made progress toward project goals, literacy achievement among Grade 2 students remains modest. However, the results are consistent with Lao curriculum expectations of gains in reading comprehension becoming more apparent from Grade 3 onward. Qualitative Findings The qualitative data offer context for the quantitative results and examine the performance of the project in five categories: relevance, effectiveness, efficiency, sustainability and impact (perceived). The graphic below summarizes outcomes in those research areas to highlight the main takeaways while Section 6. Qualitative Results provides additional details. LEAPS II Midterm Performance Evaluation Report xi  February 2020 Recommendations Based on our analysis and lessons learned, IMPAQ has developed the following recommendations to enhance the effectiveness and sustainability of LEAPS II: ▪ Work with MoES to train teachers on how to manage multi-grade classrooms and develop materials to better integrate these training concepts into national teaching guidelines. According to findings from the qualitative research, teachers reported feeling poorly equipped to handle multi-grade classrooms, and that adapting some of the current trainings have been more difficult in this context. Other research reiterates this challenge and links teachers’ inability to handle multi-grade classes to poor student learning outcomes.1 LEAPS II can address this need by collaborating with MoES to integrate management of multi-grade classrooms into trainings for teachers and other stakeholders who could potentially influence the effectiveness of these trainings, such as PAs and principals. To develop guidelines for teachers on best practices in multi￾grade settings, CRS could facilitate peer-to-peer exchanges among teachers in districts and provinces in Lao PDR or with neighboring countries, as recommended in a previous study of multi￾grade teaching practices in Southeast Asia.2 For sustainability of trainings, CRS should continue to 1 Thephavongsa, Souphanh. 2018. Enhancing the teaching skills of the multi-grade teachers through lesson study. International Journal of Learning, Teaching and Educational Research, Vol. 17, No.4, pp.71-87. UNICEF. 2015. Student Learning Outcomes in Primary Education in Lao PDR: Situational Analysis. 2 Adam Smith International. 2017. Rapid Appraisal of Literature on Multi-grade Teaching in ASEAN and Other Countries. •Teachers and principals reflected positively on the different trainings related to improved pedagogy. •Parents and students found the meals to be too uniform and wanted more diverse food options. Relevance •School meals decrease the economic burden of providing food for children and relieves anxiousness and preoccupation with hunger for students while in class, which helps them to pay attention. •Lack of behavior change within households may impede improved student health and hygiene. A more overarching WASH challenge remains access to water; even in those schools where LEAPS II or other donors have drilled boreholes and provided O&M trainings, communities struggle with effective maintenance. •Capacity-building activities could be enhanced via prolonged, more frequent, and interactive trainings. Effectiveness •CRS management of activities has been adaptive and responsive to community needs. •CRS staff were mostly satisfied with the monitoring, but school and community stakeholders want more frequent visits; DESB noted that it would like to participate more fully in monitoring efforts. Efficiency •VEDCs have had varying levels of success mobilizing the community to contribute to school meals. •Lessons learned from successes and challenges of teacher trainings can be helpful in identifying strategies for more long-lasting and impactful approaches to improve pedgagogy in the classroom. •Reading camps will be difficult to sustain without community buy-in. Sustainability •School feeding has been a critical support to enable children to attend school. •Reading camps and teacher trainings were cited as the elements of LB that had the most impact on improved literacy and learning outcomes. Impact xii IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report work closely with MoES and other relevant stakeholders to ensure MoES training materials for teachers reflects the training concepts related to multi-grade classroom management. Also useful would be to distill such documents into simple handbooks for teachers. ▪ Continue building the capacity of DESB and PAs in financial management so that these local MoES officials can better guide teachers, monitor teaching quality, and deliver refresher trainings. Teachers, PAs, CLVs, and principals expressed a need for more frequent training to help teachers apply practical, hands-on pedagogical methods in the classroom. As such, LEAPS II may want to consider increasing the frequency of the existing TOTs for government officials to enhance their capacity to deliver teacher trainings on a regular basis so that teachers can fully grasp the concepts. As well, such refresher sessions should supply teachers with handouts that they can use as reference whenever needed. Besides increasing knowledge of teaching concepts, given the challenge as reported by PAs and DESB of sufficient budget for monitoring visits, and the importance of such follow-up support to teachers, CRS could consider whether financial management trainings for DESB may be beneficial. Additionally, LEAPS II could collaborate with DESB and the schools to develop more appropriate and feasible work plans with achievable targets for monitoring taking into account the phase out of support. ▪ Reinforce engagement with VEDCs to boost community ownership of project activities. To ensure sustainability, LEAPS II must continue to engage VEDCs and build their capacity. One avenue is collaboration with DESB and other government entities to enforce VEDC fulfillment of roles and responsibilities because of the power of such directives in the Lao context. Another pathway is for LEAPS II to continue to engage with VEDCs to provide consistent and regular hands￾on learning on management of project activities. This training could help VEDCs increase their involvement to ensure successful handover of LEAPS II activities. An examination of similar school meal activities has shown that incremental involvement of local actors helps to sustain such initiatives.3 ▪ Ensure the effectiveness of any Operations and Maintenance (O&M) trainings so that communities understand how to maintain and manage essential WASH facilities for improved health and hygiene practices. As planned, CRS has drilled boreholes in select target communities (see 1.1 Overview of Program Implementation for more details on the WASH component of LEAPS II). Mothers in one such community said they were happy with the effort to provide water at their children’s school. However, many schools in the LEAPS II districts still do not have ready access to water, thus limiting improvements to overall health and hygiene of students. Any construction completed should be accompanied by regular support to help the communities maintain and repair the facilities.4 In each community, LEAPS II should identify who the relevant individuals or groups may be (usually the VEDC, the school principal, or a teacher) and target its trainings appropriately. CRS should clarify after its trainings the ways in which schools can seek help if they themselves cannot repair essential facilities. As well, the project should also establish an efficient system for monitoring both the WASH equipment and how well the beneficiaries have absorbed the lessons on O&M. Broadly, CRS could consider conducting a follow up qualitative study to its 2017 WASH in Schools: Behavior and Sustainability in Lao PDR report, which could examine issues such as maintenance of facilities, hygiene practices, and any improvements to the availability of water sources. 3 The Partnership for Child Development. 2016. Global school feeding sourcebook: lessons from 14 countries. Leslie Drake ed. Imperial College Press, London. 4 UNICEF. 2015. Evaluation of the WASH Programme of the UNICEF Country Programme (2012-2014). LEAPS II Midterm Performance Evaluation Report xiii  February 2020 ▪ Raise parent and community awareness of the importance of WASH practices in parent engagement workshops and school open houses. Besides infrastructure issues, long-established habits have hampered hygiene improvements. The project’s focus on school-level changes has not yet addressed practices at home and in the community. According to the qualitative findings, some students (like their parents) use outdoor areas as toilets because they lack latrines at home. LEAPS II could seek to align its WASH component with other donor projects that concentrate on community-wide activities so that students receive the same messages about health and hygiene at home and at school. LEAPS II could consider incorporating some key WASH messages into a presentation at school during open houses or in trainings where the project engages directly with parents, such as the SCI led parent engagement workshops. Since some of the solutions may require long-term actions, for any future iterations of LEAPS II, the program could consider the following alterations: ▪ Explore additional sustainable pathways to supplementing and diversifying school meals. Though LEAPS II has been successful in providing school meals, stakeholders expressed concern about sustainability. Government and VEDC respondents stated that they lack the financial and technical capacity to sustain school meals, and the primary challenge for VEDC remains mobilizing poor households to contribute regularly to the school meals. Beyond scaling up existing efforts to develop school and community income-generating activities, such as school gardens, fish ponds, or chicken coops, to help schools source ingredients and generate income, LEAPS II could consider in the long run various pathways to address poverty in the community. For example, if CRS could expand its scope to conduct activities beyond the school, then LEAPS II could establish Saving and Internal Lending Communities (SILCs). IMPAQ evaluations of MGD projects in Mali and Burkina Faso have shown that SILCs can be helpful to mitigate poverty by raising household incomes; thus, they can also be used as a conduit for community contributions to school expenses.5 ▪ Create custom indicators for reading comprehension. Given the complexity of the Lao language, gains in reading with comprehension in Lao PDR typically begins in Grade 3, but it is only in Grade 5 that the majority of students are able to read with fluency and comprehension.6 Our findings also confirmed the evidence from the existing literature showing students in Grade 2 have low levels of reading with comprehension, especially among non-Lao students. Thus, to get a more accurate picture of the effect of project interventions on students’ reading ability, we classified our recommendations into three parts for LEAPS II and any future studies of this project. Quick Win adjustments are addressable immediately within the current budget and data. LEAPS II could create nuanced indicators for different levels of reading comprehension, for example, reading with high comprehension, with some comprehension, and with no comprehension. The IMPAQ team could also consider the revised GoL definitions of reading competencies based on the forthcoming changes to the Grade 1 curriculum, if those benchmarks are available before the final evaluation. Medium effort changes require more effort or cost than quick win but are important to consider for bridge or future projects. Before the end of LEAPS II, the program could include a new pilot study to assess the existing reading competencies, but for students in Grade 3. With a modified budget and SOW, IMPAQ could also support LEAPS II in conducting this pilot study and expand 5 IMPAQ International. 2017. McGovern-Dole International Food For Education and Child Nutrition Program: Beoog Biiga II Midline Performance Evaluation Report. 6 The World Bank. May 2016. Reducing early grade dropout and low learning achievement in Lao PDR. Retrieved from: http://documents.worldbank.org/curated/en/161641483590757065/pdf/111627-REVISED-PUBLIC-Lao-PDR-Root￾Causes-of-Early-School-Leaving-f.pdf xiv IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report the assessment to other grades using the Annual Status of Education Report (ASER). IMPAQ can develop one ASER tool to use consistently for all grades and assess students’ reading levels. IMPAQ has successfully developed and administered this tool in other MGD programs for multiple grades in Mali, Senegal, Ivory Coast, and Burkina Faso as an alternative to EGRA or LBRA. Long-term strategies to help future studies beyond LEAPS II may require more time, effort, and cost to implement. To capture all the anticipated GoL revisions to the national curriculum, CRS could expand the pilot study described above with Grade 3 students and include other grades to the extent possible. In addition, if LEAPS II enters a third phase, CRS could consider an impact evaluation to study the effect of new curriculum revisions on reading competencies. Below, IMPAQ provides two additional recommendations that would require additional funds for the evaluation: ▪ Explore options to gain a stronger understanding of VEDC capacity and operations. Because VEDCs are important in facilitating community-level changes, their commitment is vital to sustainability. Although CRS’ annual VEDC rating tool provides some indication of VEDC success, a VEDC survey could reveal more, not just about how the VEDC performs, but also about the backgrounds of individual members, personal and group priorities and motivations, and constraints on the VEDC’s ability to muster community support for schools. Alternatively, IMPAQ could conduct in-depth qualitative interviews at endline with a greater number of VEDCs using the VEDC rating tool to sample best, average, and worst VEDCs, which could then shed light on the factors for success. Although CRS could conduct separately its own study of VEDC operations, an external evaluator who does not represent LEAPS II may be able to elicit more open dialogue with the VEDC members. ▪ Observe and complete a checklist to determine the functionality of school canteens and WASH equipment (handwashing stations and latrines). Self-reported data are often susceptible to social desirability bias, and thus direct observations would be useful to provide reliable information on food storage facilities and the presence of handwashing stations and latrines. For example, a large percentage of students reported that cooks had not served meals by early afternoon. Collecting additional data would reveal whether this might be happening in schools without functional canteens, with low quantities of food, or with broken handwashing stations. Having this “checklist” to assess the status of certain facilities may be helpful in complement to our qualitative and quantitative data on school meals and health and hygiene. Depending on the timing of the final evaluation and the dates of data collection, IMPAQ could observe meal preparation if school canteens are operational. LEAPS II Midterm Performance Evaluation Report  February 2020 1 SECTION 1. INTRODUCTION The United States Department of Agriculture (USDA) has funded Catholic Relief Services (CRS) for two phases of the Learning and Engaging All in Primary School (LEAPS) project as part of the McGovern-Dole (MGD) International Food for Education and Child Nutrition Program. The second phase, a five-year intervention (FY2016–2021) with a budget of $27 million, extends and expands on LEAPS I. This midterm performance evaluation assesses the progress of LEAPS II in achieving the desired project outcomes midway through implementation. We also recommend mid-course corrections based on lessons learned. This section provides a brief overview of the program context. Thereafter, Section 2. Evaluation Approach outlines the evaluation approach, including research questions, sampling design, data sources, and data analysis. Section 3. Fieldwork describes the data collection fieldwork, including human subject protection procedures. Section 4. Evaluation Sample describes the samples and their key characteristics. Section 5. Quantitative Results and Section 6. Qualitative Results present the quantitative and qualitative outcomes, respectively. Finally, Section 7 concludes with lessons learned, study limitations, and recommendations. 1.1 Overview of Program Implementation In response to low school attendance and performance in six educationally disadvantaged areas of Savannakhet province in Lao PDR,7 CRS implemented LEAPS I, in collaboration with local partners and the Ministry of Education and Sports (MoES), from 2012 to 2016. Supported by USDA, LEAPS I reached more than 36,000 students in 310 communities. It improved attendance and enhanced students' reading skills through approaches that empowered government, parents, teachers, and community leaders.8 To build on previous success and expand on LEAPS I, in September 2016, USDA awarded CRS $27 million to implement the second phase of the LEAPS program. CRS is leading the implementation of LEAPS II in partnership with the government, Save the Children International (SCI), and University of Oregon (UO). LEAPS II aims to improve the literacy skills, as well as health and dietary practices, of approximately 86,049 new and continuing students in 350 schools across the six districts of LEAPS I, and an additional district, Nong.9 Exhibit 1 provides a map of project areas in both phases. 7 LEAPS I did not include Nong district. 8 Learning and Engaging All in Primary School (LEAPS). 2016. "Final Evaluation Report." 9 At the end of LEAPS I, there were 309 schools in total as one school had dropped out from the original 310 schools. At the start of LEAPS II, two LEAPS I schools split to create a total of four schools, for a new total of 311 schools from LEAPS I. Then, CRS selected 50 schools in the newly added district of Nong, but four did not enroll for various reasons (e.g., they declined to participate or DESB recommended that they should not be included). After LEAPS II started, seven schools dropped out of the program leaving only 350 schools. Exhibit 1. LEAPS II Targeted Districts in Savannakhet Province 2 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Although LEAPS II retains some of the same elements as the previous phase, such as school meals, take￾home rations (THRs), and inclusive education (IE), LEAPS II has incorporated lessons learned and identified needs from LEAPS I to add several new activities. According to the LEAPS II theory of change (see Appendix B), providing school meals can both enhance attendance by keeping students in school for the full day and increase students’ attentiveness by mitigating short-term hunger. These two factors, in complement with the Literacy Boost (LB), should improve students’ literacy skills, in keeping with MGD Objective 1. In addition, the LEAPS II theory of change suggests that providing food preparers at school with training on safe food preparation and storage practices; improving teachers’ knowledge of health and hygiene practices to transfer their knowledge to students; and building/rehabilitating wells and water stations at schools will help to improve health and dietary practices (MGD Objective 2). The following paragraphs elaborate on the LEAPS II activities of most relevance to our evaluation keeping in mind the MGD objectives, including the foundational results,10 and the findings from our data. 1.1.1 Provision of School Meals Through LEAPS I, CRS provided nearly 12 million meals to more than 36,000 children. This support was instrumental in supporting students’ attendance and attentiveness improvement in the classroom.11 LEAPS II expands upon this intervention and will provide throughout the duration of the intervention more than 32 million daily meals, with a daily ration of 100 grams of rice, 35 grams of lentils, and 10 grams of vegetable oil per student.12 Communities contribute to the school meals either through cash contributions so that schools can purchase additional ingredients, or directly via the provision of food items, such as eggs and condiments. Village Education Development Committees (VEDCs) mobilize this community support, and CRS provides training to VEDC members on how to record and track such contributions as part of its modules on VEDC roles and responsibilities. This kind of capacity building contributes to the foundational result of increased engagement of local organizations and groups. Cooks, storekeepers, and teachers also receive a monthly take-home ration (THR), but to move toward sustainability of operations, CRS intends to phase down the rice in the bundle of THRs for those communities with high-performing VEDCs, based on the results from the annual VEDC rating tool. Some other activities that take place as part of the school meals component of LEAPS II include: ▪ Rehabilitation and maintenance of storehouses constructed under LEAPS I, and provision of materials (e.g. roofing sheets, nails, and cement) to support the community to build new warehouses in Nong ▪ Establishment of school gardens and subsequent training of teachers and principals to use these gardens as a tool for working with children and to improve sustainability of school meals ▪ Refresher trainings for cooks and storekeepers on safe food preparation and hygiene practices 1.1.2 Literacy Interventions CRS and SCI work jointly on this component of LEAPS II to improve literacy outcomes. SCI leads the implementation of their signature Literacy Boost intervention, which aims broadly to improve the reading 10 Please reference Section 6.4 Sustainability for more information on the LEAPS II policy development work with WFP. 11 CRS. 2016. Learning and Engaging All in Primary School (LEAPS): Final Evaluation Report. 12 CRS. 2018. LEAPS II FAQs. LEAPS II Midterm Performance Evaluation Report  February 2020 3 skills of children in the first five years of primary school. The LB intervention contains the following components: ▪ The LB reading assessment (LBRA). The LB practitioners13 assess children’s reading levels and evaluate their literacy learning needs based on those assessment. ▪ Training. The LB training centers on five core skills of reading acquisition: 1) automaticity in letter recognition, or letter knowledge; 2) phonemic awareness; 3) fluency (reasonable reading speed) and accuracy; 4) vocabulary development; and 5) reading comprehension, which is the ultimate goal of language learning, including literal, inferential, and evaluative. In addition to these five skills, a key component of the intervention is training teachers on important instructional techniques paired with other essential factors, such as children’s motivation, access to text, home environment, and reading opportunities. SCI and CRS work collaboratively to train MoES and school administrators, and then co￾facilitate with the government the roll down of such trainings to the teachers, which encourages sustainability and responds to the different objectives related to increased capacity of government institutions and increased government support, as laid out in the foundational results. ▪ Provision of reading materials. Room to Read as part of LEAPS I established school libraries, and LEAPS II enhances the availability of reading materials by setting up reading corners in Grade 1, 2, and 3 classrooms. To facilitate the summer LB activities, community literacy volunteers (CLVs) take ownership of the reading corners and transfer the books from the school to the community. At this time, books may be housed in the home of the village head or CLV, or in a public space. ▪ Community action. Quality teaching and learning environments inside and outside of schools help all readers grow their reading skills. The LB program guides parents and communities to support children as they learn to read and foster their love of reading, including encouraging reading at home and holding school open houses. SCI conducts parent engagement workshops/trainings to encourage parents to create home literacy environments. As well, CRS and SCI staff work with community literacy volunteers to implement summer reading camps that aim to further develop the literacy skills of children; these camps have now been extended to the entirety of the academic year in LB schools. The LB intervention takes place in half of the schools receiving the SF intervention, with three cohorts of 60 schools each, for a total of 180 schools in 6 districts of LEAPS II.14 Each cohort will receive project support for Grades 1 and 2. Complementing the LB approach, CRS has conducted two special studies to enhance literacy activities: 1) summer school readiness camps and research on effective teaching methods for minority language speakers. These camps provided literacy instruction to children outside of the school year to help them reach grade-level literacy in the upcoming year. CRS holds week-long trainings for CLVs to help run these camps. Finally, CRS trained teachers to engage in ‘learning circles’–meetings where teachers can reflect on the successes of and challenges encountered in using LB techniques–to foster a culture of continual learning and adaptation. 1.1.3 Inclusive Education (IE) Working in partnership with UO since LEAPS I, CRS ensures that its IE approach falls in line with MoES IE policies to ensure all children have access to school regardless of disability, gender, or any other circumstance. In LEAPS I, UO collaborated with MoES to develop an IE curriculum for MoES to implement through the Teachers’ Training College, which would train all new teachers in IE practices to match the 13 SCI staff trained PAs. 14 The LB intervention did not target schools in Nong (the LEAPS II expansion district). 4 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report new IE laws adopted by the Government of Lao PDR (GoL) in 2011. In the design of LEAPS II, CRS and UO jointly developed an online curriculum and Training-of-Trainers (TOT) model to engage a diverse group of stakeholders in IE practices. The following IE trainings were conducted during LEAPS II: ▪ Training school administrators to understand IE policy, IE management, IE concepts of differentiated instruction, and how to create a welcoming learning environment, and then supporting them in the roll-down of this IE training to teachers, which includes coaching teachers on working with students who have different disabilities and modifying teaching practices and learning resources to accommodate these students ▪ Training PESS and DESB on community-based rehabilitation (CBR), which aims to improve awareness of disabilities in the community, and then working with the government to flow down these lessons to the VEDCs ▪ Supporting VEDCs in their community mapping exercises to ensure that school management focuses on students with disability ▪ Training DESB how to enroll disabled children into school ▪ UO online IE trainings on classroom management techniques for inclusion of disabled children given to high-performing teachers 1.1.4 Water, Sanitation, and Hygiene (WASH) A new component of LEAPS in its second iteration, the WASH pilot in 65 schools will build or rehabilitate school wells and latrines to increase access to water – a resource that is a critical component of a school meals program. CRS works with Water User Committees (WUCs) to provide trainings and refreshers on roles and responsibilities and how to collect fees for WASH infrastructure management. As well, these WUCs, in addition to schools in the pilot, receive operations and maintenance (O&M) trainings on installed wells and water points. As of this midterm evaluation, CRS plans to expand the WASH pilot to an additional 25 schools and provide water filters to 253 schools. Other relevant activities under the WASH pilot include: ▪ Provision of toothbrushes, toothpaste, and soap for schools ▪ Health and hygiene trainings utilizing the Fit for Schools model developed by GIZ and adopted by the MoES, which focuses on tooth brushing, handwashing, and latrine usage 1.2 Overview of Evaluation Background CRS selected IMPAQ to design and conduct both performance and impact evaluations of LEAPS II. The performance evaluation (the focus of this report) seeks to assess the progress of LEAPS II in achieving its core objectives (SO1 and SO2) at midterm (2019) and endline (2021), using benchmark values collected at baseline (2017). The quantitative approach mainly focuses on improved literacy skills of school-aged children (SO1), while the qualitative component addresses not only SO1 but also SO2, increased use of health and dietary practices. This report presents performance evaluation results at midterm and recommends mid-course corrections. More specifically, this midterm performance evaluation has the following objectives: ▪ Critically and objectively take stock of the project’s implementation experience and environment ▪ Assess whether target beneficiaries are receiving services as expected ▪ Assess whether the project is on track to meet its stated goals and objectives ▪ Review the project-level results framework and assumptions ▪ Document initial lessons learned LEAPS II Midterm Performance Evaluation Report  February 2020 5 ▪ Discuss mid-course corrections and provide recommendations that may be necessary to effectively and efficiently meet the project’s stated goals and objectives To address these objectives at midterm, IMPAQ mirrored our approach to baseline, and incorporated similar survey instruments to collect quantitative data for the same performance indicators. The evaluation team also observed classrooms in all sampled schools for the performance evaluation to measure students’ attentiveness using the same tool as at baseline. In addition, IMPAQ built upon qualitative protocols already developed at baseline to capture midterm research questions, which helped to contextualize the quantitative results and provide valuable lessons learned. 6 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report SECTION 2. EVALUATION APPROACH To assess progress in LEAPS II implementation, IMPAQ used a mixed-methods approach, including classroom observation. This section describes that approach, focusing on the research questions, quantitative and qualitative evaluation methodology, and data sources. 2.1 Research Questions Following our performance evaluation plan for LEAPS II at baseline and the midterm terms of reference (TOR), IMPAQ assessed five dimensions of project achievements: relevance, effectiveness, efficiency, sustainability, and impact. Appendix B. LEAPS II Evaluation Questions and Conceptual Framework provides a comprehensive conceptual framework mapping the research questions to data analysis methods. 2.2 Evaluation Methodology IMPAQ designed the quantitative methodology of the midterm performance evaluation to measure progress toward the key LEAPS II performance indicators. For the qualitative methodology, IMPAQ built upon baseline protocols to capture the perceptions of LEAPS II beneficiaries and stakeholders on issues related to the midterm research questions. The evaluation team triangulated qualitative findings with survey and classroom observation results, as well as explored CRS monitoring data (e.g., WASH, student attendance, etc.), where relevant. Usage of all these data helped provide contextual information for the quantitative analysis, explore lessons learned and best practices for mid-course corrections, and provide recommendations for program sustainability. 2.2.1 Quantitative Sampling Design To reflect accurately program performance from baseline to midterm, IMPAQ measured the program indicators outlined in Appendix B. LEAPS II Evaluation Questions and Conceptual Framework, using the same methodology and sampling strategy as at baseline. At baseline, IMPAQ applied two sampling schemes for two distinct target populations. First, IMPAQ sampled students in Grades 1 through 5 in 380 schools initially considered to be part of LEAPS II in all seven districts of Savannakhet. These schools received the school feeding (SF) component of LEAPS II. Second, IMPAQ sampled Grade 2 students in a subset of 186 schools that received, in addition to SF, the LB intervention in six districts of Savannakhet.15 Following CRS' initial calculations for power and the Food and Nutrition Technical Assistance sampling guidance, IMPAQ determined sufficient sample sizes as 1,525 students in Grades 1 through 5 in 61 SF schools and 490 students in Grade 2 in 49 LB schools. 16 The other parameters in the power calculation include an alpha of 0.05 (for a 95 percent confidence level), a desired power of 80 percent, and a contingency factor of 10 percent for non-response. 15 In January 2017, IMPAQ received a database of 380 schools: 194 SF-only schools and 186 LB schools. Eventually, CRS selected 350 of these schools for implementation; 170 SF-only schools and 180 LB schools. 16 The design effect accounts for use of cluster sampling instead of simple random sampling. The sample size for the student survey is computed using a default design effect of 2. The expected design effect for the LB assessment sample is set to 3.15 because the terms of reference specified this value based on SCI’s experience with the LB assessment in Lao PDR. LEAPS II Midterm Performance Evaluation Report  February 2020 7 For the performance evaluation, IMPAQ used a two-stage sampling approach because students were nested within schools. First, IMPAQ drew the schools to be sampled; second, IMPAQ drew students to be surveyed within each selected school. In the first stage, IMPAQ selected the schools with a probability-proportional-to-size clustered sampling approach, using the total number of students per school as a school size measure. The procedure for selecting the schools was systematic-random sampling, which involves choosing schools from a list using a sampling interval. Starting from the school on the top of the list, IMPAQ selected subsequent schools by adding the sampling interval to the cumulative measure of school size. To minimize the travel costs and the time necessary to cover all selected schools, IMPAQ developed an algorithm to maximize the rate of overlap between the SF and LB samples. The potential rate of overlap ranges from 0 percent (when none of the 49 LB schools are among the 61 SF schools) and 100 percent (when all 49 LB schools are included among the 61 SF schools). As the systematic-random sampling method is sensitive to the order in which schools are listed, IMPAQ performed more than 5,000 simulations with different orders to retain the iteration that had the highest rate of overlap. This process ensured that the final sample would be representative of all 380 schools and of the 186 LB schools while keeping data collection cost-effective. As shown in Exhibit 3, the schools in the final sample fall into three groups: SF-only schools for the full sample, SF + LB schools to be included in both samples, and SF + LB schools for the LB sample. After this simulation process, the final sample at baseline included 87 schools. However, four schools in Nong dropped out17 of the program, reducing the total to 83 schools: 34 SF-only schools (444 boys and 433 girls surveyed), 23 schools that are part of both samples (353 boys and 365 girls surveyed), and 26 schools for the LB sample only (127 boys and 128 girls surveyed). In the second stage, at each sampled school, IMPAQ selected a sample of students by physically lining up boys and girls separately for each grade. To identify the n th student for random selection, IMPAQ used a simple rule: 𝑛 𝑡ℎ𝑔𝑖𝑟𝑙 𝑜𝑟 𝑏𝑜𝑦 𝑡𝑜 𝑠𝑎𝑚𝑝𝑙𝑒 = 𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑔𝑖𝑟𝑙𝑠 𝑜𝑟 𝑏𝑜𝑦𝑠 𝑖𝑛 𝑒𝑎𝑐ℎ 𝑔𝑟𝑎𝑑𝑒 𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑔𝑖𝑟𝑙𝑠 𝑜𝑟 𝑏𝑜𝑦𝑠 𝑡𝑜 𝑏𝑒 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 For example, if IMPAQ had 10 female second graders and we required five for the study (as in SF + LB schools for the LB sample), then enumerators selected every other student from the line of second-grade girls (10 ÷ 5 = 2). IMPAQ applied the same rule to select students systematically from all sampled schools and grades by gender, as outlined in Exhibit 2. In the absence of electronic class lists, this approach ensured sampling consistency across schools and achieved a random sample of students. However, the possibility of systematic absences might induce a risk of sampling bias by selecting only students present 17 We did not replace those schools that dropped out in the midterm evaluation because we did not have their baseline values. Exhibit 2. Sampling Strategy 8 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report on the day of data collection. Bias can arise, for example, by excluding children who are likely to be absent due to health issues. Although this is one of the limitations of this evaluation, random sampling of different students at each point in time (baseline, midterm, and final evaluation) mitigates this bias by selecting a new representative sample of students each round. 2.2.2 Qualitative Sampling Design To provide context for the quantitative results, our qualitative study sought to dig deeper into the research questions. Focusing on seven schools—one in each of the seven LEAPS II districts—IMPAQ conducted in-depth key informant interviews (KIIs) and focus group discussions (FGDs) with respondents. The following table provides an overview of the interview type, gender breakdown, and number of groups and individuals. See Appendix D. List of Stakeholders for a comprehensive list of stakeholders and details on those interviews. Exhibit 3. Respondents by Type and Gender Interview Type Number of Groups Men Women Total Number of Individuals Mothers FGD 7 N/A 41 41 Fathers FGD 7 33 N/A 33 VEDC FGD 7 26 2 28 Teachers FGD 7 3 10 13 Cooks FGD 7* 0 15 15 Students FGD 7 26 26 52 Principal KII 7 3 4 7 PA KII 4 4 0 4 DESB KII 7 7 0 7 PESS KII 1 0 1 1 CLV KII 4 3 1 4 Total 104 100 204 *In one district, the cooks were not preparing school meals because of a broken water pump, and the evaluation team could not locate them because they were away on holiday for the Lao New Year. Thus, the FGD in this case became a KII with one of the cooks via phone about a week after the initial visit. To obtain information at a broader level and understand the context of LEAPS II implementation, IMPAQ spoke to several members of the CRS staff: the current and former Chiefs of Party (COP), the Deputy Chief of Party (DCOP), the former Monitoring, Evaluation, and Learning (MEAL) manager, the commodities manager, the finance manager, and community mobilizers (CMs). IMPAQ also conducted KIIs with CRS partners, including SCI (MEAL manager, Regional Education Advisor, and Technical Program Manager), World Food Programme (WFP) (Head of Programs and Government Relations Advisor), and UO (Program Manager). Three USDA staff members also provided valuable insight into the desired outcomes for this evaluation and programming priorities for the second half of LEAPS II. Where relevant, Section 6. Qualitative Results also includes perspectives from CRS, partner staff, and USDA in the presentation of qualitative findings. To clarify the references to school and community stakeholders in Section 6. Qualitative Results, the table below illustrates how these two groups have been categorized. Exhibit 4. Categorization of School and Community Stakeholders Level Stakeholders School FGDs with teachers, students, parents, and cooks KIIs with principals Community FGDs with Village Education Development Committee (VEDC) members LEAPS II Midterm Performance Evaluation Report  February 2020 9 Level Stakeholders KIIs with the district education and sports bureau (DESB) representatives (technicians), pedagogical advisors (PAs), and community literacy volunteers (CLVs) Identification of Schools To select schools for the qualitative sample, IMPAQ relied on purposive sampling from a list of high-, average, and low-performing schools. For these rankings, IMPAQ considered attentiveness, attendance, and cooking rates, drawing on CRS’ monitoring data and our own baseline indicators. Exhibit 5 summarizes the statistics (or calculations if additional manipulation of data occurred) for the three required criteria. Exhibit 5. Determination of High-Performing and Low-Performing Schools Criterion Statistic or Calculation Used Source Cooking rates Average cooking rate for each month in the 2017–2018 school year CRS monitoring data Attendance Obtained attendance data from CRS for 2017-2018 school year Created threshold of passing if average attendance (days in school / total days in school) > 80% Calculated % of students in each school that passed CRS monitoring data Attentiveness Baseline indicators for attentiveness at each school IMPAQ baseline data After obtaining these figures for each school in the evaluation sample, IMPAQ examined the noticeable outliers, high data points, and low data points to narrow down the list to 15 high-, average-, and low￾performing schools in the seven districts. To benchmark cooking rates, IMPAQ used CRS’ threshold for “regular” cooking: that is, schools having cooked for at least 84 percent of school days. For attentiveness, the evaluation team scored schools as “high” if the baseline percentage exceeded 80 percent. IMPAQ could not easily categorize some schools as high- or low-performing because they scored high marks in some but not all criteria. As such, IMPAQ considered these schools average-performing; for example, one school had a low cooking rate (54 percent) but high student attendance (97 percent) and attentiveness (92 percent). To whittle down the 15 schools to seven, IMPAQ consulted with CRS to determine which schools had WASH and LB activities to ensure our sample of qualitative schools would reflect the entirety of the various LEAPS II interventions. IMPAQ also used baseline survey data to identify the proportion of students at each school who named Lao as their main language so that sampled schools would reflect differing levels of Lao language dominance. Finally, IMPAQ also took into account the logistics of coordinating qualitative fieldwork with quantitative data collection. At the end of this process, IMPAQ developed a sample with three high-performing schools (best), two average schools, and two low-performing schools (worst). Identification of Individuals To recruit individuals connected to the seven sampled schools, the evaluation team relied on community leaders. The district official responsible for the school visit contacted the school principal one or two days in advance of the field visit. The principal then coordinated the attendance of relevant individuals, drawing upon the village chief or head as needed. Participants self-selected into the KIIs and FGDs based on their availability and willingness to become involved. Exhibit 6 shows school and community participants by gender. Exhibit 31 in Appendix D. List of Stakeholders provides more detail. Although the qualitative team attempted to obtain enough parents, teachers, students, VEDC members, and cooks for FGDs, in some cases we could reach only one or two individuals. This challenge was particularly acute with teachers because some schools employed only a single teacher. 10 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Some crossover in membership occurred among the focus groups. In one school, a few cooks participated in the mothers FGD as well. Occasionally fathers overlapped with VEDC FGD participants. VEDC members primarily consisted of men; female VEDC members often represented the Lao Women’s Union. 2.3 Data Sources To answer the research questions for the midterm performance evaluation and to determine values for performance indicators, IMPAQ collected and analyzed survey data from two sources: a student survey that was administered to Grade 1-5 students in all the 83 performance schools; the LB Reading Assessment (LBRA) with Grade 2 students in the sampled schools with LB interventions (a total of 49) ; and a classroom observation tool, which was conducted in one classroom (Grade1-5) per each performance schools. The evaluation team translated the instruments and adapted them to the Lao context through cognitive interviews18 at baseline, and we did not significantly change them at midterm. 19 This consistency enabled us to capture changes from baseline to midterm in project relevance, effectiveness, and efficiency and to assess early indications of sustainability and impact. IMPAQ used the quantitative data to measure the program’s progress toward its objectives. The qualitative protocols for KIIs and FGDs focused its questions on the relevance, effectiveness, efficiency, sustainability, and impact of project interventions. See Appendix I. Survey Instruments for the evaluation instruments and Appendix J for the qualitative protocols. 2.3.1 Student Survey and LBRA The student survey collected data on the background, food security, health, and school and household environments of sampled students in Grades 1 through 5. IMPAQ adapted survey questions to the Lao context through cognitive interviews prior to data collection at baseline in March 2017. During cognitive testing, interviewers discussed the meaning of each item with students to assess the clarity of the question and appropriateness of the proposed categories. In collaboration with CRS, IMPAQ conducted the cognitive interviews in school areas where students’ main language at home was Lao and non-Lao to identify, in the local context and in different languages, what works, what does not work, and why. To measure the reading skills of students at the end of Grade 2, enumerators also administered the LBRA as part of the student survey, but only to second graders. SCI developed and adapted the LBRA to the Lao curriculum in consultation with MoES before the start of LEAPS II. The GoL has defined Grade 2 reading competencies20 as follows: ▪ Show knowledge of the objectives and content of the text ▪ Read long text such as letters, notification, reports and explanation ▪ Use information from the text in different context and objectives ▪ Identify different structure and characteristics of the text such as: heading, topic, paragraphs, section and punctuations Considering these benchmarks for reading competencies, SCI, in collaboration with CRS and IMPAQ, field tested the LBRA before baseline data collection in February 2017. During this process, IMPAQ also 18 A cognitive interview is an individual, face-to-face, in-depth interview that aims to understand how a respondent comprehends and responds to questions. 19 We added questions to capture when students started in their school and which grades they had repeated. 20 However, given the complexity of the Lao language, the government is developing new competencies to match the new curricula and match literacy expectations to the learning realities in the classroom. LEAPS II Midterm Performance Evaluation Report  February 2020 11 conducted a “passage equating” exercise with four passages from story books considered appropriate for Grade 2 level21 in Lao. IMPAQ administered this field test to strong readers in Grade 3 — the best benchmark for the reading proficiency of students at the end of Grade 2 — in two schools with Lao and non-Lao skills. After analyzing the test data, the evaluation team collaborated with SCI to select the most appropriate passage to assess second-grade students’ literacy. IMPAQ repeated this process before collecting data at baseline for the impact (September 2017) and at midterm for the performance and impact (March 2019) evaluations. 2.3.2 Classroom Observations To measure student attentiveness, IMPAQ used a time-sampling technique based on the Stallings “snapshot” method. 22 The evaluation team created a tool to measure the percentage of attentive students and to capture potential factors of attentiveness (other than school meals) such as class size, subject, class arrangement, and activities. Every three minutes, enumerators documented the classroom activity by recording what both teachers and students were doing. Then they went clockwise around the room to count distracted students, 23 disaggregated by gender. Each observation lasted 35 minutes to allow completion of 10 snapshots. To determine the percentage of attentive students, IMPAQ divided the number of attentive students by the total number of present students on the day of school visit for all 10 snapshots. 2.3.3 Qualitative Protocols To obtain information for the qualitative analysis, IMPAQ developed semi-structured KII and FGD tools adapted to the stakeholders in the evaluation (see Section 2.2.2). Building on the baseline protocols, the evaluation team added, deleted, and reworded questions to match the objectives of the midterm evaluation. IMPAQ also considered unanticipated findings from baseline and altered the protocols to shed light on these results. For example, noting that, at baseline, students often reported missing school due to illness, IMPAQ included a question for principals, teachers, and parents about how often students are absent from school because of sickness and whether absence due to illness represents an issue of concern. All protocols contained consent and assent language, which our field interviewers delivered to respondents. Please see 3.1 Human Subject Protection for more information. 2.4 Data Analysis IMPAQ started our data analysis with an exhaustive assessment of quality for both quantitative and qualitative data before proceeding with data cleaning and analysis. 2.4.1 Quantitative Data Analysis After transferring the downloaded data from our programming server, Kobo, to a secure platform for data cleaning and analysis, IMPAQ conducted a review of survey data to check for data completeness, duplicate entries, fidelity to skip pattern logic, and data cleaning. The evaluation team then used Stata to analyze the cleaned data descriptively by constructing means and percentages using data from individual or 21 The SCI LB coordinator and advisor in 2015 conducted this process for another evaluation of LB in Bolikhamxay in Lao PDR. 22 Stallings, J. and G. Mohlman. 1988. “Classroom Observation Techniques.” In Educational Research, Methodology, and Measurement: An International Handbook. Edited by J. Keeves. Pergamon: Oxford. 23 We considered a student distracted if s/he was chatting (unrelated to the class), sleeping, looking out the window, going out of class, fighting, or playing with his/her hair or clothes without paying attention to the class activities. 12 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report multiple survey items. IMPAQ followed the same calculations used in the baseline and requested in the USDA-approved performance management plan. This process helped generate an appropriate comparison between baseline and midterm outcomes. To measure the progress of LEAPS II toward its core objectives, IMPAQ conducted descriptive analyses comparing mean outcomes at baseline with those at midterm by running t-tests and calculating p-values. Where feasible, the evaluation team also outlined subgroup analyses by student gender, grade level, and district. For the literacy outcomes, we also looked at the data by students’ main language spoken at home (Lao vs. non-Lao). For attentiveness, we checked the results by various groups, including class type (multi￾grade vs. single grade), school type (SF only vs. SF+LB schools), teaching subject, class size (defined as large if above average), and class arrangement. For each subgroup analysis done, we only highlighted emerging patterns. In addition, the evaluation team used multivariate regression analyses to look deeper into the potential predictors of students’ literacy outcomes at midterm. We examined the relationship between students’ literacy skills and key demographic characteristics including students’ age, sex, main home language, participation in early childhood programming/preschool, and grade repetition. Other important factors examined include being in a multilingual household, availability of home reading materials, home literacy interactions, and socioeconomic status. In addition, IMPAQ checked whether these relationships changed depending on whether students were in SF + LB or SF schools, especially if the school was part of the Room-to-Read program before LEAPS II program. 24 2.4.2 Qualitative Data Analysis Our qualitative analysis aimed to understand stakeholders’ perception of the design and implementation of the program, their experience implementing or participating in components of the program, expectations for improved outcomes, and any other relevant contextual information. Considering the research questions, IMPAQ developed codes in line with the main points of inquiry in the areas of relevance, effectiveness, efficiency, sustainability, and impact, while also taking into account the LEAPS II results framework. After reviewing the notes from the field, the evaluation team tagged the responses to these codes using NVivo qualitative analysis software. Additionally, we set up our database so that we could analyze the results by best-worst case, based on the classification as discussed in 2.2.2 Qualitative Sampling Design IMPAQ utilized this approach to data analysis based on CRS’ request outlined in the midterm TOR. Following the data analysis, IMPAQ drafted short memos in response to each of the research questions, noting any recurring patterns, cross-cutting themes, and interesting divergences. Our approach ensured a systematic process to capture salient findings for each research domain and any similarities and differences that might be useful in complement to the quantitative results. Where findings seemed to require information beyond the data collected, IMPAQ examined available monitoring data and other reports from CRS. We also triangulated qualitative findings with survey data to provide contextual information for the quantitative analysis, where applicable. 24 In our performance evaluation sample, there were 12 schools that were previously in the Room to Read education program before LEAPS II started in four districts. Six of the schools only received a school library and six schools received a full package of literacy activities that included teacher training, school library, and in-class learning aids. LEAPS II Midterm Performance Evaluation Report  February 2020 13 SECTION 3. FIELDWORK This section describes the activities that the evaluation team conducted before, during, and after data collection, from human subject protection to field preparation, data collection, and quality assurance. It also discusses challenges the team faced during quantitative and qualitative data collection. 3.1 Human Subject Protection Prior to collecting data for the baseline evaluation, IMPAQ sought institutional review board (IRB) approval to ensure that the proposed evaluation complied with international rules and procedures. IMPAQ submitted protocol documents to Chesapeake IRB, now merged with Schulman IRB as Advarra, on February 28, 2017 (Pro00020840). In addition, for KIIs and FGDs added to the evaluation at midterm, IMPAQ submitted an amendment to the Advarra IRB and received its approval on March 4, 2019. The IRB submissions ensure that the research is ethically sound and safeguards the rights, safety, and well-being of children and other respondents. IMPAQ used the IRB-approved evaluation instruments, with informed consent and assent forms, to collect midterm data. Before administering the student survey, IMPAQ trained enumerators on procedures for interviewing respondents, protecting respondents’ privacy and confidentiality, and securing the data. IMPAQ also reprised the training from SCI on safeguarding children at school. During the data collection, the survey team first obtained written consent from teachers and/or principals to survey students. The team then asked for students’ verbal assent, assuring children that their participation was voluntary and that they could terminate the survey at any point. IMPAQ followed a similar procedure of seeking consent from adult participants when conducting the FGDs and KIIs (see Appendix J. Qualitative Protocols), and assent for students. After data collection, the evaluation team protected the privacy and confidentiality of respondents by storing the data on secure servers and separating personally identifiable information from the survey data. 3.2 Preparation for Data Collection From February 25 to March 1, 2019, the IMPAQ team trained 18 enumerators, recruited by our data collection partner SKO. All enumerators had extensive experience in collecting data from young children and in conducting reading assessments in LEAPS II project areas. The first two days of training, led by the IMPAQ project director, focused on classroom observations with seven of the most experienced enumerators. Enumerators learned how to conduct observations in real classrooms in the morning and discussed their observations in the afternoon. At the end of the second day of training, they worked to improve their inter-rater reliability (IRR) knowledge by watching videos from other classrooms and scoring them all together with the project director.25 The next two days of training involved all 18 enumerators and concentrated on administering the student survey and reading assessment. Enumerators learned the intention of each survey question, how to ask questions directed to vulnerable respondents (in this case, children under 18), how to assess students’ literacy, and how to use tablets to implement the in-person surveys offline without an internet connection. After the initial training, the IMPAQ team provided an opportunity for enumerators to 25 This incident happened in 10 schools across both impact and performance sample: one school in Nong, one school in Atsaphone, one school in Vilabouly, and seven schools in Sepon. 14 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report practice with real respondents for one day in two non-LEAPS schools in one target district whose student populations spoke Lao and minority languages. This pilot mimicked real data collection to give enumerators hands-on-practice. Afterward, enumerators regrouped with the IMPAQ team to debrief and discuss any issues they encountered. For the qualitative data collection, the IMPAQ team held an in-country training and discussion with the two local qualitative researchers, including the SKO team leader, who had been involved with the performance evaluation at baseline. Because the student FGD protocol was a new addition to the midterm evaluation and the respondents would be young (primary school age children), the team decided to pilot this protocol in advance of the fieldwork. The first test of the student FGD protocol revealed that the respondents seemed shy to speak freely and openly with the researchers. The IMPAQ team therefore designed a short interactive game for the researchers to play with students before starting the FGD. In a second pilot using this game, students seemed quite excited to play. Then, when the researcher did start asking questions, the children showed less wariness. The other qualitative protocols built on those used at baseline, which had already been field tested. However, CRS and SKO reviewed the KII and FGD guides to identify any wording issues and to ensure cultural appropriateness. 3.3 Field Activities The evaluation team started fieldwork on March 4, 2019. SKO organized the enumerators into two teams of four and two teams of five individuals. The SKO team leader followed the teams of enumerators to oversee data quality and provide technical support under IMPAQ supervision. The CRS community mobilizers and a Provincial Education and Sports Service (PESS) representative accompanied the fieldwork team to facilitate informal meetings between SKO and the District Education Sports Bureaus (DESBs) and to coordinate school visits. All enumerators regrouped with their supervisorsin the village every day to debrief, submit data collection logs, submit electronic surveys, and plan for the next days of data collection. The SKO team leader was responsible for regularly updating IMPAQ’s project director on challenges faced and decisions made. The team completed fieldwork in six weeks before the Lao New Year started on April 11. The qualitative researchers documented their progress daily—for example, the number of FGDs conducted and with whom—and noted any challenges encountered. Because they could not record interviews due to local sensitivities, the field researchers took detailed notes on paper and transcribed them electronically after conducting the FGDs and KIIs. The IMPAQ team communicated with the qualitative researchers daily to check on progress, determine whether any adaptations to the schedule would be needed, and discuss whether to probe certain topics further. At the end of each week, the qualitative team met to share insights from that week’s interviews. 3.4 Field Challenges During the six weeks of data collection, the qualitative and quantitative teams did not face major challenges that would affect the results and analysis or would delay the process. However, the evaluation team encountered a few complications in the field, especially regarding the availability of the respondents. LEAPS II Midterm Performance Evaluation Report  February 2020 15 Some schools, on the day of our visit, canceled target grades because of teacher absenteeism or traditional events in the village (e.g., village festival for the International Women’s Day on March 8th). 26 In those cases, enumerators asked other teachers or the school principal to call students to come back to school for the survey. In addition, some schools had cleaning events on Friday afternoons, so they would close the school after noon. The evaluation team had to coordinate with school principals ahead of time to ensure the availability of students for the survey. The availability of individuals for FGDs posed a challenge primarily when locating VEDC members and teachers. Gathering VEDC members presented some difficulties because of certain logistical challenges; for example, VEDC members often had prior commitments or lived too far from the school to travel there conveniently for the FGDs. The evaluation team also encountered some challenges achieving adequate gender representation. Only two VEDC FGDs included females — in each of those FGDs, only one female participated. Reports from the field suggest that female VEDC members could not participate in the FGDs because of agricultural activities. The two women who did participate represented the Lao Women’s Union in that village, but, despite some probing and direct questions from our male and female facilitators, they often felt shier to voice their opinions. Similar to the challenges faced communicating with women in the VEDC FGDs, our discussions with mothers also required significant probing and follow-up to elicit responses. Though our data collection partner hired translators in anticipation that some parents would have limited capacity to speak Lao, women in some ethnic communities still did not seem to feel comfortable to speak much or offer their opinions even when the translator was a woman. One of our qualitative researchers noted that women would comment on their poverty and illiteracy as reasons for not being able to think more broadly and discuss their children’s education. These obstacles with the Lao language were not unique to the qualitative interviews. IMPAQ designed the student survey in Lao to match the language of school instruction, but students in the sample spoke other languages, especially in Nong, Phalanxai, Phin, Sepon, and Vilabouly. Similar to baseline data collection practice, to facilitate the surveys with younger children, enumerators asked students in higher grades to interpret or translate any confusing/unfamiliar Lao terms or wording. Enumerators watched the translation carefully to ensure that the older children did not give the younger students answers. IMPAQ decided to involve older students in translation rather than teachers or principals because the educators might influence the children’s responses. Although IMPAQ faced these challenges in the field, as mentioned above, the field team followed different strategies to mitigate their effect on the results and analysis, including oversampling in larger schools, and hiring female and male facilitators for the FGDs. We discuss these challenges more in detail in the limitation of this study in 7.1 Limitations. 26 This incident happened in 4 schools in performance evaluation sample: one school in Nong because the teacher moved to China; two schools in Atsaphone due to village festivals and funeral ceremony; and one school in Vilabouly because of a village festival. 16 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report SECTION 4. EVALUATION SAMPLE This section describes the distribution of surveyed students in the schools selected for the performance evaluation in the seven project districts of Savannakhet province: Atsaphone, Nong, Outhoumphone, Phin, Phalanxai, Sepon, and Vilabouly. This section also contains summary statistics for the self-reported characteristics of students and their household environment. In addition, because IMPAQ drew a new sample at the midterm, we provide balance checks for key demographic characteristics pertaining to students and their households to ensure that the midterm sample is observationally equivalent to the baseline sample. 4.1 Schools To measure the midterm values for the performance indicators listed in Appendix C. McGovern Dole Indicators and to assess progress toward the desired outcomes, enumerators surveyed students in 83 schools in the seven districts: 34 SF-only schools, 23 schools that are part of both samples, and 26 schools for the LB sample only. Following the sampling strategy explained in 2.2.1 Quantitative Sampling Design, enumerators randomly selected students to survey from among those present the day the evaluation team visited the school: five students each in Grades 1 through 5 in each of the 34 schools in the SF sample and 10 second-grade students in each of the 26 LB sample schools. In 23 schools included in both samples, enumerators surveyed 10 second-grade students and five students in Grades 1, 3, 4, and 5. As in the baseline data collection, students in Grades 4 and 5 were often absent; additionally, schools canceled some classes because of teacher absenteeism or traditional events. To preserve a large enough sample, enumerators oversampled other schools in the same district, where possible following the same random selection rule. After oversampling to account for schools with low enrollment or attendance, our sample consisted of 1,852 students, including 744 second graders who took the LBRA. Teachers provided written consent for the children in their school. Enumerators also asked each student for his or her assent to participate in the survey; no student refused. Exhibit 6 shows the distribution of schools and students in each district. Exhibit 6. Distribution of Midterm Sample District Number of Schools Number of Students Baseline Midterm Baseline Midterm Atsaphone 19 19 423 468 Nong 7 3 161 63 Outhoumphone 14 14 370 315 Phalanxai 13 13 255 229 Phin 14 14 366 318 Sepon 8 8 174 239 Vilabouly 12 12 213 220 Total 87 83 1,962 1,852* Source: Student survey, authors’ calculations. * The number of students surveyed is lower at midterm than at baseline because of the four schools in Nong that dropped out of the program. 4.2 Students In each targeted school, enumerators randomly selected a balanced number of girls and boys (see 2.2.1 Quantitative Sampling Design for a full explanation). The balanced sample allowed us to disaggregate the LEAPS II Midterm Performance Evaluation Report  February 2020 17 data by gender and to look at differences by grade level. Exhibit 7 shows a breakdown of students surveyed by grade and gender. The midterm sample includes 924 males and 928 females. The sample is balanced by gender and by grade, with a slight difference in Grade 3, where there are more male students (53 percent) than females (47 percent). At baseline, the proportion of girls to boys in Grade 3 was reversed (57 percent girls and 43 percent boys); the difference in the proportions of girls and boys between baseline and midterm is statistically significant. However, in all other grades and in the sample as a whole, the gender balance is statistically the same at baseline and at midterm. Gender balance could be affected simply by which students attended on the day of the visit. Exhibit 7. Student Gender Distribution Grade Percent Male Percent Female Total Baseline Midterm Baseline Midterm Baseline Midterm Grade 1 43% 49% 57% 51% 304 278 Grade 2 49% 50% 51% 50% 693 744 Grade 3 43% 53% 57% 47% 313 280 Grade 4 56% 49% 56% 51% 320 277 Grade 5 50% 48% 50% 52% 332 273 Total 49% 50% 51% 50% 1,962 1,852 Source: Student survey, authors’ calculations. Exhibit 8 shows the average and range of ages for each grade in the sample. Reported ages in each grade did not vary significantly from baseline to midterm, with the exception of students in Grade 5 who, on average, were younger by about six months at midterm than at baseline, a difference that is statistically significant at the 1 percent level. Exhibit 8. Age Distribution by Grade Grade Mean Median Range Baseline Midterm Baseline Midterm Baseline Midterm Grade 1 7 7 7 7 5–14 5–12 Grade 2 8 8 8 8 5–14 5–14 Grade 3 10 10 10 9 5–15 7–16 Grade 4 11 11 11 11 7–15 7–16 Grade 5 12 12 12 12 6–17 7–16 Source: Student survey, authors’ calculations. On average, students were the correct age for their grade; however, the range of ages in each grade is quite large. Several factors likely influenced these wide ranges. Similar to baseline, 20 percent of the sample reported not knowing their age at midterm. Another possible explanation is a high incidence of grade repetition. Exhibit 9 shows a significant decline between baseline and midterm in the number of students who repeated a grade. However, the number is still quite large; over a quarter of the sample (28 percent) reported repeating at least one grade. Students in Nong reported the highest rate of grade repetition (37 percent) while students in Atsaphone had the lowest (21 percent). Grade repetition was more common among boys (30 percent) than girls (25 percent). Among sampled students in Grade 1-5 who had repeated a grade, 85 percent reported repeating Grade 1 or Grade 2. IMPAQ only included this question at midterm so we cannot compare these results to baseline. 18 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Exhibit 9. Grade Repetition by Grade Variable Baseline Midterm Difference in Means Percent Observations Percent Observations (p-value) Grade 1 43% 295 18% 276 –24%*** (0.0000) Grade 2 42% 684 26% 742 –17%*** (0.0000) Grade 3 40% 312 31% 278 –9%** (0.0169) Grade 4 41% 320 31% 277 –10%*** (0.0098) Grade 5 36% 332 37% 271 1% (0.8016) Total 41% 1,943 28% 1,844 –13%*** (0.0000) Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: Sample excludes students who did not know the answer to this question at baseline and midterm. Enumerators also asked students if they had ever attended preschool or an early childhood program. More than half of the sample at midterm, 52 percent, said yes, as compared to 46 percent at baseline. This difference is statistically significant at the 1 percent level. 4.3 Household Characteristics To capture children’s background, enumerators asked them about household size, language spoken at home, and socioeconomic status. 4.3.1 Household Size The average household size reported by students at midterm was 6.5 people per household, as shown in Exhibit 10. This average was slightly lower than the baseline average of 6.7 people; the difference is statistically significant at the 1 percent level. Regionally, Nong had the highest average household size at 7.4 people; the average in the remaining districts ranged between 6.2 and 6.8 people. Exhibit 10. Household Size Variable Baseline Observations Midterm Observations Difference in Means (p-value) Average household size 6.7 1,886 6.5 1,845 –0.2*** (0.0070) Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: Sample excludes students who did not know the answer to this question at baseline and midterm. 4.3.2 Language Spoken at Home Enumerators asked children about their primary language spoken at home, without reading the options to them. If the child reported a language other than those listed, the enumerators chose “other” as the response. As shown in Exhibit 11, 48 percent of children reported speaking Lao as their main language at home; this proportion is similar to the baseline proportion of 45 percent. The number varied widely by district, from just 2 percent in Nong to 86 percent in Outhoumphone. After Lao, Phouthay (20 percent) and Makong/Bru (18 percent) were the most common languages spoken by students in our sample. LEAPS II Midterm Performance Evaluation Report  February 2020 19 Comparison of differences in reported main languages spoken at home at baseline and midterm shows that a smaller proportion of students reported at midterm that they did not know their main language at home(p<0.01). More students at midterm (20 percent) reported Phouthay as their main home language compared to baseline (17 percent), statistically significant at the one-percent level. The percentage of Makong/Bru speakers was higher by 10 percentage points at midterm (p<0.01), but this increase should be interpreted with caution. The baseline survey used the term “Makong” as one of the options, but students usually called that language “Bru”—which enumerators captured as “other.” For the midterm survey, IMPAQ revised the options of this question to reflect the reality. The increase in Makong/Bru might represent a shift from the “other” option as captured at baseline. The fact that more students could report at midterm the language they spoke at home likely accounts for the significant increases in each language reported between baseline and midterm. In addition, enumerators asked students if they speak any other languages at home—that is, if their household is multilingual. At baseline, 40 percent said yes; at midterm, only 13 percent of students reported speaking at least one other language at home, a finding that is statistically significant at the 1 percent level. Two potential reasons could explain this decrease. First, looking at the significant drop from baseline (10 percent) to midterm (2 percent) in “Don’t Know” option shows that students more likely could name their main language and any other languages they speak at home at midterm. In addition, among multilingual students at midterm, fewer students reported speaking Lao (18 percentage points) and Phouthay (7 percentage points) in addition to their main language at home compared to baseline. Both changes are statistically significant at the 1 percent level. This means that students at midterm are more likely exposed to the school instruction Lao or Phouthay, which is the closest language to Lao. See Exhibit 32 in Appendix E. Additional Tables and Complementary Outcomes for a breakdown of other languages spoken at home. Exhibit 11. Main Language Spoken at Home Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,962 for baseline and 1,852 for midterm. 4.3.3 Socioeconomic Status To get a sense of socioeconomic status, the survey asked children if their household possessed any of eight goods or services: electricity, refrigerator, bicycle, motorbike, tok (tractor), television, mobile phone, and car. Exhibit 12 shows the results for baseline and midterm surveys. The increase in ownership of 2%*** 4%*** 3%*** 5%*** 18% 20%*** 48% 10% 15% 1% 3% 8% 17% 45% 0% 10% 20% 30% 40% 50% 60% Don't know Other Tri Katang Makong/Bru Phouthay Lao Baseline Midterm 20 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report motorbikes was large and statistically significant at the 1 percent level. Between baseline and midterm, the proportion of students who reported that their households owned mobile phones and refrigerators increased slightly, and the differences were statistically significant at the 1 percent level. Just 2 percent of the sample reported owning none of the items on the list, which was a slightly higher percentage than at baseline and not statistically significant. The average total number of goods and services households owned did not change between baseline and midterm, remaining at 4.9 items on average. On average, students whose main language at home was Lao owned one additional good or service than non-Lao speakers. Socioeconomic status varied greatly by district. In Nong, students reported owning just 2.8 goods or services; the remaining districts all averaged at least 4.1. See Exhibit 33 in Appendix E. Additional Tables and Complementary Outcomesfor the regional differences. Exhibit 12. Socioeconomic Status Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,962 for baseline and 1,852 for midterm. 2% 13% 46% 47%*** 70% 73% 79%*** 81%*** 85% 1% 14% 50% 43% 72% 73% 50% 76% 86% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% None Car Bicycle Refrigerator Television Tractor (Tok Tok) Motorbike Mobile Electricity Baseline Midterm LEAPS II Midterm Performance Evaluation Report  February 2020 21 SECTION 5. QUANTITATIVE RESULTS This section analyzes data from the student survey, including the LBRA, and classroom observations. The evaluation team examined all data by gender, grade, and district; we also examined literacy outcomes by main language spoken at home, where relevant. IMPAQ highlights the subgroup analyses if descriptive differences are statistically significant, and Appendix E. Additional Tables and Complementary Outcomes provides more detailed charts. Because IMPAQ based these analyses on data self-reported by primary school children, these results should be interpreted with caution. In analyzing performance data, the evaluation team compared mean outcomes at baseline and midterm by using t-tests and p-values to highlight statistically significant differences. Such analysis can only suggest correlations between the observed changes in outcomes and LEAPS II interventions, such as school meals or teacher training. It cannot determine conclusively whether the interventions caused the changes. Other factors could have led to the observed changes over time. Given that IMPAQ selected different individuals to survey at baseline and midterm, systematic differences in the two sets of samples could affect the outcomes. For example, improvements in literacy outcomes at midterm relative to baseline could mean either that LEAPS II teacher trainings were effective or that the students selected at midterm came from better socioeconomic backgrounds. In addition, other general trends that affected all schools could cause simple differences in outcomes between baseline and midterm. For example, a change in Grade 1 curriculum27 that will be effective from September 2019 in all Lao schools could affect the final evaluation outcomes in comparison with baseline values later in 2021. This section also presents the changes in the classroom observation outcome, that is, students’ attentiveness, from baseline to midterm. This comparison should be interpreted with caution because of the subjective nature of the measurement (See 7.2 Key Outcomes and Implications for more detail). Exhibit 13 provides an overview of the baseline and midterm levels of the key MGD evaluation performance indicators discussed in this section. The Data Source column shows the survey data used to measure each indicator. The mean of each indicator is presented with its corresponding confidence interval (95 percent), disaggregated by gender (G, B, and T represent girls, boys, and the total sample, respectively). The asterisk denotes whether the changes from baseline to midterm were statistically significant. Exhibit 30 in Appendix C. McGovern Dole Indicators also presents a full list of MGD indicators reported by IMPAQ and CRS. Exhibit 13. Midterm Levels for McGovern-Dole Performance Indicators McGovern-Dole Indicator Data Source Baseline Midterm Percentage 95% Confidence Interval Percentage 95% Confidence Interval Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand grade-level text LBRA1 G: 3% 1–6% G: 12%*** 8–16% B: 3% 1–6% B: 10%*** 6–14% T: 3% 2–5% T: 11%*** 8–14% 27 New curriculum in Laos. 2018, April 6. Retrieved June 29, 2019, from https://www.letlaoslearn.org/news/new-curriculum￾to-improve-quality 22 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report McGovern-Dole Indicator Data Source Baseline Midterm Percentage 95% Confidence Interval Percentage 95% Confidence Interval Percent of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters G: 47% 41%–54% G: 78%*** 72%–82% B: 46% 39%–52% B: 68%*** 62%–74% T: 47% 42%–51% T: 73%*** 69%–77% Percent of students who are attentive in the classroom Classroom observation2 G: 87% 84%–90% G: 60%*** 56%–65% B: 82% 79%–86% B: 58%*** 55%–63% T: 84% 81%–88% T: 60%*** 69%–76% Percent of students reporting that they are “somewhat” or “very” hungry during their afternoon class Student survey3 G: 7% 5%–12% G: 3%*** 1%–5% B: 8% 5–11 % B: 6% 4%–9% T: 8% 6%–10% T: 4%*** 3%–6% Percent of students in target schools reporting health-related absences Student survey4 G: 32% 30%–35% G: 35% 26%–31% B: 33% 30%–36% B: 25%*** 22%–28% T: 33% 30%–35% T: 27%*** 25%–29% Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. 1Grade 2 students who took the LBRA at baseline were 495 and 526 at midterm; 2enumerators observed 85 classrooms at baseline and 82 classrooms at midterm. 3This indicator is available only for those surveyed in the afternoon with N= 629 at baseline and 788 at midterm; 41,949 students at baseline and 1,845 students at midterm responded to this question. 5.1 School Environment Substantial evidence shows that access to books at school, such as a school library, and the type of classroom activities can influence children’s literacy outcomes. 28 For that reason, our survey of Grade 1– 5 studentsincluded questions to determine their current attitudestoward schooling, their access to books and inclination to read, and the classroom activities during a typical school day. In our analysis, where relevant, IMPAQ examined these results by school types (34 SF only schools vs 49 LB+SF schools) to also see whether the literacy component has changed the outcomes. 5.1.1 Student Attitudes toward Schooling Almost all students reported that they enjoyed attending school: 98 percent at baseline and 99 percent at midterm. This result did not vary by gender, geography, or grade. When probed about why they liked 28 Duflo, E., Hanna, R., and Ryan, S. P. (2012). Incentives Work: Getting Teachers to Come to School. American Economic Review, Vol. 102, NO.4. Lonsdale, M. (2003). Impact of School Libraries on Student Achievement: A Review of the Research (ISBN-0-86431- 6976) LEAPS II Midterm Performance Evaluation Report  February 2020 23 school, 80 percent at midterm pointed to writing, compared to just 40 percent at baseline. This difference is significant at the 1 percent level. The proportion of students who cited reading as a reason for liking school also increased significantly, from 48 percent at baseline to 71 percent at midterm. The least common reason for enjoying school, at midterm as at baseline, was presenting ideas in class, reported by only 2 percent of students. See Exhibit 38 in Appendix E. Additional Tables and Complementary Outcomes for details. The main language spoken at home and school type were the subgroups that accounted for statistically significant differences in perspectives on school. More specifically, more non-Lao speakers than Lao speakers gave reading as a reason for liking school. A higher percentage of non-Lao (15 percent) than Lao speakers (7 percent) named food as a factor making school enjoyable. Lao speakers more often cited being with friends and learning new things as reasons for liking school. All of these differences are significant at the 1 percent level. In addition, more students in the SF+LB schools than in SF schools reported writing, participating in class activities, working in groups, presenting ideas in school, and teacher reading as reasons why they liked school, all statistically significant at the 1 or 5 percent level. While, more students in SF gave school meals as a reason for liking school. See Exhibit 43 in the Appendix E. Additional Tables and Complementary Outcomes. Disaggregated by district, the data indicate that a smaller percentage of students in Nong and Vilabouly than in other districts pointed to reading or writing as reasons for liking school. In Nong and Phin, more students cited the provision of food as a factor in their enjoyment of school. 5.1.2 School Libraries Compared to baseline (45 percent), a higher proportion of students reported at midterm (75 percent) that their schools had a library; the difference is statistically significant at the 1 percent level. However, when students reported on how frequently they borrowed books other than textbooks from school, slightly more than half at midterm stated that they never did so, and the proportion of students choosing “a few times during the week” dropped from baseline to midterm, as shown in Exhibit 14. Despite a decrease in borrowing frequency in both groups, students in LB schools were significantly more likely to borrow books from schools once (p < 0.05) or a few times per week (p < 0.01), than those in SF only schools. Exhibit 14. Borrowing Storybooks from School Frequency Baseline Midterm Mean Observations Mean Observations Difference in Means (p-value) Every day 11% 846 < 1% 1,364 –11%*** (0.0000) A few times during the week 34% 846 22% 1,364 –12%*** (0.0000) Once during the week 25% 846 25% 1,364 0% (0.9256) Never 30% 846 53% 1,364 23% (0.0000) Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: The sample is limited to students who said their school had books to borrow. Even though students seemed to borrow books less frequently at midterm, more reported that they read somewhere outside of school at midterm (84 percent) than at baseline (77 percent). Girls read more (87 percent) than boys (81 percent). Both differences are significant at the 1 percent level. In Nong, only 45 percent of students said they read outside of school; the remaining districts ranged from 74 percent in 24 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Vilabouly to 92 percent in Sepon. Moreover, mores students in SF+LB schools (8 percentage points) read outside of schools compared to their peers in SF only schools, statistically significant at the 1 percent level. Despite rates of borrowing having declined from baseline, students seem to be reading more for their own enjoyment, probably by getting books from other resources as they reported reading more outside of school. At one school, the principal remarked that teachers do not yet know how the project will use the books and thus did not allow students to borrow the books. Generally, according to several principals interviewed, students find reading more enjoyable and fun because of LEAPS II activities. 5.2 Household Literacy Environment on the household literacy environment provides data on improvements in reading habits at home. 5.1.3 Classroom Activities Enumerators asked students about the frequency of specific teaching pedagogies: hearing a story or poem, answering questions about the story or poem, and playing alphabet games. Exhibit 15 shows that these activities were generally more frequent after the introduction of LEAPS II than at baseline, to varying degrees. For example, a greater percentage of students reported at midterm that their teachers told stories beyond the textbooks once a week or a few times a week, as compared to baseline when the response was more frequently “never.” However, the percentage of students who reported that their teacher told stories every day decreased significantly from baseline to midterm. Exhibit 15. Frequency of Classroom Activities Frequency Hear a Story or Poem Answer Questions about Story Play a Game Baseline Midterm Baseline Midterm Baseline Midterm Every day/often 10% 1%*** 6% 23%*** 8% 1%*** A few times a week/sometimes 36% 43%*** 27% 33%*** 28% 30% Once a week/rarely 14% 25%*** 13% 29%*** 13% 24%*** Never 41% 31%*** 55% 15%*** 51% 45%*** Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,764 for baseline and 1,778 for midterm. Note: The sample excludes students who missed all five days of school The results show some regional differences: 61 percent of students in Nong reported never hearing a story from their teachers, a percentage that was much higher than the range of 26–33 percent in other districts. However, these values showed improvements across the other six districts from baseline. At baseline the number was 40 percent in Nong which was about average among other districts. Our survey also inquired about the frequency with which teachers asked questions to their students about stories read or told in class. In contrast with baseline (55 percent), only 15 percent of students at midterm chose “never,” a decrease that is significant at the 1 percent level. Examining the data by district, 8 percent of students in Nong reported that their teachers sometimes asked them about stories compared to an average of 33 percent in the other six districts. A much higher proportion of students reported that they rarely received questions in Nong (54 percent) and in Phin (45 percent), compared to a range of 19-30 percent in the other five districts. See Exhibit 34, 35, and 36 in Appendix E. Additional Tables and Complementary Outcomes. Moreover, enumerators asked students how often their teachers led games that focused on the alphabet or reading. The results show a similar trend as for the other activities—the proportion of students stating “never” decreased, but regular game play did not seem to increase significantly. Only 1 percent of students answered that they had played games daily during the last week. The decrease in the “never” LEAPS II Midterm Performance Evaluation Report  February 2020 25 category from baseline (51 percent) to midterm (45 percent) was smaller than for the other teaching practices. Finally, looking at the outcomes by school type showed statistically significant differences across classroom activities, but the magnitude of changes was minimal from one option to another. According to students’ self-reported responses, teachers in SF+LB schools more likely (54 percent) told a story or read a poem to the classroom “a few times during the week” compared to SF only schools (39 percent). More students in SF schools (42 percent) than in SF+LB (30 percent) said that their teachers “sometimes” asked them questions about the story. Fewer students in SF+LB (36 percent) chose “Never” when asked about the frequency of teachers playing games around alphabet compared to their peers in SF schools (49 percent. All these differences are statistically significant at the 1 percent level. See Exhibit 44 in Appendix E for more detail. The general trend of improvement from teachers never or rarely engaging in LEAPS II interactive pedagogical practices to doing so on occasion aligns with the qualitative finding that trainings have been successful in demonstrating these practices. However, teachers still have difficulty in grasping them concretely. The progress made thus far have more so demonstrated incremental gains as teachers slowly begin to carry over some of the training concepts into their classrooms. 5.2 Household Literacy Environment Students exposed to literacy activities at home have better opportunities for literacy acquisition.29 Numerous studies point to the role of the home literacy environment in influencing early reading skills— in particular, children’s exposure to print materials at home and opportunities to engage in reading with other household members.30Our survey therefore asked questions about the availability of print materials at home and home literacy practices. 5.2.1 Availability of Reading Material at Home Exhibit 16 shows that a smaller proportion of students (4 percent) reported at midterm that they did not own any reading materials at home compared to baseline (7 percent). At midterm, more students reported having coloring and drawing books at home compared to baseline, a significant difference at the 1 percent level. There was no significant change in students’ responses about the availability of storybooks at home from baseline (23 percent) to midterm (25 percent). The participation of students in reading camps and access to book banks31 may explain both the lower frequency of borrowing storybooks from school and the unchanged level of having storybooks at home. 29 Kim, Y. S. (2009). The relationship between home literacy practices and developmental trajectories of emergent literacy and conventional literacy skills for Korean children. Reading and Writing, 22(1), 57-84. 30Hess, R. D. & Holloway, S. D. (1984). Family and school as educational institutions. Review of Child Development Research, 7, 179-222. Dowd, A.J., Pisani, L. & Borisava, I. (2016). “Evaluating Early Learning from Age 3 to Grade 3” in Understanding What Works in Oral Reading Assessments. Montreal: UNESCO Institute for Statistics (UIS). 31 As described in Section 1.1, Room to Read as part of LEAPS I established school libraries, and LEAPS II enhances the availability of reading materials by setting up reading corners in Grade 1, 2, and 3 classrooms. Book banks are another terminology used to describe the reading corners. 26 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Exhibit 16. Reading Material Available in Students’ Homes Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,962 for baseline and 1,852 for midterm. The availability of reading materials at home did demonstrate some significant variations based on student characteristics. Specifically, more Lao students than non-Lao speakers said that they had reading materials at home. This difference was most pronounced for coloring and drawing books (59 percent for non-Lao and 69 percent for Lao speakers). See Exhibit 37 in Appendix E. Additional Tables and Complementary Outcomesfor detail. Across districts, the proportion of students without any of the listed reading materials at home ranged from 2 percent in Atsaphone to 9 percent in Nong. There was no significant difference by gender. 5.2.2 Home Literacy Practices Compared to baseline, student responses at midterm indicate decreases in someone in the household reading to them or telling a story, as shown in Exhibit 17. At midterm, 29 percent of students reported that someone in their household told them a story compared to 42 percent at baseline, a significant difference at the one percent level. The proportion of students who were read to by someone in their household also decreased from 57 percent at baseline to 46 percent at midterm, significant at the one percent level. Possible explanations for this finding include students’ different backgrounds from baseline to midline, such as the educational attainment among household members, which the data did not capture. Students who participated in reading camps might also be better equipped to learn and read independently, as evidenced by our qualitative data where some mothers reported feeling satisfaction with the camps for teaching students so that they can have more time in the field. 4%*** 5%*** 10% 23% 64%*** 91%** 7% 7% 12% 22% 50% 89% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% None of these Magazines Newspapers Storybooks/Comics Coloring and drawing books Textbooks Baseline Midterm LEAPS II Midterm Performance Evaluation Report  February 2020 27 Exhibit 17. Home Literacy Environment Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,953 for baseline and 1,848 for midterm. Disaggregating the results by language, Lao speakers more often reported that someone read to them (51 percent) than non-Lao speakers (41 percent); this difference is significant at the 1 percent level. A much smaller proportion of students in Nong (20 percent) responded that someone had read to them at home in the last week than in other districts; the percentage ranges from 33 percent in Phin to 53 percent in Atsaphone. An examination of the data by gender showed no large differences, although, in a finding significant at the 5 percent level, a slightly larger proportion of female students recalled that someone had told them a story. Furthermore, looking at home literacy environment by school type showed two significant differences in students’ responses: more students in SF+LB have seen someone reading (8 percentage points), as well as reported someone read to them (13 percentage points) at home compared to SF only schools. Both differences are statistically significant at the 1 percent level. 5.3 Student Outcomes Primarily, LEAPS II strives to improve the literacy skills of school-aged children and increase their use of health and dietary practices. According to the LEAPS II theory of change, reducing health-related absences and providing school meals to mitigate short-term hunger and increase students’ attentiveness can keep students in school for the full day. As a result, students’ literacy skills can improve. Our survey included questions to measure key performance indicators in four areas: health, food security, attentiveness, and literacy outcomes. 5.3.1 Health To capture information on students’ health and its effect on their attendance, the survey asked students if they had fallen ill in the past week, and, if so, whether they had missed school because of the illness. As shown in Exhibit 18, at baseline, 41 percent of students had been ill within the last week, and 33 percent had therefore missed school. These proportions dropped significantly at midterm, a finding that matches responses from KIIs and FGDs. For example, one teacher at a low-performing school commented that “the big issue of sickness” causing students to miss school had greatly improved. The evaluation team found some minimal regional differences. Health-related absences ranged from 23 percent in Phin to 35 percent in Phalanxai. Although boys and girls got sick at about the same rate, girls were more likely to miss school, a finding that is significant at the 5 percent level. There was no significant 64% 81% 29%*** 46%*** 63% 81% 42% 57% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% During the last week, did you see anyone in your house reading? During the last week, did anyone in your house encourage you to study? During the last week, did anyone in your house tell you a story? During the last week, did anyone in your house read to you? Baseline Midterm 28 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report difference between the proportion of students who reported missing school for reasons other than illness at midterm (25 percent) and baseline (27 percent). Exhibit 18. Health-Related Absences Health Indicator Baseline Midterm Difference in Means (p-value) Student was sick any time in the last week. 41% 32% –9%*** (0.0000) Student missed school because of illness in the last week. 33% 27% –6%*** (0.0001) Student missed school for other reasons in the last week. 27% 25% –3%* (0.0550) Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,953 for baseline and 1,845 for midterm. As shown in Exhibit 19, fever was the most commonly reported illness at midterm (74 percent), followed by headache (25 percent). The proportion of students who reported fevers increased significantly compared to baseline (57 percent). Considering the number of students who had a fever in the past week remained the same from baseline to midterm, the huge drop in number of students who reported not falling ill at all could drive the resulting increase in fevers(i.e., a change in denominator rather numerator). The decline in illnesses apparently resulted predominantly from a decrease in headaches. Exhibit 19. Most Common Illnesses among Students Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 797 for baseline and 581 for midterm. There was no significant change in the proportion of students who missed school every day in the past week due to illness from baseline (11 percent) to midterm (10 percent). A higher proportion of Lao￾speaking students reported missing every day of school in the past week because of an illness (16 percent, compared to 5 percent for non-Lao speakers). The proportion of students reporting that they missed all five days of school ranged from 5 percent in Nong, Phin, and Sepon to 17 percent in Outhoumphone. Of note, this self-reported one-week-absence does not align closely with CRS monitoring data, which finds attendance rates to be 96 percent in March 2018 – the same month as the midterm data collection. However, this difference should be interpreted with caution for a few reasons. First, the monitoring data reflects 2018 data and measures a subsample of students for each full month of the school year, while 0%*** 0%*** 3%*** 5%*** 7% 25%*** 74%*** 2% 4% 8% 7% 8% 37% 57% 0% 20% 40% 60% 80% 100% Don't know Tiredness Other Coughing Sore stomach Headache Fever Baseline Midterm LEAPS II Midterm Performance Evaluation Report  February 2020 29 the midterm evaluation took place in 2019, and includes a subset of students who may not be the same as those students reflected in the monitoring data, and our survey asks about just the previous week. 5.3.2 Food Security To measure students’ hunger during school, enumerators asked students about their food intake. For the 57 percent of surveys administered in the morning, enumerators asked only whether students had eaten breakfast and felt full afterward. For the surveys administered in the afternoon, we asked about both breakfast and lunch. At midterm, almost all students (97 percent) reported eating breakfast that morning—a small but significant increase from baseline. Almost all students reported being full after breakfast; the change from 97 percent at baseline to 99 percent at midterm is significant at the 1 percent level. Among the studentssurveyed in the afternoon at midterm (43 percent of all surveys), 66 percent reported that lunch had already been served; of those, 97 percent reported that they had eaten the school meal, as shown in Exhibit 20. The proportion of students who reported that schools served lunch was higher than at baseline (42 percent), a statistically significant change at the 1 percent level. Among students surveyed after noon at midterm, only 4 percent reported feeling hungry. This proportion was significantly lower than at baseline (7 percent). There were large variations among districts. In Sepon, just 29 percent of students reported schools had served lunch; the figure was over 80 percent in Outhoumphone and Vilabouly (see Exhibit 39 in Appendix E. Additional Tables and Complementary Outcomes). According to CRS monitoring data collected during the 2017–2018 school year, schools included in the performance sample served meals during 82 percent of official school days. This gap could reflect the unreliability of self-reported data from primary school children. Alternatively, the timing of the survey in the early afternoon could have occurred prior to the cooks preparing the meals. In those instances where students reported school meals had not been served (34 percent of students), 96 percent said they still ate lunch, which could reflect the qualitative findings that some students have access to meals at home or packed lunches. Exhibit 20. School Lunches and Student Hunger Food-Related Question Baseline Midterm Difference in Means (p-value) Mean (CI)1 Observations Mean (CI) Observations Reported school meal had been served 42% (38%-45%) 633 66% (63%-69%) 788 25%*** (0.0000) Ate the served school meal 2 97% (94-98%) 262 97% (95%-98%) 521 0% (0.6707) Ate lunch despite reporting that the school meal had not been served yet3 92% (90%-95%) 376 96% (92%-98%) 266 3% (0.1635) Reported being “somewhat” or “very” hungry during the afternoon Girls: 8% 334 Girls: 3% 389 –5%*** (0.0008) Boys: 8% 295 Boys: 6% 399 –2% (0.4327) Total: 8% 7113 Total: 4% 790 –4%*** (0.0057) Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Note: Questions about lunch were asked only of students surveyed after noon.1Confidence Intervals. 2This was asked just of students who reported the school meal having been served. 3This was asked just of students who reported the school mean not having been served. 30 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report 5.3.3 Attentiveness To capture information regarding students’ attentiveness, enumerators observed classrooms in 82 out of 83 schools. 32 Every three minutes, enumerators captured the number of attentive students and the number of distracted students in the room for a total of 30 minutes, leading to 10 snapshots of attentiveness for each classroom. We also collected data on classroom characteristics (e.g., class size and arrangement) and captured information about student and teacher activities, as well as the subject being taught at the time of classroom observations to understand students’ learning environment better. To calculate the attentiveness rate, IMPAQ divided the number of attentive students by the total number of students in the classroom for all 10 observations. For the analysis, IMPAQ also looked at the attentiveness by various groups, including class type (multi-grade vs. single grade), school type (SF only schools vs. SF+LB), teaching subject, class size (defined as large if above average), and class arrangement. Across the 82 schools, the average observed class size was 15 students. In 40 percent of observed classes, teachers taught more than one grade – that is a multi-grade classroom. Only 34 percent of schools had permanent structure. Moreover, most of the classes (95 percent) had rows of desks with benches, while 5 percent had no desks or benches. The data show that the student attentiveness rate was 60 percent at midterm, a significant drop compared to 84 percent at baseline. None of the subgroup analysis accounted for any statistically significant differences in attentiveness. Although the quantitative results did not show any significant differences between girls and boys, a few responses from teachers and principals in the qualitative data reveal beliefs that girls can better focus and study in class than boys. To measure the reliability of observers’ scores, in eight schools, two enumerators simultaneously conducted the observations. Although thisIRR showed an excellent intra-class correlation of 99.9 percent, the observed decrease in attentiveness from baseline to midterm should be interpreted with caution for several reasons. First, snapshot observations measure a specific variable or indicator only at a specific point in time; they cannot determine conclusively whether the intervention caused the changes.33 For this reason, we should consider baseline and midterm values separately. Another factor that may have influenced the findings is the subjective nature of the tool. Inconsistency could be a challenge because of the inherent difficulty of making observational judgements across changing settings. For example, although the distribution of grades observed was the same at baseline and midterm, enumerators did not visit the same classes at both stages. Even in the same classrooms with the same teachers, teachers might have changed their approach from baseline to midterm. Observers were also not the same at midterm. The World Bank34 warns that observation variables can be “noisy” due to variations in characteristics across observation periods. Inconsistent adaptation of teaching techniques could also drive students’ lack of attention at midterm. The observations of teachers’ activities at midterm showed that teachers spent only 2 percent of their time in the classroom teaching compared to 43 percent at baseline. Instead, they spent an average of 61 percent of their time disciplining students and another 10 percent out of the classroom at midterm. Looking at attentiveness by teachers’ activities also confirmed that students’ attentiveness was at its lowest (13 percent) when teachers were engaged in classroom management, while it was at the highest 32 Due to an IT programming glitch, we could not retrieve the observational data for one school. 33 RTI International. (2015). Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC. 34 World Bank. (2016). Reducing Early Grade Drop Out and Low Learning Achievement in Lao PDR: Root Causes and Possible Interventions. Washington, D.C. LEAPS II Midterm Performance Evaluation Report  February 2020 31 (89 percent) when students and teachers were expressing their ideas. See Exhibits 40, 41, and 45 in Appendix E. Additional Tables and Complementary Outcomes for details. 5.3.4 Literacy Assessment Outcomes Consistent with baseline, to measure second-grade students’ literacy skills, IMPAQ administered the LBRA. The LBRA consists of seven subtests: 1. Expressive vocabulary: Total number of animals and foods in the market that the child could name in Lao 2. Phonemic awareness: Number of word pairs identified, out of three, based on similar first-letter sounds 3. Letter knowledge: Number of letter-sound pairs known out of 33 4. Word recognition: Number of words read correctly out of 20 most-used words from leveled textbooks 5. Word to picture matching: Number of words matched with their pictures of the objects, out of nine 6. Reading a short story of 134 words ▪ Fluency: Number of words read correctly in a minute ▪ Accuracy: Percentage of words read correctly (untimed) 7. Comprehension: Eight comprehension questions related to the short story were asked in one of two ways: ▪ Reading comprehension, which applied to children who could read at least five words in the story correctly in 30 seconds. These children were identified as “readers” and allowed to finish reading the passage. ▪ Listening comprehension, which applied to children who could not read five words in the story correctly in 30 seconds. The enumerator read the story aloud to these children, identified as “nonreaders.” Enumerators administered all subtests in Lao, which is the official language of instruction; however, the instructions for completing the test were provided in the local language as necessary. This section presents the key reading outcomes for LEAPS II performance indicators in all three cohorts of SF+LB schools, regardless if the cohort received the literacy package.35 Appendix F provides an overview of other subtests and outcomes. In addition, to explore possible predictors of student literacy skills (e.g. socioeconomic status, grade repetition, etc.), the evaluation team used multivariate regression analyses, described in Appendix G. Regression Analysis. Exhibit 21 shows an overview of the key literacy outcomes of second-grade students in LB schools at baseline and at midterm. Though students showed significant improvement in foundational literacy skills, this improvement did not translate to an increase in reading skills. The proportion of students classified as readers remained at 18 percent. These readers did improve significantly in comprehension, increasing from a 50 percent pass rate at baseline to 70 percent at midterm. IMPAQ defined passing as answering 35 LEAPS II is providing the literacy intervention for three cohorts over the life of the project. Cohort 1 (Sepon and Atsaphone) has already received the package for both Grade 1 and 2; Cohort 2 (schools in Outhoumphone and Vilabouly), has received the package for Grade 1 and will continue the LB intervention for Grade 2 in the coming school year; and finally Cohort 3, which includes schools in Phin and Phalanxai, will begin to receive the literacy package in the final two years of LEAPS II, starting this upcoming academic year. 32 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report 75 percent of the eight comprehension questions correctly. In general, students whose main language spoken at home was Lao did better than their peers on most of the foundational skills (subtests numbers 1 through 5 mentioned above) but fared no better in being classified as readers or passing the comprehension test. Exhibit 21. Second-Grade Students’ Literacy Skills Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 496 for baseline (87 reading and 404 listening); N = 526 for midterm (98 reading and 428 listening). Only students in LB schools were surveyed. a The baseline value for these variables appeared higher in the baseline report (March 2016) due to rounding. To capture the changes from baseline to midterm more accurately, we changed the percentages to the unrounded values. Literacy Skill Baseline Mean Midterm Mean Difference (p-value) Oral Vocabulary and Phonemic Awareness Expressive vocabulary (# out of 20) 14.3 16.1 1.8*** (0.0000) Expressive vocabulary (%) 71% 80% 9%*** (0.0000) Phonemic awareness (word pairs correct out of 3) 1.7 1.6 –0.1 (0.1555) Phonemic awarenessa (%) 58% 54% –4% (0.1555) Foundational Literacy Skills Letter knowledge (# correct out of 33) 21 27 6*** (0.0000) Letter knowledgea (% correct) 65% 82% 17%*** (0.0000) Students who were able to identify at least 75% of the letters 47% 73% 26%*** (0.0000) Word recognition (# correct out of 20) 5 8 2.4*** (0.0000) Word recognitiona (% correct) 27% 40% 12%*** (0.0000) Word to picture matching (# correct out of 9) 4 6 2*** (0.000) Word to picture matching (%) 44% 61% 18%*** (0.000) Reading Skills Students classified as readers (5+ words correct in 30 seconds) 18% 19% 0.2% (0.9191) Accuracy (% words correct in passage), readers only 87% 87% 0% (0.9011) Fluency (words correct per minute), readers only 29 28 –1 (0.5498) Reading and Comprehension Skills % reading comprehension questions correct, readers only 50% 70% 20%*** (0.0000) % listening comprehension questions correct, nonreaders only 47% 48% 1% (0.4449) Reading proficiency (readers with at least 75% comprehension) for the sample of Grade 2 students 3% 11% 8%*** (0.0000) LEAPS II Midterm Performance Evaluation Report  February 2020 33 Letter Knowledge To measure letter knowledge, enumerators showed students a chart of 33 letters in Lao and asked them to identify the sounds of the letters. Students able to identify 25 of the Lao letters (75 percent) were considered proficient in identifying letters, based on the definitions in the LEAPS II performance monitoring plan. On average, students were able to identify six letters more at midterm (27) than at baseline (21), a finding that is statistically significant at the 1 percent level. As shown in Exhibit 22, students made large progress in identifying between 26-33 letters from baseline (38 percent) to midterm (70 percent). Just 1 percent of students could not identify any of the letters, a significant difference from 3 percent at baseline. Exhibit 22. Number of Letters Identified Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 496 for baseline and 526 for midterm. Similarly, significant is the increase in letter proficiency from baseline (47 percent) to midterm (73 percent). This improvement was larger for girls (30 percentage points) compared to boys (22 percentage points) even though their level of proficiency was almost the same at baseline (46-47 percent, respectively). Students could most easily identify ‘ກ’ (96 percent). Girls could identify this letter 98 percent of the time compared to 94 percent of boys (p < 0.05). Overall, only 51 percent of students were able to identify the hardest letter -‘ຫຍ’. Again, girls outperformed boys at identifying this letter (55 percent) compared to boys (46 percent); this difference was statistically significant at the 5-percent level. 3% 9% 14% 35% 38% 1%*** 3%*** 7%*** 19%*** 70%*** 0% 20% 40% 60% 80% 100% None Identified 1 to 9 letters Identified 10 to 17 letters Identified 18 to 25 letters Identified 26 to 33 letters Baseline Midterm 34 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Exhibit 23. Letter Proficiency by Gender Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = baseline: 260 for female and 236 for boys; midterm: 275 for girls and 251 for boys. Note: Letter knowledge is defined as a student’s ability to identify at least 75 percent of the letters in the alphabet. Other subgroup analysis also shows some significant difference. Students who reported speaking Lao as their main language were able to identify 84 percent of 33 letters compared to 80 percent of non-Lao speakers, a significant difference at the 10-percent level. Lao speakers were also significantly more likely to be able to identify at least 75 percent of the 33 letters at midline (78 percent) compared to baseline (60 percent), a significant difference at the 1-percent level. However, as shown in Exhibit 42 in Appendix E. Additional Tables and Complementary Outcomes, overall the distribution of letters identified by Lao and non-Lao speakers were consistent. In addition, looking at the data by district, students in Phin seem to have much weaker letter identification skills than the other districts. For example, just 34 percent of students in Phin could identify 75 percent of the letters compared to the midterm average of 73 percent. This finding could reflect the fact that Phin has the second lowest rate of Lao speakers (29 percent), as reported by students. Reading Outcomes IMPAQ classified students as readers if they could read at least five words in the LBRA story in 30 seconds.36 As at baseline, just 18 percent of students met this criterion. IMPAQ found no differences between Lao and non-Lao speakers or between boys and girls. However, as shown in Exhibit 24, student performance varied by district. The highest rates of readers came from Outhoumphone, Sepon, and Phalanxai, while the lowest rate was in Phin, at just 7 percent. Additionally, Phin was the only district to see a significant drop in readers from baseline (21 percent) to midline (p < 0.01). Meanwhile, Vilabouly (13 percentage points) and Atsaphone (8 percentage points) were the only two districts to increase the proportion of readers, significant at the 5 and 10 percent levels, respectively. Although the proportion of readers in Phin dropped significantly, this trend should be interpreted with caution. Students at midterm might come from different background compared to those surveyed at baseline. There is a lower rate of Lao speakers in the Phin sample with poorer socioeconomic status compared to other districts. Plus, schools in Phin and Phalanxai have not completely received the literacy package yet, which could also contribute to the lower rate of readers in those districts. 36 IMPAQ used the same passage from baseline to test students’ reading skills. 47% 46% 78%*** 68%*** 0% 20% 40% 60% 80% 100% Girls Boys Baseline Midterm LEAPS II Midterm Performance Evaluation Report  February 2020 35 Exhibit 24. Proportion of Students Defined as Readers, by District Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Baseline Ns: 121 in Atsaphone, 63 in Outhoumphone, 97 in Phalanxai, 122 in Phin, 54 in Sepon, and 38 in Vilabouly. Midterm Ns 207 in Atsaphone, 50 in Outhoumphone, 75 in Phalanxai, 96 in Phin, 140 in Sepon, and 48 in Vilabouly. No schools in Nong were part of the SF + LB condition, so no LBRA data are available. Note: 18% was the average at baseline and midline The same passage was also used to measure students’ fluency (words read per minute) and accuracy (percentage of words read correctly). There were no significant improvements in fluency or accuracy between baseline and midterm. On average, readers were able to read 28 words per minute at midterm, compared to 29 at baseline. Fluency was highest in Outhoumphone, at 32 words per minute, and lowest in Phin, at 25 words per minute. Students were able to read 87 percent of the words in the story accurately at both baseline and midterm. The results did not show significant differences by gender or language. After readers read or nonreaders listened to the whole passage, enumerators asked them four types of comprehension questions: 1. Summary (one question). Summary questions tested students’ ability to identify the main ideas of a reading passage. Enumerators asked students to recall what happened in the story. IMPAQ defined the passing rate as whether the student could point to at least three out of four main events in the passage. 2. Literal (five questions). For these questions, students could find the right answer clearly and explicitly stated in the passage. 3. Inferential (one question). This question asked for information that was implied rather than stated in the passage. 4. Evaluative (one question). This type of question required cognitive and/or emotional judgment of the students and they needed to use their own opinions to answer. At baseline, readers and nonreaders performed about the same on the comprehension questions, answering 50 and 47 percent, respectively, of the questions correctly. However, at midterm, readers scored much better than nonreaders, improving to 70 percent while nonreaders stayed about the same at 48 percent. Similarly, the proportion of readers who achieved a passing score on the comprehension questions increased from 17 percent at baseline to 57 percent at midterm, a change that is statistically significant at the 1 percent level. The proportion of nonreaders who passed showed a modest but significant (at the 10 percent level) increase from 21 percent at baseline to 26 percent at midterm. This 11% 13% 34% 21% 20% 0% 19%* 24% 29% 7%*** 22% 13%** Midterm Average: 19% 0% 20% 40% 60% 80% 100% AtsaphoneOuthoumphone Phalanxai Phin Sepon Vilabouly Baseline Midterm 36 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report finding shows that, even though the proportion of readers stayed almost the same from baseline to midterm, the readers were now able to comprehend more questions. Exhibit 25 shows a breakdown of the proportion of students who passed each of the four LBRA sections. Readers improved significantly in almost all sections, with the largest gains in the evaluative section. However, in a finding consistent with another SCI study on LB implementation similar to the LEAPS program in Lao, 37 students did not fare well in the summary section. Possibly, differences exist in the way teachers instruct students on how to summarize or express their opinions. Listeners improved in comprehension generally but not in all sections. Exhibit 25. Students Who Passed Each Literacy Section Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Listener Ns: 404 for baseline and 428 for midterm; reader Ns: 91 for baseline and 98 for midterm. A score of at least 75 percent constituted a passing score for the overall score and for each section. IMPAQ also evaluated grade-level reading competency, defining it as the reader’s ability to answer at least 75 percent of the reading comprehension questions correctly. As shown in Exhibit 26, the proportion of nonreaders and readers stayed the same from baseline to midterm. However, among students who could read with comprehension, there was a significant improvement from baseline (3 percent) to midterm (11 percent). Reading with comprehension was highest in Outhoumphone, at 22 percent, and lowest in Phin, at 3 percent. As observed in the data, literacy outcomes were more or less at the lowest in Phin compared to other districts at midterm. This could be potentially explained by the fact that this district is in the third cohort and has not received the literacy package yet, as well as it has the second lowest rate of Lao speakers. However, it might be worth exploring reasons behind these regional differences to incorporate them in to the LB when implementing the intervention for the third cohort. Still, at midterm, the majority of students were not proficient at their grade level. This finding matches the literature that suggests reading with comprehension usually comes with a lag due to the curriculum that is developed and taught in Lao, which is not mother tongue for about half of the students. In fact, Lao students make significant gains in fluency and comprehension in Grades 3 and 4, but it is only in Grade 5 that the majority of students are able to read with fluency and comprehension. 38 This finding is relevant in terms of curriculum expectations, since Grade 5 coincides with the last grade of primary school in Lao PDR. SCI staff noted in KIIsthat the multilingual environment and the complexity of the Lao alphabet could complicate the ability to understand texts for younger children, especially those who have had little exposure to Lao prior to 37 Save the Children. 2016. "Laos Education Access Research and Networking (LEARN) Baseline Assessment.” 38 The World Bank. May 2016. Reducing early grade dropout and low learning achievement in Lao PDR. Retrieved from: http://documents.worldbank.org/curated/en/161641483590757065/pdf/111627-REVISED-PUBLIC-Lao-PDR-Root-Causes-of￾Early-School-Leaving-f.pdf 11% 42% 44% 25% 17% 10% 82%*** 55%* 80%*** 57%*** Summary Literal Inferential Evaluative Overall Readers Baseline Midterm 12% 35% 48% 45% 21% 8%** 40%* 38%*** 42% 26%* 0% 20% 40% 60% 80% 100% Summary Literal Inferential Evaluative Overall Listeners Baseline Midterm LEAPS II Midterm Performance Evaluation Report  February 2020 37 entering school. As much as reading comprehension may be a struggle for Lao-speaking students(program staff noted acquisition of such skills at the earliest around end of Grade 3) it usually takes even more time for non-Lao speakers to fully grasp the materials they read (Grade 4 or Grade 5). Exhibit 26. Reading Proficiency Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 496 for baseline and 616 for midterm. Note: a beginning reader is a student who reads without comprehension. 81% 82% 8% 15% 11% 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Midterm Baseline Nonreader Beginner Reader Reader with Comprehension 38 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report SECTION 6. QUALITATIVE RESULTS This section presents the most pertinent findings from our qualitative analysis, drawing mostly upon the KIIs and FGDs conducted at the school and community levels. Where relevant and appropriate, IMPAQ provides perspectives from PESS as well as CRS, its partners, and USDA. Our analysis notes divergences among high-, average, and low-performing schools (see 2.2.2 Qualitative Sampling Design for more information on how IMPAQ derived these classifications). The qualitative results reflect the opinions and attitudes of a small sample of respondents. They provide context to help with interpretation of the quantitative findings and highlight areas that could benefit from further exploration and study. We should not assume external validity and extrapolate the findings to all LEAPS II schools. Following the organization of research questions in the midterm TOR (and provided in Appendix B. LEAPS II Evaluation Questions and Conceptual Framework), we have broken down the presentation of results into the categories of relevance, effectiveness, efficiency, sustainability, and impact. Exhibit 27 summarizes the salient themes across the different research question categories, and we provide details in their respective components of this section. Exhibit 27. Summary of Qualitative Results •Teachers and principals reflected positively on the different trainings related to improved pedagogy. •Parents and students found the meals to be too uniform and wanted more diverse food options. Relevance •School meals decrease the economic burden of providing food for children and relieves anxiousness and preoccupation with hunger for students while in class, which helps them to pay attention. •Lack of behavior change within households may impede improved student health and hygiene. A more overarching WASH challenge remains access to water; even in those schools where LEAPS II or other donors have drilled boreholes and provided O&M trainings, communities struggle with effective maintenance. •Capacity-building activities could be enhanced via prolonged, more frequent, and interactive trainings. Effectiveness •CRS management of activities has been adaptive and responsive to community needs. •CRS staff were mostly satisfied with the monitoring, but school and community stakeholders want more frequent visits; DESB noted that it would like to participate more fully in monitoring efforts. Efficiency •VEDCs have had varying levels of success mobilizing the community to contribute to school meals. •Lessons learned from successes and challenges of teacher trainings can be helpful in identifying strategies for more long-lasting and impactful approaches to improve pedgagogy in the classroom. •Reading camps will be difficult to sustain without community buy-in. Sustainability •School feeding has been a critical support to enable children to attend school. •Reading camps and teacher trainings were cited as the elements of LB that had the most impact on improved literacy and learning outcomes. Impact LEAPS II Midterm Performance Evaluation Report  February 2020 39 6.1 Relevance To capture the relevance of LEAPS II for addressing community needs, our research questions consider: 1) the alignment of LEAPS II activities with government priorities and policies, 2) the extent to which the project trainings address the knowledge gaps of targeted stakeholders, and 3) the acceptability and appropriateness of the school meals and whether they function as an incentive for increased student attendance. Overall, LEAPS II has matched well with stakeholder needs. As detailed in this section, trainings on pedagogy have been particularly pertinent given the challenges that teachers face instructing students. However, teachers, PAs, and principals remarked the training format could be more interactive and frequent (increased duration and/or number of occurrences). School meals, though cited as relevant for improved attendance, lack diversity in their contents, which means that certain students with the means to do so, may choose not to eat at school. We elaborate on these main points in the sub-sections below. 6.1.1 Alignment with Government Objectives Most DESB officers and PAs noted in KIIs that LEAPS II programming is well aligned with the Government of Lao PDR’s education priorities, especially in areas related to increased school attendance, reduced dropout rates, improved literacy outcomes, and inclusive education (IE). Responses on program alignment with government priorities often took a high-level view, such as broad￾stroke assertions that the government “supports” the activities implemented in the schools. Critiques mostly related to concerns about adaptation of programming to the local context. For example, the DESB representative in one district noted that LEAPS II had distributed toothpaste and toothbrushes for WASH activities, but that some of the schools still did not have running water. Additionally, DESB technicians cited a desire for greater involvement in project monitoring activities (see 6.3 Efficiency); in one district, a PA suggested that better coordination of monitoring visits to align with the DESB monthly meeting schedule could help to synchronize the reporting of LEAPS II results with DESB activities. 6.1.2 Adequacy of Trainings to Address Knowledge Gaps Regarding the trainings delivered to teachers, principals, and CLVs (detailed in 1.1 Overview of Program Implementation), almost all stakeholders commented positively on the pedagogical methods learned to diversify teaching approaches and increase interaction with children, such as games and singing. Teachers stated that these methods have been helpful in addressing issues with student engagement. One teacher remarked that learning how to play educational games with students was valuable because the teacher did not “want them to get bored,” adding that for students to “enjoy learning is important.” Teachers and CLVs also commended the techniques to encourage reading and writing using a variety of teaching materials, such as word cards and storybooks, saying that these items have been attractive to students. One PA noted that students felt more excited and motivated to study when teachers incorporated these new methods instead of taking the traditional approach that relies on adherence to the textbook. IE trainings also aligned with teacher needs, such as rearranging classroom seating based on student learning needs or to accommodate children with disabilities. One teacher at a low-performing school stated that, after receiving the IE training, he has paid more attention to struggling students and has worked closer with After the training, the teaching practices have evolved. KII with PA in a community with a low-performing school 40 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report them to help improve their reading and writing abilities. 6.2 Effectiveness discusses in further detail the effectiveness of these IE trainings. Though many found the information relevant, some teachers and principals complained about the format of the various trainings received. Several teachers indicated that because they could not remember the training material, they found it difficult reproducing the pedagogical methods. No teacher mentioned specifically receiving any handouts or materials to take with them after receiving any of the trainings. Many teachers said they wanted more practical, hands-on, regular, and interactive training. One principal at a low-performing school stated that the trainings did not provide enough examples of how to play games with students, and these trainings relied more so on PowerPoint presentations. Respondents also commented on the relevance of trainings on safe food preparation and storage and whether these trainings address health and hygiene concerns. Cooks and principals who received food safety training consistently noted that they had learned the importance of washing all ingredients and supplies, so that students would be able to eat “cleaner” food. Compared to sentiments from the baseline evaluation, when many school-level stakeholders and cooks revealed a strong desire for more training on preparation of diverse foods, fewer respondents expressed this need. Though some cooks did still comment on wanting to learn how to diversify meal preparation (see also 6.1.3 Acceptability and Appropriateness of School Meals), more cooks complained about availability of additional ingredients to accompany the rice and lentils, and adequate storage facilities. A few cooks reported supplementing the school meals themselves by bringing eggs, salt, MSG, and garlic from their homes. Hygiene trainings delivered to teachers based on the Fit for Schools model from MoES (described in Section 1.1), do seem to have filled a knowledge gap. One CM remarked that students “seem to know more about hygiene now” upon learning about best practices from teachers, and that students were helping cooks to wash utensils after meals. Mothers in one community receiving WASH support noted that seeing their children engaged in improved handwashing and bathing practices made them worry less about their children’s health. However, many respondents, while recognizing the importance of improved hygiene, described difficulties in accessing water and certain practices, especially latrine usage, have not necessarily caught up to improved knowledge (see 6.2 Effectiveness). One area relevant to hygiene and health at school, and perhaps not sufficiently addressed in the WASH trainings, is general school cleanliness. At one WASH school, students complained about garbage on the school grounds. Respondents at other schools did not specifically point out trash, but some noted that the school environment could be cleaner. 6.1.3 Acceptability and Appropriateness of School Meals Most parents held positive views of school lunches. However, when communities cannot contribute supplementary ingredients to the school meals (see 6.2 Effectiveness for a discussion of such challenges), the donated rice and lentils alone do not seem to meet student tastes. Commonly, parents noted that children should have access to a greater variety of food in their meals. Some parents stated that their children felt satisfied with the food at first but quickly became bored eating the same thing every day, which has led to varying solutions, such as relying on packed lunches, returning home for food, or eating only a partial portion and reserving the rest to take home. Other stakeholders, including DESB Students have access to clean food because the cooks followed the steps of fetching clean water to wash rice and dishes before cooking. FGD with teachers at a low￾performing school LEAPS II Midterm Performance Evaluation Report  February 2020 41 representatives and teachers, confirmed this sentiment. Satisfaction with the food did not vary by cooking rates or classification of best/worst case schools. For school meals to be a more relevant incentive for school attendance, parents noted that these lunches should add a wider range of ingredients, such as eggs, meat (canned fish was cited several times), vegetables, and sticky rice in lieu of “plain” rice. There were also requests for ingredients such as garlic, fish sauce, and monosodium glutamate (MSG). Although certain communities have been able to mobilize to contribute such items, cooks reported that they bring such ingredients themselves to the schools if the community cannot collect together these inputs because children would not find the meals tasty otherwise. Fathers were slightly less critical of the foods provided compared to mothers. According to one FGD with fathers, the “food served at the school is sometimes much better than food at home that parents could provide.” Fathers appreciated that school meals helped their children focus in class. However, in only one FGD did a mother explicitly state, “The taste is yummy.” One mother said that the schools should better engage the community to provide foods to make the meals more appetizing. Of note, only women, often mothers, prepare the school meals. Generally, students described feeling happy and full after eating the school meal, which aligns with the high percentage of children responding in the quantitative survey they like the taste of the meal. However, in the FGDs with students, which allowed for more probes into their responses, they consistently echoed their parents’ comments that the food has been “boring.” When asked about the types of food they would like to consume at school, almost all named some kind of meat or protein, such as fish, omelet, or grilled meat. In one FGD at a high-performing school, a student reported tasting the school meal before determining whether to remain at school for lunch. In two schools, respondents remarked that if children can eat better food at home (e.g. access to grilled fish or fish stew, etc.), they would prefer to leave school for lunch. However, this preference to eat outside of school arose at two of our seven qualitative schools, and we cannot know with certainty the factors behind such findings, e.g. the socioeconomic well-being of the households in the community. As well, the qualitative findings cannot be extrapolated to the entirety of LEAPS II schools. Returning home to eat lunch also emerged as just one solution to tackle the issue with the diversity of food options – some parents reported that they pack lunches for their children to eat at school and share with their friends. 6.2 Effectiveness In examining the effectiveness of LEAPS II at this midpoint of intervention, we assessed the following four topic areas: 1) the effectiveness of school meals as an incentive for school attendance and enhancement to classroom attentiveness; 2) the extent to which WASH interventions, including training on safe food preparation and storage, improved knowledge, attitudes, and practices; 3) any increases in capacity for MoES staff, teachers and principals, and VEDC; and 4) the extent to which learning has occurred among the stakeholders and partners. Our findings revealed, broadly, that school meals have been an important social safety net for poor families and have helped students to focus more on their lessons in the classroom rather than on their Cooks should change the menu at least three times a month to make the food more appetizing for children. FGD with mothers at a high￾performing school 42 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report next meal. As for WASH interventions, respondents at the two WASH schools in our sample reported that the availability of water from CRS-provided boreholes has been helpful for handwashing stations. However, at the non-WASH schools, teachers and principals commented on the tainted water sources and the limited treatment of such water for drinking purposes, which has hampered children from maintaining good hygiene practices. As well, no matter whether schools have received support to increase water supply on site, students continue to engage in certain behavior that impede improved hygiene and health, such as open defecation. CRS’ monitoring data confirms that open defection still remains at high rates (42 percent). Thus, challenges remain that may limit the effectiveness of WASH interventions. Regarding capacity building and trainings, all recipients of LEAPS II trainings generally found them valuable, and reported applying the lessons learned to their responsibilities. However, teachers especially, said they would like more interactive, longer, and frequent trainings to internalize fully the concepts. As well, school and community-level stakeholders noted that having the opportunity for greater dialogue not only with CRS but also between communities could be helpful for learning that is more effective. The subsequent sub-sections elaborate on each of these general trends and themes in detail. 6.2.1 Effect of School Meals on Attendance and Attentiveness Across high-, average, and low-performing schools, parents explained that school meals had eased the burden of preparing lunch every day for their children. School meals help offset the costs of preparing food at home. A few parent FGDs revealed that students sometimes brought leftover food from school to share with their household. Most of the discussions relating school meals to attendance focused on how the meals motivate students to attend school rather than helping parents reinforce attendance. Ultimately, as parents in almost all FGDs explained, school attendance is up to the children. One father explained, “Because of the school meal, children are willing to go to school.” Another parent stated, “The school meal convinces them to go to school.” Teachers and principals also said that students were motivated to attend school because of the food. One principal reported that, when students did not see cooks at school, they might be absent the next day. A teacher in a different district said that the school meals program “inspires students to come to school regularly.” Stakeholders other than parents seemed to feel more strongly that the school meals have encouraged attendance. One CRS partner stated adamantly that, without lunch in schools, enrollment rates would decline. Parents, though they acknowledged that the food offers a “convenience” and felt happy that their children could eat at school, considered the importance of education to be the primary driver behind their decision to enroll their children. As for the effect of the school meals on student attentiveness, school and community-level stakeholders generally applauded the school meals program for helping students to concentrate better on their coursework because they do not need to constantly worry about what to eat and who will feed them at home. Almost all student FGDs confirmed that students felt full after eating the school meal and were therefore less anxious to leave school to find food. One teacher noted that, although attentiveness had improved generally, high-performing students continued to pay more attention than low-performing students. This respondent also said that female students seem to concentrate better than males. A teacher in a different district, agreeing that young girls outperformed boys, said that they see gender differences mostly in the early grades. These comments When I eat at school…it helps me to concentrate on studying. When I am hungry, I cannot concentrate, and I only want to go home to get something to eat. FGD with students in a high￾performing school LEAPS II Midterm Performance Evaluation Report  February 2020 43 suggest that, though the project seems to have helped students focus less on their next meal, the concentration on learning in the classroom may be attributed to factors other than food, such as whether teachers have been able to create an engaging environment that elicits student participation. Consequently, school meals alone may not address other perceived obstacles to student performance. 6.2.2 WASH Knowledge, Attitudes, and Practices Lack of access to a consistent water source presents a cross-cutting challenge to improved WASH practices. For example, where CRS or other donors have constructed toilets, these facilities sometimes become non-functional without connection to a water source. The lack of water can cripple safe food preparation, too; at one school, cooks stated they had not prepared any mealsfor several months because of a broken water pump. Sourcing water at the school often seemed to rely on community engagement. Cooks across districts stated that they had to walk to fetch water from nearby households because they cannot readily draw upon a water source on site, making several trips to get enough water to wash the food, pots, and dishes. Children, too, sometimes needed to travel 50 to 200 meters to obtain water for activities such as handwashing. Teachers and cooks mentioned that children in lower grades almost never want to make the effort of going to the nearest house for water. Although the challenge of physically being able to access water has been mitigated in the WASH schools where CRS-provided boreholes offer a safe water source for handwashing and drinking, a few respondents across non-WASH schools pointed to children accessing unsafe water, even despite sourcing it from functional water pumps or boreholes, which has hindered their overall cleanliness. A potential solution to increase access to treated water may lie in water filters, which CRS has already planned to distribute across all intervention schools in five of the seven LEAPS II districts. As a principal at a high-performing school noted, after the school received a water filter from a USAID project, students switched to drink only filtered water, thus encouraging their households to boil their water. VEDC members at an average-performing school explained that they had received training from the district health office on water filters, but that the cost of such equipment was prohibitive for most households. Quite possibly then, after CRS introduces water filters in this upcoming school year water treatment may improve at the school and community level. In terms of sanitation, students, VEDC members, teachers, and principals across schools in our qualitative sample said that children and households in the village often engage in open defecation in the forest. Although some schools had toilets, these facilities did not always function and these schools could not connect reliably to a water source. A VEDC FGD at one non-WASH school noted that, though the school had a water hand pump, lack of sufficient water underground required an hour’s wait after obtaining a full bucket of water for more water to become available. This issue becomes particularly acute during the dry season. We may also attribute infrequent toilet usage to ingrained practice and habit. Several VEDC members remarked that most households in their village lacked latrines. Children even walk back home to drink water in the day. FGD with VEDC in a high￾performing, non-WASH school Children have not gotten used to using the toilet. When they want to defecate, they go into the forest. KII with principal at an average￾performing school 44 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report Consequently, according to a few teachers, VEDC members, and principals, students were not accustomed to using toilets. Teachers and principals commonly expressed frustration and defeat over their inability to change latrine usage practices. One principal at a low-performing school complained that, despite having plastered posters all over the school encouraging students to use the toilet, most students still chose to go in the forest. The VEDC in a community with a WASH school reported, “Only teachers use [the toilet]… [Students] do not have toilet at home and they have not gotten used to it.” Members of this VEDC suggested that improvements in health and hygiene practices might require more village-level education campaigns to change behavior among all community members. CRS’ 2017 Baseline Qualitative Findings Report on WASH in schools noted the preference for open defecation over use of latrines among students. However, our findings point more broadly to preference because of habit when it comes to continued open defecation, while the CRS report explained that students may not be using latrines because of lack of water, conditions of toilets, locked latrines, and the darkness of facilities. Quite possibly, these other factors may be underscoring student preference for open defecation, but we did not see such results in our FGDs and KIIs. School and community-level stakeholders varied in their responses regarding handwashing at critical moments, such as before eating, after using the latrines, and before preparing meals. Differing perspectives did not seem to fall along best-worst case school classification or type of respondent as opinions diverged even within FGDs. For example, while VEDC members in one community explained that diarrhea and other illnesses have decreased at school because students had incorporated handwashing into their daily practice prior to eating meals, teachers expressed doubt at another school that parents would enforce handwashing sufficiently so that students truly make this practice a part of their routine. A few teachers and cooks explained that students in Grades 4 and 5 washed their hands more frequently than students in lower grades because the younger children had difficulties fetching water or using the water pump. Students reported that they learned how and when to wash their hands from teachers and parents; almost no student explicitly mentioned using soap. Although this omission may not indicate that the children did not in fact use soap, it does match the low uptake of soap usage reported in CRS’ monitoring data. The two WASH schools in our sample either were currently receiving or had previously received WASH support from donors other than CRS. In terms of safe food preparation, cooks seemed to have strong knowledge of the importance of washing food, such as fruits, vegetables, and rice, prior to cooking and then cleaning the dishes and the kitchen afterward. However, because women from the community rotate cooking duties constantly, they expressed a need for more frequent training so that every cook would have a strong grasp of safety standards. Further, some cooks mentioned the lack of an indoor space in which to prepare food. Working outside, cooks face weather conditions that can have a detrimental impact on cleanliness, for example, when wind kicks dirt or even dry cow manure into food or water. As for storage, generally, school and community respondents at midterm as at baseline evaluation, recognized the value of having a separate storage area for food. One VEDC explained that it managed to Children in this area cannot keep clean. They do not take showers, and they drink water right from the hand pump water without boiling. They live not very clean. KII with principal at an average￾performing school (non-WASH) We do not have any difficulties in washing hands every time before meals as we wash hands with water that the cooks fetch from the village. FGD with Grade 4 and Grade 5 students at a low-performing WASH school LEAPS II Midterm Performance Evaluation Report  February 2020 45 mobilize the community to contribute wooden poles, walls, and labor to construct a storage house. The school principal in another district utilized a grant from the DESB to hire workers from the community to build a storage space for cooking pots and rice. VEDC across communities also noted plans to renovate and maintain existing storerooms. These examples demonstrate that school and community stakeholders largely understood the value of proper food storage, although the specifics of how to store food according to appropriate protocols did not arise in any of the FGDs and KIIs. 6.2.3 Increased Capacity after Trainings MoES stakeholders (PESS, DESB, and PAs) remarked that LEAPS II training has given them the skills and knowledge to develop appropriate monthly and yearly work plans grounded in practical and concrete steps. However, all MoES stakeholders complained about being able to monitor the schools to gather data. PESS noted the government must “be able to collect our own data efficiently” to effectively support schools. DESB technicians commonly said that they would like CRS to involve them in more school visits so they could learn how to collect data in both easy and challenging communities. They also said that more frequent training would be helpful forstaff to gain sufficient knowledge and skills. A DESB technician in one district with a low-performing school recommended that LEAPS II provide training specifically on the new Grade 1 curriculum, IE, and approaches to teaching in multi-grade classrooms. However, even despite having received training on monitoring, DESB technicians and PAs reported that they would not be able to support themselves financially to visit schools consistently without LEAPS II support (6.3.2 Monitoring for more details on monitoring). Capacity building for VEDCs included training on topics such as warehouse management, food safety, community engagement, and the value of education. One teacher at a low-performing school noted that the VEDC and CLVs had worked together effectively on the village education development plan. However, cooks in this same community reported that the VEDC had given them no support. Considering the varying involvement of VEDC with different actors, as well as the variation of VEDC commitment to school development and education, CRS has developed a VEDC rating tool to better understand and score VEDC management and understand how their engagement with schools can be improved. VEDC members themselves most commonly reported having learned how to encourage parents to send their children to school and support children to study at home. Across high-, average, and low-performing schools, respondents said that VEDC members had played an active role in responding to teacher and student absenteeism. At one high-performing school, teachers explained that VEDC members visit students’ homes to follow up on absences. However, mobilizing the VEDC can be difficult; as explained by one CM and reiterated by other stakeholders, some VEDCs have made little progress because other issues occupy their time, such as farming, so they remain less involved with education. One DESB officer suggested that LEAPS II expand its capacity-building efforts to include other organizations affected by school meals, such as the District Agriculture and Forestry Office and the District Health Office. Stakeholders considered the LB and IE trainings delivered to teachers and principals not only highly relevant (see 6.1 Relevance) but also effective in improving teaching and classroom management. Many teachers reported that they could now recognize when students might be distracted and respond by The most important thing is getting along well with the VEDC members because otherwise nothing gets done. They are the command center. KII with CM (CRS staff) 46 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report breaking lessons into shorter chunks; focusing on specific vowels, consonants, or words; and transitioning to more interactive practices, such as getting students to write on the board or creating discussion groups. Teachers also latched on to the IE trainings and the idea of creating a welcoming environment for all students. Although the “value-based pieces” of setting up public education campaigns and organizing classrooms to facilitate inclusion of all students have been incredibly well received, UO staff explained that some of the more technical IE concepts, such as task analysis, have been difficult for teachers to grasp. Thus, though teachers and principals showed, according to this respondent, tremendous excitement about educating all children regardless of disability or impediment, not all aspects of IE programming have been as easy to master. Acknowledging the shortcomings of IE trainings, and expanding upon the critique of adequate retention of training materials, PAs in almost all schools commented that teachers may be able to more effectively absorb the pedagogies taught, whether that be for IE or literacy acquisition, if trainings were longer and/or more frequent. One CRS partner concurred that the trainings provided some base knowledge but their design and structure did not necessarily incorporate sufficient practical classroom application of the pedagogical methods. Teachers confirmed that they had difficulty internalizing the training “because the training was too short, and none of the participants got a chance to practice.” However, even despite such complaints about the short duration, teachers generally liked the trainings and found them effective to start making small improvements in their teaching techniques. For principals, a CRS staff member reflected financial or leadership training might strengthen their ability to improve engagement between the school and the community. One principal pointed to the usefulness of a previous training with DESB staff on daily tasks for school management. 6.2.4 Learning and Stakeholder Dialogue Almost all stakeholders who received training said that LEAPS II activities had increased their knowledge and that, to some extent, they felt better equipped to carry out their job duties. Several PAs praised the project for teaching them how to give feedback to teachers politely without being overly critical, and teachers noted the value of learning from PAs. DESB technicians said that they had improved their interpersonal communication skills in ways that helped their own facilitation approaches when they conduct trainings. At the provincial level, an interview with the PESS representative revealed a desire for more field involvement to learn about and feel more connected to CRS project activities. VEDC members in different communities reported varying levels of interaction with CRS. In the community of one low-performing school, VEDC remarked that they have worked with CRS staff on the school meals program, awareness of hygiene practices, and rice storage room construction; VEDC members reported they feel comfortable speaking to CRS during visits from CRS staff to provide feedback on community needs. Meanwhile, interaction seemed more limited at a few high-performing schools where the VEDC only interact with CRS if necessary and such meetings happen infrequently. CMs explained that more effective learning might occur between the VEDC and LEAPS II if the project could bring together high￾functioning VEDCs with those VEDC that have not been as successful engaging with schools for dialogue about best practices. Although such exchanges occurred in the past, the CMs explained that coordinating such visits requires a significant budget, and therefore such endeavors have not occurred frequently. Teachers make teaching more fun and attractive to students, and the classroom is full of joy. KII with PA at a high-performing school LEAPS II Midterm Performance Evaluation Report  February 2020 47 Other partners generally relayed positive statements regarding their interaction with LEAPS II. WFP staff specifically noted that they had learned from CRS how to strengthen IE and community mobilization efforts. Both CRS and WFP staff members remarked on learning from each other to advance the broader goal of integrating school meals into a cohesive national policy; at present, the two organizations co-chair the School Meals Technical Working Group. In particular, this collaborative effort and increased dialogue can be helpful to pave a path toward sustainability of school meals (see 6.4.1 School Meals). 6.3 Efficiency When discussing the efficiency of LEAPS II, we mainly consider how the management structure and the monitoring system have met the needs of the project and ensure successful implementation. Responses from program and partner staff as well as government stakeholders indicated that generally LEAPS II has been running efficiently to encourage ongoing monitoring, learning, and timely decision-making. The oversight of project activities and monitoring of LEAPS II progress have benefited from the fidelity of implementation to established work plans. The following sections elaborate on the management structure and feedback from respondents regarding the design of the monitoring system. 6.3.1 Management Structure PESS and DESB representatives described close mutual partnerships with CRS staff. They said that these cooperative relationships have facilitated efficient execution of project activities in light of Lao PDR’s top-down structure, in which communities often mobilize only in response to government directives. DESB and VEDC representatives, along with some school principals, also remarked on CRS’ adaptive implementation and praised the project for including their perspectives and opinions to alter programming given the unique local context. For example, VEDC members in one community shared that the initial project design called for three permanent cooks, but the project conceded to a rotation proposed by the VEDC to account for the varying constraints on community members’ time. SCI also praised the initial design of LEAPS II and the workshop that CRS put together to brainstorm on activities, which included participation from the field and government. In discussing financial management, CRS staff spoke positively about the allocation of resources thus far, attributing some of the ease with the budget to lessons learned and procedures established during LEAPS I. A member of the project finance team stated that support from CRS’ regional office via online or in￾person trainings have been beneficial. 6.3.2 Monitoring CRS staff generally expressed positive sentiments about the monitoring system used to track LEAPS II progress, especially, as one CRS staff member noted, in light of the challenges collecting data when the government strictly regulates access. CMs stated that they regularly track cooking activities and proper commodity management procedures for storing and disbursing food. Despite this oversight from CMs, CRS staff did note that collection of cooking data currently lags by one month. To address this challenge, CRS staff explained the CMs would begin to do quick spot checks to keep numbers updated throughout the month. Community engagement in Laos is not a matter of choice. This is a system where authorities have a lot of influence and power. KII with SCI staff 48 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report At the community level, monitoring visits from CRS have encouraged more regular teacher attendance, according to a few FGDs with teachers. Some teachers and principals remarked that visits from project staff have also motivated students to attend school and pay attention. However, the VEDCs reported varying levels of interaction with the project, and in FGDs the members often requested more monitoring and check-ins so that they could deliver to LEAPS II staff their feedback more regularly. On the tracking of LB activities, DESB and PAs commended CRS for its support of their monitoring visits, and PAs specifically provided positive feedback on the project monitoring forms they use when checking on schools. However, SCI staff noted that, though PAs more frequently monitor schools, the quality of the visits and observations still needed improvement to assess whether teachers have carried over lessons learned from the trainings. 6.4 Sustainability In considering whether LEAPS II has laid the groundwork for the government and communities to continue activities after the project ends, the research questions first seek to understand the main types of support provided to schools and then consider whether these mechanisms are adequate to continue school meals and scale-up literacy education. As well, we provide notes where relevant on CRS efforts to ensure the sustainability of such activities. Broadly, all communities acknowledged the difficulty of mobilizing already-poor households to donate food items or cash. However, in those places where VEDCs seemed more active and engaged with the school, respondents noted that the community has been able to come together more effectively to contribute to school meals. As for literacy education, trainings delivered via local MoES stakeholders (DESB and PAs) seem to have provided long-lasting enhancements to teaching practices and reading camps have complemented well these teacher trainings. However, sustaining the reading camps may run into certain obstacles related to community resources, such as identifying an incentive structure so that CLVs can continue to lead these activities. We end this discussion of sustainability with our findings related to the maintenance of WASH facilities, such as school latrines, water points, and handwashing stations. We found that primary responsibility for this task varied considerably between the communities in our sample – however, respondents provided no reason as to why in some communities the VEDC assumes this task while in others it falls first to teachers and/or principals. We discuss this matter in more detail in 6.4.3 WASH and School Infrastructure. 6.4.1 School Meals All stakeholders acknowledged the importance of village contributions to school meals. VEDCs often said that part of their responsibilities was encouraging community members to donate supplementary food or collecting money from each household to buy food, so that cooks could prepare a more diverse menu. How well VEDCs managed this task varied. Only VEDCs at the high￾performing schools reported some success in collecting payment from community members. Even these VEDCs explained that certain households could not afford to pay the requisite amount. Therefore, money from the community was not always sufficient, and VEDCs could not count on receiving resources at set times. Respondents at average and low-performing schools relayed diverse opinions about the community’s capacity to contribute. At one average-performing school, cooks and the school principal expressed skepticism about the VEDC’s ability to mobilize villagers to contribute Villagers do not contribute locally grown food to the school meals because villagers themselves have nothing to eat at home. FGD with VEDC at an average￾performing school LEAPS II Midterm Performance Evaluation Report  February 2020 49 either money or food for school meals. At a low-performing school, teachers and cooks both noted that contributions have declined, but disagreed on the reason. Teachers explained that villagers felt more enthusiastic at the beginning of LEAPS II implementation, but then such sentiment dissipated, and alongside it a decrease in contributions. Cooks, however, explained the decrease differently – they noted villagers could not collect food from the forest during the dry season, and thus lower rates of contribution reflect seasonal variability. Although motivating the community to contribute ingredients has been difficult, it seems that VEDCs have managed to ensure a continuous supply of cooks to prepare the meals. Cooks said in FGDs that they accepted this duty out of a sense of obligation to the community, not because they enjoyed the tasks or had free time. Although some cooks noted appreciation for the THRs, they largely do not consider these THRs a strong incentive. Rather, as reflected in a statement from one cook in an FGD at a high￾performing school, “Every cook has to do it because it is community engagement.” Without a strong directive from the VEDC, as one principal at an average-performing school said, cooks might not show up. Thus, even though VEDCs cannot always strongly convince community members to supplement school meals, they seem to have enough authority to mobilize support for cooking the meals. This finding would indicate that community contributions may be difficult not because VEDCs cannot effectively mobilize such resources, but rather that households simply do not have the means. Poverty could also explain why certain communities may rely more on school meals as a safety net compared to slightly better off areas where parents can either contribute a greater diversity of food or provide well-balanced meals for their children outside of school. Few schools seem able to staff permanent, full-time cooks. Consequently, when women take turns cooking, the level of knowledge among them regarding safe food preparation varies based on whether their rotation happened to occur, or coincided with, project trainings. This variability can affect the frequency and quality of the meals prepared. To address such turnover, CRS administers periodic refresher trainings to cooks as well as on-the-job trainings through the CMs, although principals and teachers do not always uphold consistently the responsibility for monitoring the quality of meals. One CRS staff member also commented that they face a similar issue with volunteer storekeepers who are often transitory, and therefore time expended training them on recordkeeping might be futile as they may not stay in their position for very long. Considering the above challenges, CRS staff noted they would draw upon lessons learned from the handover of school meals in a portion of WFP schools to develop a sustainability plan for LEAPS II schools. WFP staff especially remarked on the necessity of integrating school meals into the national agenda, which CRS and partners have also noted as integral to continue school meals after LEAPS II ends because of the top-down order in the Lao structure. Additionally, several WFP staff members emphasized the importance of linking the school meals program to agricultural development, which could yield a larger quantity of food to support a full-fledged school meals program, and complement the current output from CRS-supported activities working with the We need to work closely with the local government. They need to be a key partner because they are the key to continuing to work with the communities. KII with CRS staff Where VEDCs are more active, cooks prepare meals better and more regularly. KII with CM 50 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report community on school gardens, fish ponds, and chicken coops. Other advice from WFP staff included early planning to embed handover into project design and involving various government ministries so that school meals become an all-encompassing issue rather than simply an education initiative. As mentioned in 6.2.4 Learning and Stakeholder Dialogue, CRS and WFP currently partner together on efforts to ensure that the national budget considers school meals as a line item so that the government can then allocate appropriate funding. To achieve this goal, CRS supports WFP efforts to establish a school meal handover decree and guidelines, and CRS participates in annual steering committee meetings with national MoES official, the National Assembly, provincial officials, and district officials to discuss program activities and sustainability. This effort also responds to the LEAPS II foundational results objective of an improved policy and regulatory framework, as reflected in the project’s ToC. Additionally, recognizing the centrality of VEDCs to successful school meal operations, CRS has taken several steps to strengthen these organizations so that they can more effectively assume operations without project support, such as the VEDC grants and reduction of THRs to motivate increased community contributions (See 1.1 Overview of Program Implementation for details). CRS also suggested that those communities with greater means and active VEDCs could be considered for an earlier transition under the sustainability plan. However, school and community-level stakeholders often expressed pessimism that school meals could continue without LEAPS II support given a limited budget for such an initiative and the poverty of community members. According to one DESB officer, the government ought to seek donors or funders to support school meals. The PA in this same district pointed out that aid organizations have always financially backed school meals. Although some respondents suggested turning to packed lunches, which would require working with parents, one DESB officer noted that this approach would still require project involvement to cover the costs of DESB visits to communities to encourage parents to prepare packed meals and to monitor the effort. Lastly, although THRs may be less of an incentive for cooks to conduct their jobs, principals and teachers explained that the THRs for teachers can be quite helpful, especially for volunteer teachers. Teachers also explained that with the THRs they feel more included in the project activities. 6.4.2 Literacy Education DESB representatives, including the PAs, stated that they support schools to improve literacy outcomes through trainings to enhance pedagogy. DESB technicians also reported that they have collaborated with VEDCs to work with parents toward the creation of home learning environments. At one high-performing school, the DESB officer described role-playing with mothers and fathers to help them understand practically how to encourage their children to study. This approach was somewhat more passive at a low￾performing school where the PA and a teacher explained that they train VEDC to encourage parents to hang posters on the wall and to create reading corners, rather than doing any kind of interactive training with parents. Hands down the government is going to have to allocate a budget to school meals for it to continue. KII with CRS staff LEAPS II Midterm Performance Evaluation Report  February 2020 51 Outside of this local MoES involvement, collaboration between communities and schools to enhance literacy education seemed to be somewhat limited. CLVs, drawn from the community, did run the reading camps, but as one SCI staff noted, identifying volunteers to participate who possess adequate literacy levels can be challenging because “education levels in these places can be quite low.” At one low-performing school, the CLV commented that the VEDC followed up only to check whether they organized reading camps each week because the VEDC needed to record this information in the ledger for CRS. One CM explained that VEDC members might not be particularly involved in education because they care more about enhancing livelihood opportunities. An SCI staff member remarked that perhaps sufficient appreciation of the reading camps could lead communities to find some means to continue these activities, but the transition to genuine volunteerism from stipends would be difficult. As well, even with the stipends paid to CLVs currently, they rotate in and out of the positions frequently when other commitments require their attention and time. This means capacity building of CLVs at one point in time may not be enough given that this role changes hands often. Other SCI ideas to maintain reading camps included utilizing resources from the village or school improvement fund, reducing the frequency of the camps, and collecting rice from community members to pay volunteers. SCI staff spoke more optimistically about teacher trainings, which one staff member cited as the most successful element of the LB activity because “the teachers get skills and knowledge from the project, and the skills will go with them to implement activities when we leave.” However, they cited limited funds and low capacity of DESB on staff management as obstacles to long-lasting and effective teaching improvements. As one PA explained, if PAs do not have the funding to follow up with schools, then they cannot monitor whether teachers have adequately absorbed the trainings. SCI staff and community respondents also noted that scaling up LB activities should entail adaptation to the realities of a multilingual culture. UO, who work on IE programming, noted that children who do not speak the Lao language often feel intimidated about going to school. Finding a long-term solution to ensure positive discrimination in multilingual communities is imperative. However, as SCI staff reflected, tailoring the curriculum to linguistic variations would go against the government preference for uniformity of classroom instruction. Complicating the ability of schools to pay more attention to students who struggle with Lao, teachers commonly complained, at both the baseline performance evaluation and midterm, about having to manage multi-grade classrooms. The DESB representative in a district with a high-performing school suggested that teacher trainings could have more impact if PESS worked with DESB to provide training on multi-grade teaching practices. 6.4.3 WASH and School Infrastructure VEDCs reported differing approaches to maintaining latrines, water sources, and handwashing facilities. At two schools, VEDC members stated that they rely primarily on teachers and students to maintain and clean these facilities. If they need repairs beyond what the school can manage, the principal reaches out After the program ends, we don’t know what support the community can provide to continue these literacy activities. KII with CLV at a low-performing school It seems impossible for the community itself to continue the [school meals and literacy] activity even though the project has provided trainings for VEDC and teachers. KII with DESB 52 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report to the VEDC to recruit community members for support. In one of these two schools, VEDC members noted that they meet with the principal annually to review the operational status of the facilities, especially the latrines. Among those VEDCs reporting to take on maintenance of WASH infrastructure, one VEDC explained they do so by utilizing a portion of the student enrollment fee. Because some families have struggled to afford this amount, schools sometimes can only collect a small amount; furthermore, the money fulfills several purposes in addition to maintaining school facilities. VEDC respondents across all schools frequently said that they support schools with infrastructure repairs beyond WASH facilities. Examples of collaborative construction initiatives to improve the school environment include fences to keep out animals, upgraded or new classrooms, storage facilities for food commodities, and covered eating spaces for students. However, teachers often stated that VEDCs might not act quickly enough to address school needs because of laziness or attention diverted to other activities, such as agriculture. Mothers, fathers, and students commonly complained about the condition of facilities and wanted improvements such as tables and chairs for all students, improved (non-dirt) floors, and playgrounds to encourage students to remain on campus. VEDC members, noting these same deficiencies, blamed villagers for being slow to contribute labor or cited insufficient support from the project to provide materials. 6.5 Perceived Impact Although the qualitative data cannot make definitive conclusions about program impact, stakeholder perceptions of changes brought about by LEAPS II indicate areas where the project may have had an effect while pointing out other areasthat may require attention. This section primarily focuses on changes in literacy and drop-out rates as a result of the literacy education and/or school meal interventions, but also addresses some of the broader intended and unintended impacts from the entirety of LEAPS II activities, and especially how such effects may have varied by gender in the targeted areas. The literacy education activities cited most frequently as having the greatest impact include the reading camps and teacher trainings; the former has been particularly helpful for letter recognition and generating excitement about reading while the latter has led to interactive and student-centered classroom pedagogies (see 6.1.2 Adequacy of Trainings to Address Knowledge Gaps). 6.5.1 Changes in Literacy Education Respondents at schools in our sample that received both the SF and the LB components generally praised the project and noted improvements in student literacy outcomes. Principals at all of these schools pointed out specifically that children now love to read because of the availability of non-textbook reading materials, such as storybook bags with books for students to borrow and take home. Two literacy activities received the most praise from respondents: reading camps and teacher trainings. Fathers in the FGD at a high￾performing school reported that children had become more [Reading camps] help students to love reading more and more. Students will benefit a lot when they love reading. KII with principal at a high￾performing school Parents are happy their children are able to read… They see the importance of education. It can improve the quality of life for their children. KII with CLV at a low-performing school LEAPS II Midterm Performance Evaluation Report  February 2020 53 creative because of the varied learning techniques at reading camps, such as games and singing. Mothers at another high-performing school commended the camps for teaching their children additional lessons because they could not themselves always provide support in this area. At one average-performing school, teachers pointed to the reading camps as having “the greatest impact.” The CLV in this community noted that access to books, games, and other activities in the summer especially helped students to practice reading and to sustain their learning. Communities perceived the LB approach to have influenced the home literacy environment. In one community with a low￾performing school, a few fathers stated that they hung posters of Lao consonants and vowels on the wall by the television, so that when their children watched TV, they could see the posters as well. Parents also reported that they sometimes sat with their children while the children read, even though they could not read themselves. Challenged to support their children because of long hours working, some parents reported that they have instructed older siblings to teach younger ones and/or invited children’s friends to their homes to study together. The CLV at one high￾performing school noted that some parents had built bookshelves for their children. Still, mothers and fathers remarked that they depended on teachers to educate their children and felt that they could not do much encouraging other than forbidding their children to watch TV after dinner. The success of literacy acquisition may rest upon other aspects of the school context. For example, one SCI staff member emphasized that the absence of pre-primary education would diminish any program impact. As he noted, the current system “sets up kids to fail from the start,” because the national curriculum assumes children enter primary school with the knowledge acquired from two years of pre￾primary, although pre-primary is often unavailable in rural areas of the country. According to this individual, even the revisions to the Grade 1 curriculum for the forthcoming school year would only “tinker with the system at the edges.” A further challenge to literacy beyond access to pre-primary education, lies in the complexity of the Lao language, which poses an obstacle not only for Lao speakers but especially for those students in remote areas who have little exposure to Lao prior to school. SCI noted that some of the language groups in Lao differ quite drastically from each other and often have different sounds entirely, and this “wide spectrum” of language differences complicates effective learning and literacy acquisition. In this context, SCI remarked that relying on the textbook alone to develop students’ oral and written language skills would be wholly insufficient, and the reading camps can be particularly helpful to target those gaps in classroom instruction. 6.5.2 Attendance and Drop-out Rates Despite the many complaints about the sameness of school meals, respondents overall did say that the meals have been critical in supporting children to attend school. However, opinions also diverged. For example, cooks at one low-performing school noted that attendance had not changed since they started …Sometimes even [if] illiterate, parents must sit and listen to the children reading. FGD with fathers at a low￾performing school There is no one, child or adult, who can read and understand a text in a language they do not speak. I could not do it. You could not do it. Nobody could do it. KII with SCI staff 54 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report preparing school meals, but the VEDC and principal in this same community reported that the lunches have encouraged parents to send their children to school. Second, considering the entirety of LEAPS II interventions and not just school meals alone, respondents tended to report lowered school dropout rates and increased class attendance because of project activities. The DESB representative in one district said that the district’s dropout rate had decreased because of reading camps and school meals, which “motivate students to come to school.” However, the DESB representative in another district commented that LEAPS II interventions can help only slightly to increase student attendance but did not elaborate on why. Potentially, as pointed out by teachers in an average-performing school, students may drop out after repeating grades because they feel uncomfortable being older than their classmates. According to DESB at one high-performing school, this issue of grade repetition may arise because of multi-grade classrooms, insufficient teaching materials to attract students, and irregular school attendance. See 6.2.1 Effect of School Meals on Attendance and Attentiveness for more information on the effectiveness of school meals to motivate attendance. 6.5.3 Unintended and Differentiated Impacts An unintended impact of school meals may be increased social interaction among children. Parents, teachers, and VEDC members frequently connected eating at school to playing with friends. As mothers at one average-performing school explained, once their children ate at school, then they stayed to play. Other respondents echoed this observation. A few respondents of different types remarked that LB activities increased children’s confidence. A principal at an average-performing school said, “Children are not shy anymore, and they dare to speak with strangers.” A PA in another district observed that, as he conducted assessments at schools, children more readily answered questions, whereas before, these students felt more hesitant to speak and did not understand much Lao. Much of the commentary regarding gender focused on roles and responsibilities rather than on differing effects of project activities. For example, only women cooked the school meals, so they received most of the food safety training. However, a few program partners noted that women in non–Lao-dominant communities struggled with communication, a fact that could affect the quality and impact of the trainings. Principals, parents, and VEDC members commented on the roles of girls and boys, explaining that some families keep girls at home to take care of younger siblings or encourage boys to go hunting instead of attending school. When asked why only girls assume household responsibilities, a principal explained, “Girls know how to deal with these things, while boys do not.” Despite such statements, parents tended to say that families should treat children equally regardless of gender, emphasizing the importance of educating both girls and boys because they have the same opportunities to achieve success. Consequently, though some families may consider gender in deciding which of their children to send to school, the overall sentiment among parents remains that all children should be educated. This sentiment underscores the lack of For some people, school lunches have little meaning, but for vulnerable children it means much more because they can come to school and learn. KII with CRS Staff (CM) Boys and girls are not different. They should learn what they like because it depends on their talent and ability to learn. FGD with fathers at a low￾performing school LEAPS II Midterm Performance Evaluation Report  February 2020 55 gender differentiation in our survey data when examining attendance rates as well, monitoring data from LEAPS II showed no difference between boys and girls in those attending 80 percent of school days (the pass rate). 56 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report SECTION 7. CONCLUSION This midterm performance evaluation of LEAPS II in seven districts of Savannakhet province in Lao PDR highlights changes in outcomes related to the core program objectives through a mixed-method approach and sets the stage for a full examination of program effects at the endline evaluation. To examine LEAPS II performance over time, we followed the same performance evaluation methodology and sampling strategy used for the baseline evaluation to generate an appropriate comparison that adheres to USDA guidelines. We based our findings on data collected from: 1) student surveys and LBRA scores, 2) classroom observations and 3) FGDs and KIIs with school- and community-level stakeholders (including MoES officials) and LEAPS II project partners. We triangulated qualitative findings with survey data and classroom observations to provide contextual information for the quantitative results, where applicable. In this section, based on the analyses of all the data sources, we discuss the limitations of this study, describe the implications of key outcomes for the next two years of LEAPS II, and finally, put forth recommendations to enhance programming and future evaluation efforts. 7.1 Limitations This study faced a few limitations in the evaluation design and analysis. 3.4 Field Challenges provides details on challenges specific to data collection. Reliance on self-reported data. The quantitative approach relies on self-reported data from children for a number of socially and culturally sensitive subjects such as food security or health-related absences from school. The data should be interpreted with caution because it is particularly susceptible to social desirability bias and we have young children, especially in Grade 1, that are not always emotionally and cognitively able to answer survey questions effectively. To minimize this limitation and improve data reliability, we devoted considerable attention to cognitive testing of the survey instrument with children in Grades 1–5 before the baseline performance evaluation in 2017. In consultation with our data collection partners, we adjusted question phrasing to make sure children could understand the questions and feel comfortable answering. The qualitative research reflects individual perspectives, which are subject to bias and preconceptions. Gathering data from mothers proved particularly challenging. In non-Lao communities, women often struggled to present their opinions about education. Our experienced interviewers dedicated significant time to explain the questions to these mothers as needed. Another limitation, also evident in the baseline evaluation, is that parents who are motivated to support their children’s education may be more likely than others to participate in a FGD. Thus, enthusiastic responses from parents about the importance of education should be interpreted with caution. Absence of electronic class lists. In the absence of electronic class lists, enumerators had to conduct random sampling from the students present on the day of data collection, rather than from the full roster of enrolled students. Selecting only students who are present risks sampling bias by excluding information on children who are more likely to be absent. Subjectivity of classroom observations and lack of comparability. Snapshot observations measure a specific variable or indicator—in this case, student attention—at a specific point in time. They do not support conclusive statements about whether an intervention—in this case, school meals—caused observed changes. In addition, the subjective nature of the tool could create inconsistencies in findings due to the inherently difficult job of making observational judgements across changing settings. For example, if one observer observes one classroom with the same teacher at baseline and midterm, the LEAPS II Midterm Performance Evaluation Report  February 2020 57 teachers might have changed their approach from baseline to midterm and make the comparison challenging between baseline and midterm. Observers were also not the same at midterm. Classroom observations thus are more useful for learning about one point in time than for comparing two points in time. 7.2 Key Outcomes and Implications Triangulation of the quantitative findings in Section 5. Quantitative Results on observed changes in outcomes since baseline (2017) with the qualitative findings in Section 6. Qualitative Results on the relevance, effectiveness, efficiency, impact, and sustainability of LEAPS II reveals several important outcomes at this stage of implementation, which have valuable implications for the theory of change. Overall, the performance evaluation data suggest that the project is moving toward its intended objectives, though achievements in some areas have been limited. IMPAQ outlines below the implication of the midterm evaluation outcomes regarding the LEAPS II theory of change. ▪ Health-related absences. Among students who reported that they were sick in the last week, a significantly lower proportion at midterm than at baseline reported having missed school. This finding was consistent with reports from FGDs and KIIs noting that illness no longer poses a large hurdle to school attendance. For example, at one low-performing school, a teacher explained that “the big issue of sickness,” often arising from unsafe food, has begun to improve. However, this outcome should be interpreted with caution as health related absences are usually subject to social desirability bias. In addition, there are different factors that could affect students’ health related absences that could go beyond the control of the program such as endemic or epidemic diseases. ▪ Short-term hunger; increased access to food. Significantly, more students reported at midterm than at baseline that the school had served food; almost all had eaten the meal. Consistent with this finding, self-reported hunger fell significantly from baseline to midterm, with no gender differences. Most parents expressed positive views on schools’ provision of midday meals, but they wanted to see more variety in the foods served. Interviews with students also showed that they feel they can better concentrate in class because they worry less about their next meal. ▪ Attentiveness. Although students reported being less hungry at midterm than at baseline and stated being less worried about their next meal in the qualitative interviews, their attentiveness (measured by classroom observations) dropped significantly. The observational data did not show any gender differences. These varied findings could perhaps reflect different moments in time for when the data (observations, quantitative, and qualitative) were collected from different sampled respondents. It is also noteworthy that IMPAQ only visited seven schools for the qualitative interviews. Further, classroom observations indicate that teachers spent more time on class management than teaching, which could potentially lead to a lower attentiveness rate on the day of their visit at midterm. However, the observation tool remains a subjective measurement and given the many potential unique conditions of each classroom setting on a particular day, midterm results cannot be compared with baseline. The qualitative notes suggest several factors could influence how well students pay attention in class, including unappealing teaching materials and ineffective classroom management of multi-grade classes. ▪ Literacy. The midterm data show significant improvements in students’ literacy skills, especially foundational skillssuch as letter recognition, with no gender differences. Although the proportion of students who were identified as readers did not change from baseline to midterm, the 58 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report comprehension level of those readers improved significantly. Achievements in this area may be lagging behind the project’s midterm targets, but the results are consistent with Lao curriculum expectations of gains in reading comprehension becoming more apparent from Grade 3 onward. SCI and other LEAPS II partners remarked on the difficulty among young children to grasp fully the Lao language, and such challenges present particularly acute obstacles for those students from ethnic backgrounds who often have little exposure to Lao prior to school. As described in Section 4. Evaluation Sample, the midterm evaluation sampled different individuals from the baseline study. Comparing mean outcomes at baseline and midterm using t-tests can suggest correlations between the observed changes and LEAPS II interventions but cannot determine that the interventions caused the changes. 7.3 Recommendations Considering the key outcomes and the limitations, IMPAQ presents below some recommendations along with the lessons learned to improve LEAPS II and the project evaluation in the second half of the project. 7.3.1 Project Recommendations The following recommendations can help to enhance the effectiveness and sustainability of LEAPS II. ▪ Work with MoES to provide teachers and other educators with pedagogical strategies that help them respond to multi-grade classrooms, when needed, and develop materials to better integrate these training concepts into national teaching guidelines. According to findings from the qualitative research, teachers reported feeling poorly equipped to handle multi-grade classrooms, and that adapting some of the current trainings have been more difficult in this context. The literature also reiterates this challenge and often links teachers’ inability to handle multi-grade classes to poor student learning outcomes.39 Provision of more trained teachers to schools to address this known issue permanently might not be realistic given limited MoES budget. However, LEAPS II can address this need by collaborating with MoES to integrate pedagogical strategies into trainings for teachers and other educators, such as PAs and principals, to be able to respond to multi-grade classrooms. To develop guidelines for teachers on best practices in multi-grade settings, CRS could facilitate peer-to-peer exchanges among teachers in districts and provinces in Lao PDR or with neighboring countries, as recommended in a previous study of multi-grade teaching practices in Southeast Asia.40 For sustainability of trainings, CRS should continue to work closely with MoES and other relevant stakeholdersto ensure MoES training materials for teachers reflects the training concepts on how to teach in multi-grade classrooms. Also useful would be to distill such documents into simple handbooks or handouts so that teachers can use as a reference whenever needed. ▪ Continue building the capacity of DESB and PAs in financial management so that these local MoES officials can better guide teachers, monitor teaching quality, and deliver refresher trainings. Besides increasing knowledge of teaching concepts, given the challenge as reported by PAs and DESB of sufficient budget for monitoring visits, and the importance of such follow-up support to teachers, CRS could consider whether financial management trainings for DESB may 39 Thephavongsa, Souphanh. 2018. Enhancing the teaching skills of the multi-grade teachers through lesson study. International Journal of Learning, Teaching and Educational Research, Vol. 17, No.4, pp.71-87. UNICEF. 2015. Student Learning Outcomes in Primary Education in Lao PDR: Situational Analysis. 40 Adam Smith International. 2017. Rapid Appraisal of Literature on Multi-grade Teaching in ASEAN and Other Countries. LEAPS II Midterm Performance Evaluation Report  February 2020 59 be beneficial. Additionally, LEAPS II could collaborate with DESB and the schools to develop more appropriate and feasible work plans with achievable targets for monitoring taking into account the phase out of support. ▪ Reinforce engagement with VEDCs to boost community ownership of project activities. To ensure sustainability, LEAPS II must continue to engage VEDCs and build their capacity. One avenue is collaboration with DESB and other government entities to enforce VEDC fulfillment of roles and responsibilities because of the power of such directives in the Lao context. Another pathway is for LEAPS II to continue to engage with VEDCs to provide consistent and regular hands￾on learning on management of project activities. This training could help VEDCs increase their involvement to ensure successful handover of LEAPS II activities. An examination of similar school meal activities has shown that incremental involvement of local actors helps to sustain such initiatives.41 ▪ Ensure the effectiveness of any O&M trainings so that communities understand how to maintain and manage essential WASH facilities for improved health and hygiene practices. As planned, CRS has drilled boreholes in select target communities (see 1.1 Overview of Program Implementation 1 for more details on the WASH component of LEAPS II). Mothers in one such community said they were happy with the effort to provide water at their children’s school. However, any construction completed should be accompanied by regular support to help the communities maintain and repair the facilities.42 In each community, LEAPS II should identify who the relevant individuals or groups may be (usually the VEDC, the school principal, or a teacher) and target its trainings appropriately. CRS should clarify after its trainings the ways in which schools can seek help if they themselves cannot repair essential facilities. As well, the project should also establish an efficient system for monitoring both the WASH equipment and how well the beneficiaries have absorbed the lessons on O&M. Broadly, CRS could consider conducting a follow up qualitative study to its 2017 WASH in Schools: Behavior and Sustainability in Lao PDR report, which could examine issues such as maintenance of facilities, hygiene practices, and any improvements to the availability of water sources. ▪ Raise parent and community awareness of the importance of WASH practices in parent engagement workshops and school open houses. Besides infrastructure issues, long-established habits have hampered hygiene improvements. The project’s focus on school-level changes has not yet addressed practices at home and in the community. According to the qualitative findings, some students (like their parents) use outdoor areas as toilets because they lack latrines at home. LEAPS II could seek to align its WASH component with other donor projects that concentrate on community-wide activities so that students receive the same messages about health and hygiene at home and at school. LEAPS II could consider incorporating some key WASH messages into a presentation at school during open houses or in trainings where the project engages directly with parents, such as the SCI led parent engagement workshops. Since some of the solutions may require long-term actions, for any future iterations of LEAPS II, the program could consider the following alterations: ▪ Explore additional sustainable pathways to supplementing and diversifying school meals. Though LEAPS II has been successful in providing school meals, stakeholders expressed concern about sustainability. Government and VEDC respondents stated that they lack the financial and 41 The Partnership for Child Development. 2016. Global school feeding sourcebook: lessons from 14 countries. Leslie Drake ed. Imperial College Press, London. 42 UNICEF. 2015. Evaluation of the WASH Programme of the UNICEF Country Programme (2012-2014). 60 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report technical capacity to sustain school meals, and the primary challenge for VEDC remains mobilizing poor households to contribute regularly to the school meals. Beyond scaling up existing efforts to develop school and community income-generating activities, such as school gardens, fish ponds, or chicken coops, to help schools source ingredients and generate income, LEAPS II could consider in the long run various pathways to address poverty in the community. For example, if CRS could expand its scope to conduct activities beyond the school, then LEAPS II could establish Saving and Internal Lending Communities (SILCs). IMPAQ evaluations of MGD projects in Mali and Burkina Faso have shown that SILCs can be helpful to mitigate poverty by raising household incomes; thus, they can also be used as a conduit for community contributions to school expenses.43 ▪ Create custom indicators for reading comprehension. Given the complexity of the Lao language, gains in reading with comprehension in Lao PDR typically begins in Grade 3, but it is only in Grade 5 that the majority of students are able to read with fluency and comprehension.44 Our findings also confirmed the evidence from the existing literature showing students in Grade 2 have low levels of reading with comprehension, especially among non-Lao students. Thus, to get a more accurate picture of the effect of project interventions on students’ reading ability, we classified our recommendations into three parts for LEAPS II and any future studies of this project. Quick Win adjustments are addressable immediately within the current budget and data. LEAPS II could mainly rely on foundational literacy skills (e.g., letter and word recognition) when evaluating Grade 2 students’ learning needs rather focusing on their reading comprehension outcomes. The IMPAQ team could also consider the revised GoL definitions of reading competencies based on the forthcoming changes to the Grade 1 curriculum, if those benchmarks are available before the final evaluation. Medium effort changes require more effort or cost than quick win but are important to consider for bridge or future projects. Before the end of LEAPS II, the program could include a new pilot study to assess the existing reading competencies, but for students in Grade 3. With a modified budget and SOW, IMPAQ could also support LEAPS II in conducting this pilot study and expand the assessment to other grades using the Annual Status of Education Report (ASER). IMPAQ can develop one ASER tool to use consistently for all grades and assess students’ reading levels. IMPAQ has successfully developed and administered this tool in other MGD programs for multiple grades in Mali, Senegal, Ivory Coast, and Burkina Faso as an alternative to EGRA or LBRA. Long-term strategies to help future studies beyond LEAPS II may require more time, effort, and cost to implement. To capture all the anticipated GoL revisions to the national curriculum, CRS could expand the pilot study described above with Grade 3 students and include other grades to the extent possible. In addition, if LEAPS II enters a third phase, CRS could consider an impact evaluation to study the effect of new curriculum revisions on reading competencies. 7.3.2 Evaluation-Specific Recommendation Below, IMPAQ provides two additional recommendations that would require additional funds for the evaluation: 43 IMPAQ International. 2017. McGovern-Dole International Food For Education and Child Nutrition Program: Beoog Biiga II Midline Performance Evaluation Report. 44 The World Bank. May 2016. Reducing early grade dropout and low learning achievement in Lao PDR. Retrieved from: http://documents.worldbank.org/curated/en/161641483590757065/pdf/111627-REVISED-PUBLIC-Lao-PDR-Root-Causes-of￾Early-School-Leaving-f.pdf LEAPS II Midterm Performance Evaluation Report  February 2020 61 ▪ Explore options to gain a stronger understanding of VEDC capacity and operations. Because VEDCs are important in facilitating community-level changes, their commitment is vital to sustainability. Although CRS’ annual VEDC rating tool provides some indication of VEDC success, a VEDC survey could reveal more, not just about how the VEDC performs, but also about the backgrounds of individual members, personal and group priorities and motivations, and constraints on the VEDC’s ability to muster community support for schools. Alternatively, IMPAQ could conduct in-depth qualitative interviews at endline with a greater number of VEDCs using the VEDC rating tool to sample best, average, and worst VEDCs, which could then shed light on the factors for success. Although CRS could conduct separately its own study of VEDC operations, an external evaluator who does not represent LEAPS II may be able to elicit more open dialogue with the VEDC members. ▪ Observe and complete a checklist to determine the functionality of school canteens and WASH equipment (handwashing stations and latrines). Self-reported data are often susceptible to social desirability bias, and thus direct observations would be useful to provide reliable information on food storage facilities and the presence of handwashing stations and latrines. For example, a large percentage of students in some schools reported that cooks had not served meals by early afternoon. Collecting additional data would reveal whether this might be happening in schools without functional canteens, with low quantities of food, or with broken handwashing stations. Having this “checklist” to assess the status of certain facilities may be helpful in complement to our qualitative and quantitative data on school meals and health and hygiene. Depending on the timing of the final evaluation and the dates of data collection, IMPAQ could observe meal preparation if school canteens are operational. 62 IMPAQ International, LLC  LEAPS II Midterm Performance Evaluation Report APPENDICES A. References B. LEAPS II Evaluation Questions and Conceptual Framework C. McGovern Dole Indicators D. List of Stakeholder Groups E. Additional Tables and Complementary Outcomes F. Other Subtests of Reading Assessment G. Regression Analysis H. Inter-rater Reliability I. Survey Instruments J. Qualitative Protocols K. Midterm Evaluation SOW and TOR A1 LEAPS II – Final Impact Evaluation Report  February 2020 APPENDIX A. REFERENCES Adam Smith International. 2017. Rapid Appraisal of Literature on Multi-grade Teaching in ASEAN and Other Countries. BEQUAL. n.d. Library. BEQUAL Website. Accessed June 28, 2019. http://www.bequal-laos.org/launch-of￾the-in-service-training-for-the-new-grade-1-curriculum/ Catholic Relief Services. 2016. Learning and Engaging All in Primary School (LEAPS): Final Evaluation Report. Dowd, A.J., Pisani, L. & Borisava, I. 2016. “Evaluating Early Learning from Age 3 to Grade 3” in Understanding What Works in Oral Reading Assessments. Montreal: UNESCO Institute for Statistics (UIS) Duflo, E., Hanna, R., and Ryan, S. P. 2012. Incentives Work: Getting Teachers to Come to School. American Economic Review, Vol. 102, NO.4. Fleiss, J. L., & Cohen, J. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 613-619. Hess, R. D. & Holloway, S. D. 1984. Family and school as educational institutions. Review of Child Development Research, 7, 179-222. IMPAQ International. 2017. McGovern-Dole International Food For Education and Child Nutrition Program: Beoog Biiga II Midline Performance Evaluation Report. Kim, Y. S. 2009. The relationship between home literacy practices and developmental trajectories of emergent literacy and conventional literacy skills for Korean children. Reading and Writing, 22(1), 57-84. Lonsdale, M. 2003. Impact of School Libraries on Student Achievement: A Review of the Research (ISBN￾0-86431-6976) New curriculum in Laos. 2018, April 6. Retrieved June 29, 2019, from https://www.letlaoslearn.org/news/new-curriculum-to-improve-quality RTI International. (2015). Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC. Save the Children. 2016. LEARN Literacy Boost Baseline Assessment: Lao PDR. Seel, A. et al. 2015. Village Education Development Committees in Lao PDR: Their functionality and Impact. Stallings, J. and G. Mohlman. 1988. Classroom Observation Techniques. In Educational Research, Methodology, and Measurement: An International Handbook. Edited by J. Keeves. Pergamon: Oxford. Thephavongsa, Souphanh. 2018. Enhancing the teaching skills of the multi-grade teachers through lesson study. International Journal of Learning, Teaching and Educational Research, Vol. 17, No.4, pp.71-87. The Partnership for Child Development. 2016. Global school feeding sourcebook: lessons from 14 countries. Leslie Drake ed. Imperial College Press, London. A2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report UNICEF. 2015. Evaluation of the WASH Programme of the UNICEF Country Programme (2012-2014). UNICEF. 2015. Student Learning Outcomes in Primary Education in Lao PDR: Situational Analysis. World Food Programme. 2018. Decentralized Evaluation of the USDA McGovern-Dole FY14 Endline Evaluation in Lao PDR [FY 14-16]. The World Bank. 2016. Reducing early grade dropout and low learning achievement in Lao PDR. LEAPS II – Final Impact Evaluation Report  February 2020 B1 APPENDIX B. LEAPS II EVALUATION QUESTIONS AND CONCEPTUAL FRAMEWORK Exhibit 28. Key Evaluation Questions Question Data Analysis Impact Progress on Project’s Key Indicators • To what extent, has students’ level of reading and understanding of second-grade-level text changed? • To what extent, has the students’ ability in identifying letters changed among second graders? • To what extent, has the level of students’ attentiveness changed in classroom? • To what extent have the levels in prevalence of afternoon hunger changed among students? • Has the levels in student health-related absences changed? LBRA (only Grade 2) Classroom Observations (Grade 1-5) Student survey (Grade 1-5) Perceived Impact • What has been the impact of the program, in terms of both intended results and changes? Did the impact vary for different groups (e.g. boys vs. girls, ethnic vs. Lao communities, etc.) within the targeted areas? If so, why? • Did school feeding and educational quality interventions have the intended impact on literacy of school-age children? • What changes in literacy and drop-out rates have been observed as a result of the Literacy Boost and/or school feeding interventions? To what extent are these changes attributable to Literacy Boost program and/or school feeding? Qualitative Interviews Relevance • Do stakeholders feel that their voices are heard and their needs considered throughout the project? Are stakeholders aware of how to share information with CRS? • To what extent is school feeding a relevant incentive for parents to send their children to school? • To what extent is the program’s take home ration (THR) component a relevant incentive for teacher attendance, or support from cooks/storekeepers? How important is the distinction between no take home ration, a partial ration, and a full-sized ration as an incentive? • How well do teacher and administrator trainings organized by the LEAPS project support teachers to address the issues they face in their schools and communities? Are the topics being offered relevant to their needs? • To what extent are the trainings organized by the LEAPS project in health/hygiene, safe food prep/storage practices, and clean water/sanitation services addressing pre-existing knowledge gaps? • Did the project staff properly identify school/community priority needs? To what extent has the program design and intervention met the needs of the participants, including students, teachers, school administrators, and parents? Qualitative Interviews B2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Question Data Analysis • To what extent are project activities aligned with Government of Laos priorities and policies? If they contrast, why? • According to parents and students, how appropriate/acceptable are the food items, the timing of the meal, and the food preparation? Effectiveness • To what extent has the training of teachers led to improved teaching practices? To what degree are objectives related to improved quality of instruction likely to be achieved by the end of the project? • To what extent what have capacity strengthening activities contributed to the implementation of Inclusive Education (IE) policies and programs? • To what extent have WASH interventions contributed to changes in the use of health and dietary practices? To what degree is the objective of increased use of health and dietary practices likely to be achieved by the end of the project? To what extent are schools demonstrating improved knowledge of health and hygiene practices? • To what extent are cooks and storekeepers displaying improved knowledge of safe food preparation and storage practices? To what degree is the objective of increased use of health and dietary practices likely to be achieved by the end of the project? • To what extent are schools achieving increased access to clean water and sanitation services? • To what extent is school feeding an effective incentive for parents to send their children to school? What has been the resulting effect on attendance? • To what extent does school feeding improve student attentiveness? To what degree is the objective of improved attentiveness likely to be achieved by the end of the project? • In what ways is the project increasing the capacities of MoES at various levels, teachers, administrators, and VEDC/communities? • To what extent is learning occurring between the various partners? What is the value-added of the government-NGO-academic partnership model? Are the learning mechanisms effective in facilitating learning between all? Qualitative Interviews Efficiency • How efficient are the management and staffing structures for project implementation, monitoring, learning, and timely decision￾making? • Was the monitoring system designed efficiently to meet the needs and requirements of the project? Qualitative Interviews Sustainability • What types of support do communities, including parents and VEDC members, provide to support schools, including school meals? Based on current practice, which type(s) of support are most commonly provided by communities? • What mechanisms are in place for the local stakeholders (including VEDC) to continue school feeding after the duration of the project, and with high standards for health/dietary practices? How well did the VEDCs and cooks incorporate locally-available foods into school meals? What mechanisms are and/or need to be in place for MoES staff to support and scale-up literacy education in LB schools upon completion of project support? • What, or who, incentivizes VEDCs and water user committees (WUCs) to sustain/maintain school latrines, water points, and handwashing facilities? Qualitative Interviews Source: CRS Midterm Terms of Reference LEAPS II – Final Impact Evaluation Report  February 2020 B3 Exhibit 29. LEAPS II Conceptual Framework SO1: Improved Literacy of School-Age Children 1.1: Improved Quality of Literacy Instruction 1.2: Improved Attentiveness 1.3: Improved Student Attendance 1.3.5: Increased Community Under￾standing of Benefits of Education 1.3.4: Increased Student Enroll￾ment 1.3.1: Increased Economic and Cultural Incentives (or Decreased Disincentives) 1.2.1: Reduced Short￾Term Hunger 1.1.3: Improved Literacy Instruct￾ional Materials Activities: 1.1.1.1. Teacher recognition Activities: 1.2.1.1.1, 1.3.1.1.1. Provide school meals (provision) 1.2.1.1.2, 1.3.1.1.2. Take Home Rations (provision) 1.2.1.1.3, 1.3.1.1.3. Establish school gardens Activities: 1.3.3.1. Building/Rehab: kitchens 1.3.3.2. Building/Rehab: warehouses and storerooms Activities: 1.3.4.1. Enrollment Campaign Already accomplished under current award 1.1.1: More Consistent Teacher Attend￾ance 1.1.2: Better Access to School Supplies & Materials 1.3.3: Improved School Infra￾structure 1.3.2: Reduced Health￾Related Absences 1.1.4: Increased Skills and Knowledge of Teachers 1.1.5: Increased Skills and Knowledge of School Admin￾istrators 1.2.1.1, 1.3.1.1: Increased Access to Activities: Food (School Feeding) 1.1.3.1. Production of books and supplementary Activities: 1.1.2.1. Establish libraries Activities: 1.1.4.1. Training: Teachers Activities: 1.1.5.1. Training: School Administrators SO2: Increased Use of Health and Dietary Practices* *see next page for activities 1.4.1: Increased Capacity of Government Institutions 1.4.2: Improved Policy and Regulatory Framework 1.4.3: Increased Government Support 1.4.4: Increased Engagement of Local Organizations and Community Groups Activities: 1.4.1.1. Capacity building: local, regional, national level (LB, Parent￾Teacher Assoc., training modules, IE Assessment/Action Plan) Foundational Results The Aus-DFAT BEQUAL program will work on this, particularly in relation to the revised curriculum Activities: 1.4.3.1. Provide school meals (advocacy) 1.4.3.2. Take Home Rations Activities: 1.4.4.1. Establish activities to promote literacy 1.4.4.2. Training: Parent-Teacher B4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report This was not identified as a priority within the target geography area. SO2: Increased Use of Health and Dietary Practices 2.1: Improved Knowledge of Health and Hygiene Practices 2.6: Increased Access to Requisite Food Prep and Storage Tools and Equipment 2.4 Increased Access to Clean Water and Sanitation Services 2.5: Increased Access to Preventative Health Interventions 2.2: Increased Knowledge of Safe Food Prep and Storage Practices Activities: 2.1.1. Extra￾curricular activities Activities: 2.2.1. Training: Food preparation and storage practices Activities: 2.4.1. Building/Rehab: Wells and water stations/systems (provision) 2.4.2. Building/Rehab: Latrines This was not identified as a priority within the target geography area. 2.3: Increased Knowledge of Nutrition This will not be focused on directly, although the project does expect to communicate nutrition messaging through its use of school gardens. 2.7.1: Increased Capacity of Government Institutions 2.7.2: Improved Policy and Regulatory Framework 2.7.3: Increased Government Support 2.7.4: Increased Engagement of Local Organizations and Community Groups The project will focus more on government capacity under SO1 and leave this result to health/nutrition forums Foundational Results The project will focus more on government capacity under SO1 and leave this result to health/nutrition forums The project will focus more on government capacity under SO1 and leave this result to health/nutrition forums Activities:2.7.4.1. Building/Rehab: Wells and water stations/systems (WUC) LEAPS II – Final Impact Evaluation Report  February 2020 C1 APPENDIX C. MCGOVERN DOLE INDICATORS Exhibit 30. McGovern Dole Indicators Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Standard #26 Evaluation LBRA 3% 11% 26% Percent of students who, by the end of two grades of primary schooling, demonstrate proficiency in identifying letters Custom Evaluation LBRA 47% 73% 55% Number of individuals benefiting directly from USDA-funded interventions Standard #27 CRS/ Monitoring CRS 0 62,983 91,833 Number of individuals benefiting indirectly from USDA-funded interventions Standard #28 CRS/ Monitoring CRS 0 314,915 459,165 Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools Standard #5 CRS/ Monitoring CRS 0 229 518 Percent of teachers in attendance on day of school visit at USDA supported schools Custom CRS/ Monitoring CRS 88% 90.01% 95% Number of textbooks and other teaching and learning materials provided as a result of USDA assistance Standard #2 CRS/ Monitoring CRS 0 45,658 84,052 C2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Number of schools with improved literacy instructional materials as a result of USDA assistance. Custom CRS/ Monitoring CRS 0 110 180 Number of teachers/educators/teaching assistants trained or certified as a result of USDA assistance Standard #6 CRS/ Monitoring CRS 0 767 590 Number of school administrators in targeted schools who demonstrate use of new techniques or tools Standard #3 CRS/ Monitoring CRS 0 241 274 Number of school administrators and officials trained or certified as a result of USDA assistance Standard #4 CRS/ Monitoring CRS 0 424 280 Percent of students who are attentive in the classroom Custom Evaluation Classroom Observation 84% 60% 92% Percent of students reporting that they are 'somewhat' or 'very' hungry during their afternoon class Custom Evaluation Student Survey 7% 4% 5% Number of school-aged children receiving daily school meals (lunch) as a result of USDA assistance Standard #16 CRS/ Monitoring CRS 0 55,861 86,049 Number of daily school meals (lunch) provided to school-age children as a result of USDA assistance Standard #15 CRS/ Monitoring CRS 0 9,487,535 32,705,154 Number of take-home rations provided as a result of USDA assistance Standard #13 CRS/ Monitoring CRS 0 42,059 94,177 Number of individuals receiving take-home rations as a result of USDA assistance Standard #14 CRS/ Monitoring CRS 0 4,090 2,694 LEAPS II – Final Impact Evaluation Report  February 2020 C3 Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Number of people trained in child health and nutrition as a result of USDA assistance Standard #18 CRS/ Monitoring CRS 0 84 134 Number of students (males/females) regularly (80%) attending USDA supported classrooms/schools Standard #1 CRS/ Monitoring CRS 0 54,743 79,165 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance Standard #17 CRS/ Monitoring CRS 0 60,564 88,753 Percent of students in target schools with health-related absences. Custom CRS/ Monitoring CRS 33% 27% 20% Number of educational facilities (i.e. school buildings, classrooms, and latrines) rehabilitated/constructed as a result of USDA assistance Standard #7 CRS/ Monitoring CRS 0 291 212 Number of students enrolled in schools receiving USDA assistance Standard #8 CRS/ Monitoring CRS 0 55,861 40,429 Percent of government staff trained in the literacy ToT who go on to conduct literacy trainings Custom CRS/ Monitoring CRS 0% 86% 80% Value of public and private sector investments leveraged as a result of USDA assistance Standard #11 CRS/ Monitoring CRS $0.00 $31,092.49 133,512 Number of public-private partnerships formed as a result of USDA assistance. Standard #10 CRS/ Monitoring CRS 0 1 1 C4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Number of Parent-Teacher Associations or similar “school” governance structures supported as a result of USDA assistance Standard #9 CRS/ Monitoring CRS 0 350 350 Percent of VEDCs that are 'high functioning' Custom CRS/ Monitoring CRS 11% 34% 35% Number of individuals who demonstrate use of new child health and nutrition practices as a result of USDA assistance Standard #19 CRS/ Monitoring CRS 0 794 465 Number of individuals who demonstrate use of new safe food preparation and storage practices as a result of USDA assistance Standard #21 CRS/ Monitoring CRS 0 947 710 Percent of schools targeted for WASH interventions that achieve at least a minimum standard Custom CRS/ Monitoring CRS 0% 10% 87% Number of individuals trained in safe food preparation and storage as a result of USDA assistance Standard #20 CRS/ Monitoring CRS 0 1,308 1,000 Number of schools using an improved water source Standard #22 CRS/ Monitoring CRS 178 269 208 Number of schools with improved sanitary facilities Standard #23 CRS/ Monitoring CRS 160 246 190 Percent of water user committees collecting user fees for maintenance of water points provided through USDA assistance Custom CRS/ Monitoring CRS 0% 3% 90% Number of teachers receiving teacher recognition awards Custom CRS/ Monitoring CRS 0 140 280 Number of libraries established Custom CRS/ Monitoring CRS 0 215 407 LEAPS II – Final Impact Evaluation Report  February 2020 C5 Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Number of books and supplementary materials produced (by type) as a result of USDA assistance Custom CRS/ Monitoring CRS 0 45,658 79,252 Number of teachers in target schools trained on Literacy Custom CRS/ Monitoring CRS 0 182 360 Number of teachers in target schools trained on Inclusive Education Custom CRS/ Monitoring CRS 0 145 460 Number of school administrators trained on Literacy Custom CRS/ Monitoring CRS 0 77 134 Number of school administrators trained on Inclusive Education Custom CRS/ Monitoring CRS 0 55 91 Number of daily school meals (lunch) provided to school-age children as a result of USDA assistance Standard #15 CRS/ Monitoring CRS 0 9,487,535 32,705,154 Number of take-home rations provided as a result of USDA assistance Standard #13 CRS/ Monitoring CRS 0 42,059 94,177 Number of school gardens established Custom CRS/ Monitoring CRS 0 277 150 Number of kitchens at target schools constructed or rehabilitated Custom CRS/ Monitoring CRS 0 210 112 Number of storerooms at target schools constructed or rehabilitated Custom CRS/ Monitoring CRS 0 52 70 Number of CBR trainings conducted in communities Custom CRS/ Monitoring CRS 0 5 8 Number of children screened for disabilities Custom CRS/ Monitoring CRS 0 172 444 C6 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Targets McGovern-Dole Performance Indicator Indicator Number Data Collection Method Data Source Baseline Midterm Endline Life of Project Target Number of children receiving assistive devices Custom CRS/ Monitoring CRS 0 42 196 Number of days Pedagogical Advisors supported to conduct in￾school literacy coaching with teachers Custom CRS/ Monitoring CRS 0 201 300 Number of steering committee visits Custom CRS/ Monitoring CRS 0 4 5 Number of Literacy Boost advocacy meetings Custom CRS/ Monitoring CRS 0 8 5 Number of literacy promotion activities completed (by type) as a result of USDA assistance Custom CRS/ Monitoring CRS 0 5,202 6,300 Number of VEDCs trained Custom CRS/ Monitoring CRS 0 350 350 Number of VEDC graduation grants provided Custom CRS/ Monitoring CRS 0 31 126 Value (USD) of VEDC graduation grants provided Custom CRS/ Monitoring CRS 0 $ 20,750.41 75,600 Number of student WASH ambassadors Custom CRS/ Monitoring CRS 0 84 150 Number of VEDC members/food preparers trained on food preparation and storage practices Custom CRS/ Monitoring CRS 0 1,238 3,377 Number of wells and water stations/systems built or rehabilitated Custom CRS/ Monitoring CRS 0 58 30 Number of latrines built or rehabilitated Custom CRS/ Monitoring CRS 0 63 30 Number of active water user committees trained Custom CRS/ Monitoring CRS 0 62 30 LEAPS II – Final Impact Evaluation Report  February 2020 D1 APPENDIX D. LIST OF STAKEHOLDERS Redacted LEAPS II – Final Impact Evaluation Report  February 2020 E1 APPENDIX E. ADDITIONAL TABLES AND COMPLEMENTARY OUTCOMES Exhibit 31. Other Languages Spoken at Home Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 1,962 for baseline and 1,850 midterm Exhibit 32. Total Number of SES Goods Owned by District Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 468 for Atsaphone, 62 for Nong, 315 for Outhomphone, 228 for Phalanxai. 318 for Phin, 239 for Sepon, 220 for Vilabouly 1%*** 1%*** 4%*** 3%*** 8%*** 27% 0% 0% 0% 1% 1% 10% 0% 10% 20% 30% 40% Katang Tri Thai Makong/Bru Phooutay Lao Baseline Midterm 5.2 2.8 5.9 5.1 4.1 4.4 5.2 Average: 4.9 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 Atsaphone Nong OuthoumphonePhalanxai Phin Sepon Vilabouly E2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Exhibit 33. Frequency of Teachers Asking Questions about Stories by District Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 313 for Atsaphone, 24 for Nong, 222 for Outhomphone, 154 for Phalanxai. 208 for Phin, 167 for Sepon, 138 for Vilabouly. Note: This question was not asked of students who missed all five days of school. Exhibit 34. Frequency of Teachers Playing Alphabet or Reading Games by District Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 313 for Atsaphone, 24 for Nong, 222 for Outhomphone, 154 for Phalanxai. 208 for Phin, 167 for Sepon, 138 for Vilabouly. Note: This question was not asked of students who missed all five days of school. 14% 33% 30% 16% 23% 29% 28% 48% 8% 38% 34% 21% 26% 20% 28% 54% 19% 26% 45% 25% 30% 10% 4% 14% 24% 12% 20% 22% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Often Sometimes Rarely Never 7% 1% 2% 2% 1% 30% 15% 33% 27% 27% 35% 33% 30% 27% 13% 30% 18% 20% 21% 25% 24% 42% 66% 37% 55% 53% 42% 41% 45% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Every day A few times during the week Once during the week Never LEAPS II – Final Impact Evaluation Report  February 2020 E3 Exhibit 35. Frequency that Teachers Tell a Story Not from Textbooks by District Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 313 for Atsaphone, 24 for Nong, 222 for Outhomphone, 154 for Phalanxai. 208 for Phin, 167 for Sepon, 138 for Vilabouly. Note: This question was not asked of students who missed all five days of school. Exhibit 36. Availability of Books at Home Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 922 for non-Lao and 866 for Lao. Note: Sample excludes those who did not know their language. 3% 1% 2% 3% 1% 46% 11% 48% 44% 36% 50% 39% 43% 22% 25% 26% 27% 30% 20% 26% 25% 31% 61% 26% 28% 33% 28% 32% 31% 0% 20% 40% 60% 80% 100% 120% Every day A few times during the week Once during the week Never 6%** 13%*** 26%*** 69%*** 92% 4% 7% 20% 59% 91% 0% 20% 40% 60% 80% 100% Magazines Newspapers Storybooks/Comics Coloring and drawing books Textbooks Non-Lao Lao E4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Exhibit 37. What Students Like about School Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. Listeners: N = 1962 at baseline and 1850 at midterm Exhibit 38. Lunch was Served at School by District 2% 2% 3% 3%* 6%*** 8%** 11%*** 19%*** 22%*** 23%*** 46%*** 71%*** 80%*** 1% 3% 3% 4% 14% 7% 3% 32% 9% 42% 31% 48% 39% 0% 20% 40% 60% 80% 100% Presenting ideas in class Working in groups Answering questions Dancing Physical education/sports in school Teacher reading Food is provided Other Participating in class activities/games Learning new things Being w/my friends Reading Writting Baseline Midterm 45% 80% 67% 79% 29% 82% Average: 66% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% LEAPS II – Final Impact Evaluation Report  February 2020 E5 Source: Student survey, authors’ calculations. N = 184 for Atsaphone, 0 for Nong, 150 for Outhomphone, 148 for Phalanxai. 154 for Phin, 51 for Sepon, 101 for Vilabouly. Note: This question was only asked during surveys that took place in the afternoon. Exhibit 39. Amount of Time Spent on Each Student Activity Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 82 classrooms Exhibit 40. Amount of Time Spent on Each Teacher Activity Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 82 classrooms 1%*** 2% 3% 5%*** 5% 6%** 6%** 6%** 6% 7% 9% 11% 33%*** 11% 3% 4% 10% 5% 10% 9% 3% 5% 7% 7% 13% 6% 0% 5% 10% 15% 20% 25% 30% 35% Copy Games Problem solving/debating Questioning/answering Teacher tells story Discussion Repetition Dictation Silent reading Commands/lecture Students left alone Read aloud Classroom management Baseline Midterm 2%*** 10%** 26% 62%*** 43% 6% 30% 6% 0% 10% 20% 30% 40% 50% 60% 70% Same activity as the students Outside the classroom Classroom management Disciplining a child Baseline Midterm E6 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Exhibit 41. Distribution of Letter Identification by Language Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 361 for non-Lao and 235 for Lao. Exhibit 42. Reasons for Liking School by Treatment Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 463 for LB + SF and 1,387 for SF. 1% 4% 8% 19% 68% 0% 3% 5% 18% 74% 0% 10% 20% 30% 40% 50% 60% 70% 80% None Identified 1 to 9 letters Identified 10 to 17 letters Identified 18 to 25 letters Identified 26 to 33 letters Non-Lao Lao 2%** 3% 2%*** 2%** 13%*** 8%** 20%*** 47% 70% 78%*** 3% 4% 4% 4% 6% 11% 27% 44% 73% 86% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Presenting ideas in class Dancing Working in groups Answering questions Food in provided Teacher reading Participating in class activities/games Being w/my friends Reading Writting LB + SF SF LEAPS II – Final Impact Evaluation Report  February 2020 E7 Exhibit 43. Frequency of Classroom Activities by SF+LB Schools Frequency Hear a Story or Poem Answer Questions about Story Play a Game SF only SF+LB SF only SF+LB SF only SF+LB Every day/often 1% 0% 23% 22% 0% 1% A few times a week/sometimes 39% 54%*** 30% 41%*** 28% 36%*** Once a week/rarely 27% 21%*** 32% 22%*** 23% 26% Never 33% 25%*** 15% 15% 48% 37%*** Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. For hearing a story, N = 1,329 for Non-LB and 449 for LB; For answering questions about a story, N = 890 for Non-LB and 336 for LB; For playing a game, N = 1,332 for Non-LB and 449 for LB. Note: The sample excludes students who missed all five days of school Exhibit 44. Attentiveness by Student Activity Source: Classroom observations, authors’ calculations. N = 82 13% 45% 54% 57% 57% 58% 59% 60% 61% 62% 65% 66% 69% 77% 89% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Classroom management Students are left with no guidance or direction Reading aloud Dictation Silent reading Educative games Commands, lecture Questioning/Answering Repetition The teacher is reading a story or another text to students Practicing Copy Discussion Problem solving, debating, discussing in group Production/expression F1 LEAPS II – Final Impact Evaluation Report  February 2020 APPENDIX F. OTHER SUBTESTS OF READING ASSESSMENT As mentioned in Section 5.3.4 Literacy Assessment Outcomes IMPAQ also tested students on other literacy skills including expressive vocabulary, phonemic awareness, word recognition, and object-to￾picture matching. This appendix presents the outcomes of these subtests to shed more lights on children’s literacy outcomes. Expressive Vocabulary To assess children’s vocabulary skills in Lao, IMPAQ asked students to name as many animals and foods as they could in Lao in two different questions. Enumerators were supposed to count the total number and mark down children’s responses from 0 to 10, for each question. At midterm, students were able to name on average 16 out of 20 animals and foods (8 animals and 8 foods in Lao), which was significantly higher by 1.8 words compared to baseline (14.3 animals and foods on average). Students who speak Lao as their main language did slightly better (16.9 words) than their non-Lao speaking peers (15.8 words), a significant difference at the one percent level. Exhibit 46 shows the distribution of expressive vocabulary ability by district. On average, in Vilabouly, students identified only 11.4 words while students in Outhoumphone named 17.7 on average. Exhibit 45. Expressive Vocabulary by District Source: Student survey, authors’ calculations. N = 158 for Atsaphone, 0 for Nong, 50 for Outhomphone, 66 for Phalanxai. 96 for Phin, 110 for Sepon, 46 for Vilabouly. Note: This question was only asked in LB schools. Phonemic Awareness IMPAQ instructed enumerators to read a word to the child and then tell her/him the letter sound (e.g., Cats starts with /c/) that starts the word. Then, enumerators read three words aloud for students and asked them to identify the one that starts with the same sound (for three different sets of words). At midterm, students paired on average 1.6 words with the first-letter sounds (out of three sets), which was slightly lower than at baseline (1.7 word pairs with the first-letter sounds). However, this minimal decrease was not statistically significant. Exhibit 47 shows the breakdown of students’ phonemic awareness by the total number of pairs they identified. The graph ranges from zero – i.e., not being able to pair any set of words with their first-letter 17.0 17.7 16.3 15.1 16.7 11.4 Average: 16.1 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 F2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report sound, to three, which is identifying the letter sound in all three sets of words. The proportion of students who identified one to two pairs remained the same from baseline to midterm. However, the shift was from fewer students pairing all the three sets to more students not identifying any pairs. At midterm, 34 percent of students were able to identify all three word-pairs, while 27 percent could not identify any pairs. The proportion of Lao speakers (23 percent) who could not identify any word pairs was significantly lower than non-Lao speakers (30 percent), statistically significant at the 10 percent level. However, there was no significant difference in identifying all the three word-pairs between Lao and non-Lao speakers. This outcome greatly varied by district. In Vilabouly, 65 percent of students identified zero word pairs correctly while the remaining districts ranged between 21 and 35 percent. Students were most likely to identify all the three word pairs correctly in Phalanxai (39 percent) while just 9 percent of students in Vilabouly could do so. There was no significant difference between boys and girls. Exhibit 46. Word Pairs Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 496 at baseline and 526 at midterm Word Recognition To assess children’s word recognition skill, IMPAQ gave students a chart of 20 words comprising the most frequently used words from learners’ language arts textbooks, selected based on frequency of their appearance. On average, students could read 7.9 of the 20 words (40 percent) – a significant increase from 5.5 words (27 percent) at baseline (p < 0.01). As shown in Exhibit 48, this increase was driven by a shift from fewer students not being able to read any of the most common words to more students identifying all the 20 words. More specifically, 23 percent of students could not read any of the words – a significant (p < 0.01) 12-percentage point decrease from baseline (35 percent). On the other hand, the proportion of students who could identify all 20 most common words increased from baseline (5 percent) to midterm (14 percent), significant at the one-percent level. Baseline 24% 18% 20% 38% Midline 27% 18% 22% 34% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Zero One Two Three F3 LEAPS II – Final Impact Evaluation Report  February 2020 Exhibit 47. Most Common Used Words at Midterm Source: Student survey, authors’ calculations. *p < 0.10; ** p < 0.05; *** p < 0.01. N = 496 at baseline and 526 at midterm At midterm, girls identified words at about the same rate. However, the midterm data shows that 26 percent of boys could not read any of the words while only 20 percent of girls could not do so; this difference is statistically significant at the ten-percent level. IMPAQ also found differences by district in reading the most commonly used words correctly, ranging from 3.7 and 4.0 words in Phin and Vilabouly to 9.1 in Sepon. The proportion of students who could not identify any of the words ranged from 12 percent in Sepon to 50 percent in Vilabouly. Non-Lao speaking students (19 percent) were significantly (p < 0.01) more likely to identify all 20 words compared to Lao speaking (8 percent). Students could identify most easily “ງູ”, which means snake in English (65 percent of students could do so). Similar to baseline, the hardest word for students to identify was “ອະນຸຍາດ”, which translates to allow in English, as only 23 percent of students were able to identify this word. Surprisingly, students whose main language at home was not Lao (29 percent) had more success than Lao speaking students (17 percent) in identifying the most used words, which was a significant difference at the five-percent level. Object to Picture Matching To measure children’s ability to understand symbolic representation, enumerators showed students five words and asked them to match those objects with their corresponding pictures. Enumerators repeated this activity twice with two different sets of words and pictures to match (10 words in total). To clarify the instructions, enumerators used one of the words as an example, which IMPAQ excluded from the analyses. Students’ ability to match objects with their pictures increased significantly from 44 percent at baseline to 61 percent at midterm (p < 0.01). In fact, all of our observed changes were significant at the one percent level, unless otherwise stated (see results below). At midterm, significantly fewer students (11 percent) could not match any of the objects with their pictures than at baseline (29 percent). More boys (15 percent) than girls (8 percent) could not match any objects with their pictures, significant at the five-percent level. Similarly, fewer Lao-speaking students (6 percent) could not match any words compared to 15 percent of non-Lao speakers. The proportion of students who could not match any words ranged from 2 percent in Sepon to 34 percent in Phin. 35% 30% 11% 9% 14% 23%*** 28% 15% 8% 26%*** 0% 5% 10% 15% 20% 25% 30% 35% 40% None 1-5 Words 6-10 Words 11-15 Words 16-20 Words Baseline Midterm F4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report In addition, the proportion of students who matched all nine words increased significantly from baseline (23 percent) to midterm (37 percent). Girls were able to match all of the words 43 percent of the time while only 31 percent of the boys could do so, a statistically significant difference at the one-percent level. Students were most often able to match all nine words in Phalanxai at 48 percent, while they were least often able to match them all in Phin (15 percent). Similar to baseline, the easiest object for students to match with its picture was “ງູ” – the word “snake” in English, which 76 percent of students could do. Disaggregating these results by gender, 82 percent of girls compared to 71 percent of boys were able to make the match. By district, the proportion of students who were able to match “snake” ranged from 57 percent in Phin to 82 percent in Sepon. The hardest word to match was “ກົ ບ” which means “frog” in English, as only 55 percent of the students were able to match this word. At baseline, the hardest words were “cup” (ຈອກ) and “book” (້ ປ ມ). Girls performed better at this task – 59 percent of girls could match it compared to just 51 percent of boys, a significant difference at the five-percent level. When examining the results by district, 64 percent of students in Sepon could match it compared to just 26 percent in Phin. Overall, the literacy outcomes at midterm show significant improvements from baseline to midterm with a consistent trend where girls outperformed boys in most of the subtests, if not showing equal literacy skills. However, the data show mixed trends between Lao and non-Lao speakers. LEAPS II – Final Impact Evaluation Report  February 2020 G1 APPENDIX G. REGRESSION ANALYSIS The evaluation team used multivariate regression analyses to shed light on the potential predictors of students’ literacy outcomes at midterm. IMPAQ examined the relationship between students’ literacy skills and some key demographic characteristics mostly in format of an indicator, including: ▪ Age – Students’ age measured in years ▪ Sex – Students’ gender (1=female) ▪ Language (Lao) –Lao as the primary language spoken at home ▪ Multilingual household – IMPAQ created an indicator if the student speaks more than one language at home, regardless of which language ▪ ECD experience – Student attended ECD or preschool ▪ Grade Repetition – Student has repeated a grade ▪ Reading materials – IMPAQ created an index by adding all reading materials (textbooks, newspapers, storybooks/comics, coloring/drawing books) at home. IMPAQ weighted child￾friendly materials, such as storybooks/comics and coloring/drawing books double in the regression analysis. ▪ Home literacy interactions – Based on all home literacy interactions, including whether at home o the student has seen someone reading, or o someone encouraged the student to study, or o anyone told the student a story, or o anyone read to the student IMPAQ made an index by adding all these interactions, but weighted the reading to student double because it plays an important role in improving students’ literacy skills. ▪ Socio-economic status (SES) – Sum of all household durable possessions, including mobile phone, electricity, fridge, bike, TV, motorbike, car, and tractor. In addition, IMPAQ checked whether these relationships changed depending on students’ enrollment in SF + LB or SF schools, especially if the school was part of the Room-to-Read program before LEAPS program, using the following indicators ▪ LB school – IMPAQ created an indicator equal to one if the school received the LB teacher training as a part of the first cohort of the LEAPS II program, and zero if it only received SF package. ▪ Room to Read – Schools which received the Room to Read intervention prior to LEAPS II Exhibit 49 summarizes associations between students or household characteristics and students’ literacy skills. The table shows either positive or negative associations that are statistically significant (p < 0.05). A “+” indicates that the factor is positively associated with the respective literacy outcome, while a “-“ indicates a negative association. Detailed regression results can be found in the full OLS regression table in Exhibit 50 below. The results show a positive association between LB and Room to Read schools for many of the underlying literacy skills however; the associations disappear when considering reading and comprehension G2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report abilities.45 This is consistent with our findings that students largely improved in terms of foundational literacy skills but showed little improvement in terms of reading ability. All other possible determinates show little to no association with literacy skills. Exhibit 48. Predictors of Literacy Skills at Midterm Independent Variable Letter Knowledge Word Recognition Matching Vocab Word Pair Fluency Reading with Comprehension Age Female + + Lao Speaker Multilingual ECD Ever Repeated a Grade Reading materials Home literacy index + SES index − + LB school + + Room to Read school + + Source: Student survey, authors calculation. Note: Only statistically significant predictors (p-value < 0.05) are presented in the table. Exhibit 49. Predictors of Literacy Skills at Midterm Detailed Regression Independent Variable Letter Knowledge Word Recognition Matching Vocab Word Pair Fluency Reading with Comprehension Age 0.004 0.017 0.012 0.005 0.016 -0.103 0.019 (0.006) (0.015) (0.015) (0.007) (0.015) (0.924) (0.013) Female 0.050*** 0.055 0.117*** -0.000 0.029 2.794 0.037 (0.017) (0.046) (0.041) (0.018) (0.044) (2.730) (0.039) Lao Speaker 0.042* -0.105* 0.045 0.003 -0.006 -1.283 -0.076* (0.021) (0.063) (0.052) (0.022) (0.050) (3.323) (0.044) Multilingual 0.039 0.016 0.023 -0.013 -0.045 5.952 0.058 (0.026) (0.062) (0.053) (0.034) (0.065) (4.432) (0.063) ECD -0.000 -0.043 -0.020 -0.036 0.041 -2.804 0.024 (0.023) (0.059) (0.047) (0.023) (0.053) (3.077) (0.042) Ever repeated a grade -0.001 -0.021 0.005 -0.021 0.046 3.958 -0.030 (0.022) (0.048) (0.039) (0.022) (0.056) (2.759) (0.036) Reading materials -0.004 0.005 -0.002 0.002 0.010 -0.646 0.002 (0.006) (0.013) (0.011) (0.007) (0.013) (0.767) (0.014) Home literacy index 0.004 0.010 -0.000 0.018*** -0.004 0.584 -0.006 (0.006) (0.013) (0.011) (0.006) (0.012) (0.556) (0.012) 45 These associations are also null in regard to readers, reader accuracy and comprehension, and listener comprehension. LEAPS II – Final Impact Evaluation Report  February 2020 G3 Independent Variable Letter Knowledge Word Recognition Matching Vocab Word Pair Fluency Reading with Comprehension SES Index 0.000 -0.033 -0.047** 0.029** -0.038 -1.272 0.023 (0.009) (0.029) (0.023) (0.011) (0.025) (1.338) (0.024) LB school 0.165*** 0.114 0.157** -0.008 0.114* 1.249 -0.026 (0.024) (0.084) (0.065) (0.027) (0.061) (2.836) (0.058) Room to Read school 0.001 0.197** 0.064 0.061** 0.015 -1.389 0.012 (0.019) (0.094) (0.055) (0.026) (0.053) (2.618) (0.047) Constant 0.684*** 0.230 0.423** 0.770*** 0.353** 26.564*** -0.006 (0.059) (0.155) (0.159) (0.067) (0.164) (9.086) (0.117) R 2 0.237 0.123 0.098 0.087 0.043 0.140 0.029 N 366 366 366 366 366 80 366 Source: Student survey; authors’ calculation. *p-value <0.1, ** p-value < 0.05, *** p-value < 0.01 Note: Robust standard errors are clustered at the school level and shown in parentheses below the coefficients. Fluency is among readers only. LEAPS II – Final Impact Evaluation Report  February 2020 H1 APPENDIX H. INTER-RATER RELIABILITY Reading Assessment To measure the reliability and level of homogeneity of enumerators’ scores on children’s literacy skills, two enumerators assessed 5 percent of the overall second grade sample (99 out of 1,850) simultaneously. In calculating the intra-class correlation within pairs of assessors to determine inter-rater reliability (IRR), IMPAQ used long one-way Analysis of Variance (ANOVA) techniques, which involve determining whether the mean of a dependent variable is the same in two or more unrelated and independent groups. Our interpretation of the intra-class correlations was adapted from Fleiss et al.46 as follows: ▪ Less than .40 – Poor ▪ Between .40 and .75 – Good or fair ▪ Greater than .75 – Excellent Exhibit 51 shows the percent of agreement between the raters, as well as IRR ratings for the project evaluation sample. Overall, the IRR across the project evaluation sample was excellent for each literacy skill measure, showing high internal validity of the scores. IMPAQ saw no variation in the data for letter knowledge, expressive vocabulary, readers, and reading comprehension. Therefore, the ANOVA test could not calculate the IRR. Exhibit 50. IRR by Literacy Skill Subtests for Performance Sample Literacy Skill Sub-test IRR Rating Letter Knowledge n/a n/a Expressive Vocabulary n/a n/a Word Pair 99% Excellent Matching 100% Excellent Most Commonly Used Words 100% Excellent Reader n/a n/a Reader Comprehension n/a n/a Listener Comprehension 100% Excellent Source: Student survey, authors’ calculation. N = 321 Grade 2 students 46 Fleiss, J. L., & Cohen, J. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 613-619. LEAPS II – Final Impact Evaluation Report  February 2020 I1 APPENDIX I. SURVEY INSTRUMENTS DATA COLLECTION FOR USDA FOOD FOR EDUCATION (LEAPS II) IN LAOS Student Survey Start Time Date INTRODUCTION districts Enter the name of the district ----------------------------- school name Enter the school name ------------------ motherconsent Has the teacher given consent for her child to participate in this survey? No consent → thank them and terminate the survey and select the next child on your list. Teacher Consented → ” timeofsuevey” I__I If teacher says No, thank them, and terminate the survey and proceed to the next child on your list. timeofsurvey Is the survey administered in the… 1. Morning (before 12 pm) 2. Noon (between 12pm and 1pm) 3. Afternoon (after 1pm) I__I select only one option Dear student: Hi, my name is ___, and I am here asking some questions from children like you to understand more about the LEAPS project also known as the Primary School Lunch Program for Small Children. Nothing you say here will be repeated to your parents or teacher will be kept a secret. There aren’t any right or wrong answers. I want you to answer honestly and as best as you can. Do you have any questions for me? You can interrupt me to ask a question at any time. Also, of you don't know the answer to a question or don't want to answer it, just let me know and we can skip it. Are you ready to begin? assent Do you accept if I ask you some questions? I__I I2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report No → thank him/her, terminate the survey and proceed to the next child on your list. Yes → continue with the background section. If child says No, thank him/her, terminate the survey and proceed to the next child on your list. Student Code: Studentcode Please get the student code from the team leader. It is very important to use the correct student code, so please enter the code twice. If you are unsure, please check again with the team leader Track Is this student: A new student Selected from tracking list I__I Stcode1 Please enter the student code CAREFULLY Record student code Stcode2 Please enter the student code CAREFULLY again Record student code reliab Is this an individual assessment or a pair assessment? Individual → “nickname Pair assessment → “reliabtype” I__I reliabtype Talking enumerator or observing enumerator? Talking Watching I__I Background information nickname What is your nickname? fname What is your first name? mothername What is your mother's name? ------------------------------------- LEAPS II – Final Impact Evaluation Report  February 2020 I3 fathername What is your father’s name? ------------------------------------- gender Male Female I__I *Ask only if necessary age How old are you? …… *RECORD AGE >=5 & <17 If the child doesn not know her/his age enter 999 newsch When you started at this school, which grade were you in? Preschool Grade 1 Grade 2 Grade 3 Grade 4 0. Grade 5 999. Don’t know I__I *Select only one option grade Which grade/class are you in? Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 I__I *Select only one option I4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report everrpt Did you repeat any grades? No Yes 999. Don't know I__I *Select only one option Everrpt_b Which grade had you repeated? Grade 1 Grade 2 Grade 3 Grade 4 1. Grade 5 I__I *Select all that apply Everrpt_c1 How many times did you repeat Grade 1? … Enter the frequency Everrpt_c2 How many times did you repeat Grade 2? … Enter the frequency Everrpt_c3 How many times did you repeat Grade 3? … Enter the frequency Everrpt_c4 How many times did you repeat Grade 4? … Enter the frequency Everrpt_c5 How many times did you repeat Grade 5? … Enter the frequency ecdattended Did you attend ECD/preschool? No Yes 999. Don't know *Select only one option LEAPS II – Final Impact Evaluation Report  February 2020 I5 mainlang What language do you often speak at home? 1. Lao 2. Phouthay 3. Makong/Bru 4. Tri 5. Taoy 6. Katang 7. Thai 8. Other 999 Don'tKnow/No response I__I *Select only one option otherlang At home, do you speak any other languages? 1. Lao 2. Phouthay 3. Makong/Bru 4. Tri 5. Taoy 6. Katang 7. Thai 8. Other (Specify) 9. None 999 Don' tKnow/No response I__I I__I I__I I__I I__I I__I I__I *Select all that apply ses Does your home have the following: 1. Mobile 2. Electricity 3. Refrigerator 4. Bicycle 5. TV 6. Motorbike 7. Car 8. Tractor [TOK TOK] ) 9. None 999. Don't know I__I I__I I__I I__I I__I I__I I__I I__I I__I *Please read all the options to the child and select all that apply I6 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report book At home do you have : 1. Textbooks 2. Magazines 3. Newspapers 4. Storybooks/COMICS 5. Coloring an drawing books 6. None of these 999. Don't know I__I I__I I__I I__I I__I *Please read all the options to the child and select all that apply HEALTH Thank you! Now, I would like to ask you a few questions about your health. health1 Have you been sick anytime during the last week? No → “health4” Yes → “health1a” 999. Don't know I__I *Select only one health1a What kind of sickness did you have? Sore stomach Fever Headache Tiredness Coughing Other 999. Don’t Know I__I I__I I__I I__I I__I I__I *If the child cannot name the sickness, ask them to describe the symptoms for you and select the right answer. *Give examples for a sore stomach such as food poisoning or diarrhoea *Select all that apply health2 During last week, did you miss school because you were sick? No → “health4” Yes → “health3” 999. Don't know I__I *Explain that this means they stayed at home *Select only one health3 How many days did you miss school because you were sick during the last week? 1 2 3 I__I *Select only one LEAPS II – Final Impact Evaluation Report  February 2020 I7 4 5 → fs3 999. Don’t Know Health4 During the last week did you miss school for any other reasons? No Yes 999. Don't know I__I *Explain to student if necessary: When the school was open, not because the school was closed *Select only one FOOD SECURITY Thank you! Now, I would like you to think about all the meals you had today. fs3 Did you eat something for breakfast today? No → “fs5” Yes → “fs4a” 999. Don't know I__I *Select only one fs4a Did you feel full after eating breakfast? Or could you have eaten more? I felt full → “fs5” I could have eaten more → “fs4b” I__I *Select only one fs4b Why didn’t you eat more in the morning? There wasn’t any more food There was nothing I liked Other I__I *Select only one The next set of questions will be asked if “timeofsurvey” is equal to 2 or 3, else go to “fsx3”. fs5 Has the school lunch meal already been served today? No → “fs6” Yes → “fsX1” 999. Don't know I__I *Select only one I8 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report fs6 Did you already eat something for lunch today? No →“fsX3” Yes →“fsX3” 999. Don't know I__I *Select only one fsX1 Did you already eat the school meal? No Yes 999. Don't know I__I *Select only one fsX2 Did you like the taste of the school meal? Not at all A little bit Somewhat Yes, very much I__I *Select only one fX2b Why didn’t you eat the school meal? I was sick I was playing There was not enough food I didn’t like the taste Other 999. Don’t know I__I *Don’t read the options to the child *Select only one fsX3 How hungry do you feel right now? Not at all hungry A little hungry Somewhat hungry Very hungry 999. Don’t know I__I *Select only one Household Environment Very good! We have a few more questions about your home. nhhold How many people are there in your household, including yourself? ……. *Record the number > 0 & < 21 LEAPS II – Final Impact Evaluation Report  February 2020 I9 *Mark 999 if no response/don’t know seeread During the last week, did you see anyone in your house reading? No → “helpstudy” Yes → “nseeread” 999. Don't know I__I *Select only one nseeread Who did you see reading last week? 1 2 3 4 5 6 999. Don’t know I__I (enter the number of household members the child counted) *If the child counted more than 6 select 6 helpstudy During the last week, did anyone in your house encourage you to study? No → “story2u” Yes → “nhelpstudy” 999. Don't know I__I Select only one option nhelpstudy Who encouraged you to study last week? 1 2 3 4 5 6 999. Don’t know I__I (enter the number of household members the child counted) *If the child counted more than 6 select 6 story2u During the last week, did anyone in your house tell you a story? No → “read2u” Yes → “nstory2u” I__I Select only one option I10 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report 999. Don't know nstory2u Who told you a story in the past week? 1 2 3 4 5 6 999. Don’t know I__I (enter the number of household members the child counted) *If the child counted more than 6 select 6 Read2u During the last week, did anyone in your house read to you? No → “hhenviro3” Yes 999. Don't know I__I *Select only one option nread2u Who read to you last week? 1 2 3 4 5 6 999. Don’t know I__I (enter the number of household members the child counted) *If the child counted more than 6 select 6 Hhenviro3 Are your parents or family members asking you questions about the stories they tell you or read to you? No Yes 999. Don't know I__I Select only one option Only ask if the answer to read2u OR story2u is yes readout During the last week, did you read outside of school? No Yes 999. Don't know I__I select only one option LEAPS II – Final Impact Evaluation Report  February 2020 I11 SCHOOL ENVIRONMENT AND PARTICIPATION We are almost done! Now, I would like to ask you some questions about your school. enviro1 Do you like coming to school? No Yes 999. Don't know I__I *Select only one option enviro2 What do you like about your school? Reading Writing Participating in class activities/games Dancing Teacher reading Being with my friends Working in groups Answering questions Presenting ideas in class Food is provided Physical education/sports at school Learning new things other Nothing 999. Don't know I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I I__I *Don’t give examples or read the list to the respondent *Select all that apply Skip enviro 6, enviro4, and enviro7 if health3 = 5 (missed school for 5 days) enviro6 How often in the last week did the teacher tell a story or read a poem to the classroom which was not in the textbook? Every day A few times during the week; Once during the week; I__I *Read the list to the respondent, but don't read 'don't know' *Select only one I12 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Never → “enviro7” 999. Don't know enviro4 How often in the last week did the teacher ask you about the story s/he told or the poem s/he read during class? Often Sometimes Rarely Never 999. Don’t know I__I *Read the list to the respondent, but don’t read “don’t know” *Select only one enviro7 How often in the last week did you play a game in the classroom around the alphabet or a reading activity? Every day A few times during the week; Once during the week; Never 999. Don't know I__I *Read the list to the respondent, but don't read 'don't know' *Select only one enviro10 Does your school have story books other than textbook for you to take home and borrow? No → expressvocab1 if the child is in second grade; otherwise thanks the child and terminate the survey. Yes → enviro10a 999. Don't know I__I *Select only one Enviro10a How often in the last week did you borrow story books other than textbook from school to take home to read? Every day A few times during the week; Once during the week; Never 999. Don't know I__I *Read the list to the respondent, but don't read 'don't know' *Select only one LEAPS II – Final Impact Evaluation Report  February 2020 I13 If child is not in grade 2 thank him/her and terminate the survey. If s/he is in grade two continue with expressive vocabulary LITERACY BOOST ASSESSMENT: Expressive vocabulary Now let’s try a word game. Imagine you are going to the market and name some foods that you can eat. Try to name as many things as you can think of. Record the number of items the child lists until the child has listed 10 items. You can tally on the score sheet as the child enumerates the objects. When the child pauses for 5 seconds or more, PROMPT ONCE by saying, Can you think of any others? When the child cannot think of more items, move on to the next question and say: Now, I want to know what animals you are familiar with. Tell me the names of some animals that you know. Try to name as many animals as you can think of and I will keep count again. When the child pauses for 5 seconds or more, PROMPT ONCE by saying, Can you think of any others? expressvocab1 Can you tell me the names of things you can eat in Lao? (Specify the number of items child says they can eat in Lao 0-10) 0 1 2 3 4 5 6 7 8 9 10 I__I *Select only one option Expressvocab2 Can you tell me the names of animals in Lao? (Specify the number of animals a child counted in Lao 0-10) 0 1 2 3 I__I *Select only one option I14 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report 4 5 6 7 8 9 10 Letter Sounds Now we will play a listening game. This one is about the sounds in words. The word “cat” starts with /c/ (Say the sound, not the letter name). /c/ is the first sound in cat. Now listen to the words I say and tell me which one starts with the same sound, the sound /c/ (Say the sound, not the letter name) star, ball, or cup? If the child gives an incorrect response, say: cup starts with /c/ just like cat. Wordpair1 I will read three words to you with the sound (”ດ“Child matches the letter “ດ” and the correct word) Not able to match/Don’t know Able to match 999. Did not understand the question I__I ຫມາ, ແມວ, ເດື ອນ Wordpair2 I will read three words to you with the sound ”ຕ“-child matches the letter "ຕ" with the correct word Not able to match/Don’t know Able to match 999. Did not understand the question I__I ໄມ , ກ ວຍ, ຕົ ນ Wordpair3 I will read three words to you with the sound "ມ" - Child was able to matchs the letter "ມ" with the correct word Not able to match/Don’t know Able to match 999. Did not understand the question I__I ໄກ , ຫນູ, ມ າ LEAPS II – Final Impact Evaluation Report  February 2020 I15 Understanding Letters Give the child the list of letters and say to the child: Say: Let’s look at some letters. Can you start here (point to first letter) and tell me what these letters are moving in this direction? (indicate left to right direction) Do you understand? Ok, you can begin. Correct letters are: the letter name in the home language or language of instruction any sound that is acceptable for in the home or instructional language a response which says “It begins like…” giving a word for which the letter is the initial letter If the child read the letters out of order, then remember to bring his/her attention to the ones they might have skipped. Make sure you marked all of the incorrect letters Move to the Most Used Words section. What to do if a student is struggling: If the student is struggling, and hesitates at any letter for five seconds, ask follow up questions: Do you know its name? What sound does it make? Do you know a word that starts with this letter? If the student still hesitates for five seconds, ask: Can you tell me any of these letters? If the student still hesitates for five seconds, then stop and thank him/her for trying his/her best. Mark letters not identified or not attempted as incorrect. Move to the Most Used Words section. ຈ ນ ຮ ຊ ຝ ມ ຂ ພ ຟ ງ ດ ອ ຫ ທ ລ ປ ຍ ກ ວ ຖ ຄ ຣ ບ ສ ຜ ຕ ຢ ຫວ ຫງ ຫຍ ໝ ຫຼ ໜ I16 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Most Used Words Give the pupil the laminated copy of the "Most Used Words" list. Say: I would like you to read some words to me. They are words from your textbook. Please point to and say each of these words starting here (point to first word) and moving across each line like this (indicate left to right direction). Do you understand? Ok, you can begin. Remember that pronunciations of words in local dialects are acceptable. If the child read the words out of order then remember to bring his/her attention to the ones they might have skipped. Make sure you marked all of the incorrect words. Move to the Matching Section. What to do if a student is struggling: If the student is struggling, and hesitates at any letter for five seconds ask the child, Are there any words on the list that you know? Tell me or say the words you know. Repeat the request to encourage the child to continue. If the student still hesitates for five seconds, then stop and thank him/her for trying his/her best. Mark words not identified or not attempted as incorrect. Move to the Decoding section. ລົ ດ ນາ ງູ ຈານ ມາ ກອບ ຍຸງ ກະປູ ໄຟ ປ ມ ຕະຫຼາດ ອ າຍ ແຂງແຮງ ເສ ອ ອະນຸຍາດ ໂສ ງ ອະນາໄມ ແມ ຕ ງ ເດ ອນ Matching 1 Point to the words and ask to the child which of these pictures the words mean. If the child cannot match the first three words then mark everything incorrect and move to the next test. Practice: Car → Point at the word for "car". Then point at the picture of the car. Ask if the child understands. shirt, snake, glass, crab LEAPS II – Final Impact Evaluation Report  February 2020 I17 Matching 2 Point to the words and ask to the child which of these pictures the words mean. If the child cannot match the first three words then mark everything incorrect and move to the next test. frog, mosquito, chair, fire, book I18 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report COMPREHENSION PASSAGES AND QUESTIONS The mother chicken teaches her chick. One day when the sky was transparent and clear, the sunlight was shining down on the ground and the cool wind sometimes blew. A flock of little chicks running under their mother’s tail happily went to find food. The mother chicken ordered her children “Don’t you go far away from me because a hawk will come and take you to eat. You won’t see your mother again!” One of the chicks went to peck food by itself and became separated from the group and was alone. The chicken felt afraid and tried to call for its mother. At the same time a hawk approached flying in the sky which made the chick even more scared. Later when the mother found the lost chick she took them all to safety. LEAPS II – Final Impact Evaluation Report  February 2020 I19 reader Is child a reader or a non-reader? A non-reader read less accurate than 5 per 30 seconds) A reader (read correctly 5 per 30 seconds) I__I Select only one option Comprehension Questions Comp1 What happened in the story? mother chicken and baby chickens go out to find food mother chicken tells baby chickens to stay close to her one baby wanders away from the flock and is scared of the hawk in the end, the mother chicken finds the baby and brings him home safely None I__I I__I I__I I__I mark every main point mentioned by the child Comp2 How many baby chickens were lost? (one chicken) False True I__I Don’t read the answer to them Comp3 Why did the baby chicken get lost? (a chicken finds food alone) I__I Don’t read the answer to them Comp4 Why was the lost chicken frightened? (The baby chicken saw a hawk flying in the sky/separated from her mother) False True I__I Don’t read the answer to them Comp5 Did the mother chicken find the lost chicken? (Yes) False True I__I Don’t read the answer to them I20 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Comp6 What could happen if the hawk catches the baby chicken? (the chicken will be hawk’s food/chicken has died/ can’t find mother chicken) False True I__I Don’t read the answer to them Comp7 Where did baby chicken find food? (Somewhere around rice field) False True I__I Don’t read the answer to them Comp8 Why should baby chicken not look for food by themselves? Student could explain their answer with information from the story Student count NOT explain their answer with information from the story I__I Don’t read the answer to them Thank you very much for answering my questions. End Time, Date __________________________ DATA COLLECTION FOR USDA FOOD FOR EDUCATION (LEAPS II) IN LAOS ການເກັ ບການຂໍໍ້ມູນສໍ າຫຼວດເບືໍ້ອງຕົ ນສໍ າລັ ບໂຄງການອາຫານເພືື່ ອການສຶ ກສາ (LEAPS II) ຂອງ USDA ຢູື່ ສປປ ລາວ Classroom observation ການສັ ງເກດຫໍ້ອງຮຽນ Attentiveness – All grades and subjects ຄວາມເອົ າໃຈໃສື່-ທຸກຊັໍ້ນຮຽນ ແລະ ທຸກວິ ຊາ ID ລະຫັ ດ District ເມື ອງ ❑ Outhoumphone ເມື ອງອຸ ທຸ ມພອນ ❑ Phin ເມື ອງພີ ນ ❑ Nong ເມື ອງນອງ ❑ Vilabouly ເມື ອງວີ ລະບຸ ີລ ❑ Atsaphone ເມື ອງອາດສະພອນ ❑ Phalanxai ເມື ອງພະລານໄຊ LEAPS II – Final Impact Evaluation Report  February 2020 I21 ❑ Xepon ເມື ອງເຊໂປນ School’s name ຊືື່ ໂຮງຮຽນ Teacher’s first name ຊືື່ ແທໍ້ຂອງນາຍຄູ Teacher’s last name ນາມສະກຸນຂອງນາຍຄູ Teacher’s gender ເພດ ❑ Male ຊາຍ ❑ Female ຍິ ງ The Observation ການສັ ງເກດ Enumerator ຊືື່ ຜູໍ້ ເກັ ບຂໍໍ້ມູນ Date ວັ ນທີ, ເດື ອນ, ປີ _ _ / _ _ / _ _ _ _ Subject ວິ ຊາ ❑ Math ຄະນິ ດສາດ ❑ Laos Language ພາສາລາວ ❑ English ພາສາອັ ງກິ ດ ❑ Drawing ຫັ ດແຕໍ້ມ ❑ The World Around Us ໂລກອໍ້ອມຕົ ວ ❑ Physical Education/Sport ພາລະສຶ ກສາ/ກິ ລາ ❑ Art (Song and Dance) ສີ ລະປະດົ ນຕີ(ຮໍ້ອງເພງ ແລະ ຟໍ້ອນ) The Class ຫໍ້ອງຮຽນ Observed grade ສັ ງເກດຫໍ້ອງ(ປໍ ) ❑ 1 ❑ 2 ❑ 3 ❑ 4 ❑ 5 single choice ເລື ອກໄດໍ້ຫໍ້ອງດຽວ Are there other grades in the classroom? ກໍ ລະນີ ໃນຫ້ອງຮຽນມີ ຫ້ອງອື ື່ ນຮຽ ນຄວບກັນ? ❑ Yes ໄດໍ້ເຮັ ດ ❑ No ບໍື່ ໄດໍ້ເຮັ ດ I22 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Other grades in the classroom (for multigrade classes) ກໍ ລະນີ ໃນຫໍ້ອງຮຽນມີ ຫໍ້ອງອືື່ ນຮຽ ນຄວບກັ ນ(ສໍ າລັ ບຫໍ້ອງຄວບ) ❑ 1 ❑ 2 ❑ 3 ❑ 4 ❑ 5 multiple choice ເລື ອກໄດໍ້ຫຼາຍຫໍ້ອງ Usual/Official class size ຈໍ ານວນນັ ກຮຽນທັ ງໝົ ດໃນຫໍ້ອງ Boys: ____ Girls: ____ ຊາຍ: ____ ຍິ ງ: ____ Actual # of students attending ຈໍ ານວນນັ ກຮຽນທີື່ ມາຮຽນຕົ ວຈິ ງ ມືໍ້ນີໍ້ Boys: ____ Girls: ____ ຊາຍ: ____ ຍິ ງ: ____ Learning Environment ສະພາບແວດລໍ້ອມຂອງການຮຽນ Learning Space ພືໍ້ນທີື່ ໃຊໍ້ຮຽນໜັ ງສື ❑ Fixed-permanent structure ໂຄໍ້ງສໍ້າງແບບທົ ນທານ ❑ Semi-permanent structure ໂຄງສໍ້າງແບບເຄິື່ ງທົ ນທານ ❑ Temporary ແບບຊົື່ ວຄາວ Arrangement ການຈັ ດໂຕະຕັື່ ງ ❑ Rows of desks with benches ຈັ ດເປັ ນແຖວຍາວມີ ໂຕະກັ ບຕັື່ ງ ❑ Grouped desks with benches ຈັ ດໂຕະເຂົໍ້າກັ ນມີ ຕັື່ ງອໍ້ອມແບບເປັ ນກຸື່ມ ❑ Benches without desks ມີ ຕັື່ ງແຕື່ວື່າບໍື່ ມີ ໂຕະ ❑ Neither desks nor benches ບໍື່ ມີ ທັ ງໂຕະ ແລະ ຕັື່ ງ ❑ Mats ປູສາດ ❑ Other (please specify) ອືື່ ນໆ (ກະລຸນາລະບຸ) Literate environment ສະພາບແວດລ ອມກ ຽວກ ບການ ຮຽນຮູ ❑ Students’ drawings ການແຕໍ້ມຮູບຂອງນັ ກຮຽນ ❑ Students’ written production’ ຜົ ນງານ, ສິື່ ງປະດິ ດການຂຽນຂອງນັ ກຮຽນ LEAPS II – Final Impact Evaluation Report  February 2020 I23 ❑ Reference posters (printed or home-made): letters, numbers, months, days of the week, key words, conjugations, etc. ແຜື່ ນເຈໍ້ຍຕິ ດຝາ (ແບບພິ ມເອົ າ ຫືຼເຮັ ດດໍ້ວຍມື): ຕົ ວອັ ກສອນ, ຕົ ວເລກ, ເດື ອນ, ວັ ນໃນອາທິ ດ, ຄໍ າສໍ າຄັ ນ, ການຜັ ນແປຄໍ າສັ ບ ແລະ ອືື່ ນໆ ❑ Geography, science, and other informational posters ແຜື່ ນເຈໍ້ຍຕິ ດຝາກື່ຽວກັ ບພູມສາດ, ວິ ທະຍາສາດ ແລະ ຂໍໍ້ມູນຂື່າວສານອືື່ ນໆ ❑ Reading corner ມຸມອື່ານ ❑ Poetry, national song ບົ ດກາບກອນ, ບົ ດກະວີ, ເພງຊາດ ❑ Slogan, proverb, motto ຄໍ າຂັ ວນ, ຄໍ າສຸພາສິ ດ ❑ None ບ ມີຫຍ ງ Students’ attentiveness and instructional practices (snapshots) ຄວາມເອົ າໃຈໃສື່ຂອງນັ ກຮຽນ ແລະ ການປະຕິ ບັ ດການຮຽນການສອນຕົ ວຈິ ງ(snapshots) a. Starting time ເວລາເລີ ມ _ _ h _ _ mn _ _ ໂມງ _ _ ນາທີ I24 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report ສະແນັ ບຊອດສ໌ 1 STUDENTS’ activities ກິ ດຈະກໍ າຂອງນັ ກຮຽນ N° Activity ກິ ດຈະກໍ າ 1 Reading aloud (collective or one by one) ການອື່ານອອກສຽງ (ອື່ານພໍ້ ອມກັ ນ ຫືຼອື່ານເທືື່ ອລະຄົ ນ) ❑ 2 The teacher is reading a story or another text to students who are listening ນາຍຄູອື່ານເລືໍ້ອງ ຫືຼອື່ານໜັ ງສື ໃຫໍ້ຟັ ງ (ນັ ກຮຽນຟັ ງຢື່າງດຽວ) ❑ 3 Students are left with no guidance or direction about what they should do in class ນັ ກຮຽນຖື ກປະໄວໍ້ໂດຍບໍື່ ມີ ການແນະນໍ າ ຫືຼບອກໃຫໍ້ເຂົ າເຈົໍ້າຄວນເຮັ ດຫຍັ ງຢູື່ໃນຫໍ້ອງ ❑ 4 Classroom management (materials distribution, chairs arrangements; transition between 2 activities...) ການຈັ ດການຫໍ້ອງຮຽນ (ແຈກຢາຍອຸປະກອນ, ຈັ ດບື່ ອນນັື່ ງ; ການປື່ຽນແປງກິ ດຈະກໍ າສອນຈາກວິ ຊາໜຶື່ ງໄປອີ ກວິ ຊາໜຶື່ ງ....) ❑ 5 Commands, lecture ການສັື່ ງໃຫໍ້ເຮັ ດ, ການສອນ ຫືຼອະທິ ບາຍບົ ດຮຽນ ❑ 6 Educative games ເກມໃຫໍ້ຄວາມຮູໍ້ ❑ 7 Problem solving, debating, discussing in group ການແກໍ້ ໄຂບັ ນຫາ, ການໂຕໍ້ວາທີ, ການສົ ນທະນາເປັ ນກຸື່ມ ❑ 8 Dictation ຂຽນທວຍ ❑ 9 Silent reading ການອື່ານໃນໃຈ ❑ 10 Repetition ການອື່ານຕາມນາຍຄູ, ການອື່ານຄໍ າຖາມເກົື່ າຊໍໍ້າກັ ນຫຼາຍໆຄັໍ້ງ ❑ 11 Discussion ການສົ ນທະນາ ❑ 12 Questioning/Answering ການຖາມ/ການຕອບ ❑ 13 Copy ຫັ ດກື່າຍ ❑ 14 Production/expression ການປະດິ ດສໍ້າງ/ການເຮັ ດສິື່ ື່ງໃດໜຶື່ ງດໍ້ວຍຄວາມຄິ ດຂອງຕົ ນເອງ ❑ 15 Practicing (individual, in group, or at the blackboard) ການຝຶ ກຝົ ນ ຫືຼ ເຮັ ດບົ ດຝຶ ກຫັ ດ (ດໍ້ວຍຕົ ນເອງ, ເປັ ນກື່ມ, ຫືຼເຮັ ດຢູື່ເທິ ງກະດານ) ❑ 16 Other (specify) : ື ື່ອນໆ (ກະລຸ ນາລະບຸ ) ❑ Teacher activities if different ກິ ດຈະກໍ າຂອງນາຍຄູ ຖໍ້າບໍື່ ຄື ກັ ນ 1 The teacher is outside of the classroom or does something unrelated to the classroom ນາຍຄູບໍື່ ຢູື່ໃນຫໍ້ອງຮຽນ ຫືຼເຮັ ດຢື່າງອືື່ ນທີື່ ບໍື່ ກື່ຽວຂໍ້ອງກັ ບການສອນໃນຫໍ້ອງຮຽນ ❑ LEAPS II – Final Impact Evaluation Report  February 2020 I25 2 The teacher is engaged in disciplining a child child ນາຍຄູໄດໍ້ລົ ງໂທດ ຫືຼ ຕັ ກເຕື ອນເດັ ກນໍ້ ອຍໃຫໍ້ຢູື່ໃນລະບຽບວິ ໄນ ❑ 3 The teacher is engaged in classroom management management ນາຍຄູໄດໍ້ມີ ສື່ວນຮື່ວມໃນການຄຸໍ້ມຄອງຈັ ດການຫໍ້ອງຮຽນ ❑ 4 The teacher is engaged in the same activity as the students ນາຍຄູເຮັ ດກິ ດຈະກໍ າອັ ນດຽວກັ ນກັ ບນັ ກຮຽນ ❑ 5 The teacher is engaged with another grade (only in case of multigrade) ນາຍຄູສອນບົ ດຮຽນໃຫໍ້ກັ ບນັ ກຮຽນຫໍ້ອງອືື່ ນ (ສໍ າລັ ບຫໍ້ອງສອນຄວບ) ❑ Number of distracted students (out of class, chatting, sleeping, drawing, looking by the window, fighting): ຈໍ ານວນນັ ກຮຽນທີື່ ບໍື່ ເອົ າໃຈໃສື່ຮຽນ (ບໍື່ ຂືໍ້ນຫໍ້ອງ, ລົ ມກັ ນ, ນອນ, ແຕໍ້ມຮູບ, ເບິື່ ງອອກໄປນອກປື່ອງຢໍ້ຽມ, ຕີ ກັ ນ): B : _____ G : _____ All : _____ ຊາຍ: _____ ຍິ ງ: ____ທັ ງໝົ ດ____ THIS PAGE IS REPEATED 10 TIMES, FOR SNAPSHOT ONE TO 10. ໜ ານີແມ ນໃຫ ເຮ ດຊ າຄ ນ 10 ເທ ອ, ສ າລ ບ SNAPSHOT ທີ1 ຫາ 10 b. Ending time ເວລາສີ ້ ນສຸ ດ _ _ h _ _ mn ໂມງ _ _ ນາທີ Comments ຄ າເຫ ນ I26 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report 6. Teaching and learning materials used (to fill after the observation) ປະກອນທີື່ ນໍ າໃຊໍ້ໃນການຮຽນການສອນ(ຕືື່ ມຂໍໍ້ມູນໃສື່ຫັ ງຈາກເຮັ ດການສັ ງເກດ ຼ ) Teaching and learning materials used during the lesson ອຸປະກອນທີື່ ນໍ າໃຊໍ້ໃນການ ຮຽນການສອນໃນລະຫວື່າງ ທີື່ ສອນບົ ດຮຽນ ❑ Little objects that students can touch and handle (letters, numbers, geoboards, spinners, puzzle...) ວັ ດສະດຸນໍ້ ອຍທີື່ ນັ ກຮຽນສາມາດຈັ ບ ແລະ ຖື ໄດໍ້(ຕົ ວໜັ ງສື, ຕົ ວເລກ, ຈິ ກຊໍ ....) ❑ Official school book ປືໍ້ມແບບຮຽນ ❑ Exercise book ປືໍ້ມບົ ດຝຶ ກຫັ ດ ❑ Slates ກະດານນໍ້ ອຍ ❑ Blackboard ກະດານດໍ າ ❑ Reading illustrated texts ປືໍ້ມອື່ານທີື່ ມີ ຮູບພາບປະກອບ ຫືຼປືໍ້ມກະຕູນ ❑ Map, globe ແຜນທີື່ , ໜື່ ວຍໂລກ ❑ Others (please specify): ອືື່ ນໆ (ກະລຸນາລະບຸ): 7. Individual and small groups’ instruction ການແນະນໍ າແບບບຸ ກຄົນ ແລະ ແບບເປັ ນກຸື່ ມນ້ອຍ a. During the lesson, did the teacher organize different activities for small group of students or some students? ໃນລະຫວື່າງສອນບົ ດຮຽນ, ນາຍຄູໃຫໍ້ນັ ກຮຽນເຮັ ດກິ ດຈະກໍ າທີ ແຕກຕ າງກ ນ ໃຫ ກ ບນ ກຮຽນເປ ນກຸ ມນ ອຍ ຫລ ກ ບນ ກຮຽນ ບາງຄົ ນບ ? ❑ No ບໍື່ ໄດໍ້ເຮັ ດ ❑ Yes ໄດໍ້ເຮັ ດ [If yes] How much of the lesson did the teacher spend working individually with students? ຖໍ້າໄດໍ້ເຮັ ດ ນາຍຄູເຂົໍ້າໄປເວົໍ້າອະທິ ບາຍກັ ບນ ກຮຽນໂດຍສະ ເພາະ ຫຼາຍປານໃດໃນຕອນສອນບົ ດຮຽນນັໍ້ນ? ❑ Less than half the lesson ໜໍ້ ອຍກວື່າເຄິື່ ງໜຶື່ ງຂອງການສອນບົ ດຮຽນ ❑ About half the lesson ປະມານເຄິື່ ງໜຶື່ ງຂອງການສອນບົ ດຮຽນ ❑ More than half the lesson ຫຼາຍກວື່າການສອນບົ ດຮຽນເຄິື່ ງບົ ດ ❑ The entire lesson ຕະຫລອດການສອນບົ ດຮຽນ LEAPS II – Final Impact Evaluation Report  February 2020 I27 Comments (on attentiveness, on learning environment, on activities carried out, on teacher’s behavior and attitudes, or on any features you will find relevant) ອະທິ ບາຍ (ໃນຄວາມເອົ າໃຈໃສື່, ສະພາບແວດລໍ້ອມການຮຽນ, ກິ ດຈະກໍ າທີື່ ໄດໍ້ເຮັ ດ, ພຶ ດຕິ ກໍ າ ແລະ ທັ ດສະນະຂອງນາຍຄູ, ຫືຼ ທຸກຢື່າງທີື່ ເຈົໍ້າເຫັ ນວື່າກື່ຽວຂໍ້ອງ) ______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ ______________________________________________________________________ _______________________________________________________________________ LEAPS II – Final Impact Evaluation Report  February 2020 J1 APPENDIX J. QUALITATIVE PROTOCOLS MIDTERM DATA COLLECTION FOR USDA FOOD FOR EDUCATION (LEAPS II) IN LAOS Qualitative Instruments (FGD and KII) Consent Language Hi! Our names are XXXX, and we are here talking to parents such as yourself on behalf of Catholic Relief Services (CRS) to inform an education related project in Savannakhet Province. CRS will use this information so that they can improve future programs. It’s important that everyone here can feel comfortable to speak freely. Everything that you say here will be confidential, and we will not record your name or any personal information. I want you to talk openly and freely, so please also keep everything that anyone says here private. This is a friendly discussion, and there aren’t any right or wrong answers, so it’s okay if you disagree with someone else. If I ask you anything that makes you uncomfortable, it’s okay for you not to answer. We want to hear about positive as well as negative stories, because hearing about things that have and haven’t worked well can be really useful so we can design better programs in the future—so please do not feel ‘kreng jai’ [not speaking one’s mind for fear of offending the listener, putting social protocol ahead of truth telling]. Does anyone have any questions for me? You can ask me a question at any time during our talk today. Let’s start by going around and introducing ourselves. What is your name? [Interviewer begins by introducing yourself first.] Additional Language if Recording was Requested from Non School and Community-Level Stakeholders With your permission, I would like to record the conversation to help with my notes later – I’m not sharing this with anyone. Do I have your permission to do this? It is ok to say no if you would prefer that I did not. [Note taker will take detailed notes even if permission is given. The note taker will also record gender and ethnicity if known.] Focus Group Discussion: Parents 1. Tell me about your family: How many children do you have in primary school? [if yes, number of boys, number of girls] 2. Do they go willingly and look forward to attending school? Why or why not? What do they like or dislike about school? 3. How commonly do your children become ill and unable to attend school? If this is a significant issue of concern, what kind of support would you need to improve their health outcomes? 4. What do you like about the school your child attends? What, if anything, could be better? 5. Do you know your child’s teacher? What do you like about your child’s teacher? 6. Do you think children need to go to school? For how long? Why? Who decides if a child will go to school or not? (Probe for differences between boys and girls) 7. What are the particular skills that are important for children to learn at school? Are there particular skills that are more important for girls to learn than for boys? Are there particular skills that are more important for boys to learn than for girls? J2 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report 8. Do you encourage your children to study at home? Why or why not? How else are you involved in your child’s education? Is this typical of families in your community? Let’s talk a little bit about the benefits of education for children. 9. How do children benefit from school? [Probe for both immediate benefits and future benefits] a. What are some of the benefits of sending BOYS to school? b. What are some of the benefits of sending GIRLS to school? 10. How does your child attending school benefit your household? 11. Do children attending school benefit your community (village)? Why? Why not? I’d like to talk now about some of the challenges of sending children to school. 12. In some villages, not all children are able to attend school on a regular basis. Does this happen in your village? Are there some children who attend school more than others? What prevents some children in this community from going to school? (Probe to understand if there are different things that prevent boys and girls from going to school) 13. Are there sometimes specific difficulties for your family in sending your children to school? What are they? What helps you overcome these obstacles? 14. What are the consequences if a child does not regularly attend school? (Probe to understand immediate consequences/actions taken and longer term effects of not regularly attending.) 15. What do you think would encourage children in your community to go to school more often? (Probe to understand if what will encourage boys and girls to go to school is the same or different) 16. Do your children have any challenges at school? What are they? What can you do to help them to overcome these? 17. Which year of schooling do children in this village generally complete? Grade 3? Grade 5? Lower secondary school? Would children in the village like to continue studying after [last year of school group mentions] Why? Why not? 18. How many years of school would you like to see your own child/children complete? Why? What difference will it make it their lives to have this level of education? Now, I’d like to understand more about your opinions of the school meals program. 19. Are you familiar with the school meals program? What do you know about it? If you have been involved, what kind of support have you provided? 20. What do you think of the food items served at school and/or the take-home rations? 21. Do you think the program supports children to attend school? Why or why not? 22. From your perspective, does the program adequately address the needs of children and their families? What would you do differently? What about the needs of the community at large? 23. In your opinion, do you feel more encouraged to send children to school knowing that they have school meals available? Why? Why not? 24. If your children attended the summer reading camps, what did you think of the activities? How would you improve the program? LEAPS II – Final Impact Evaluation Report  February 2020 J3 25. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today? Thank you all very much for your sharing with me today, your feedback is much appreciated. J4 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Focus Group Discussion: Students 1. Tell me about your family: Who takes care of you at home? How many people are there in your household? Do you have any siblings attending school with you? 2. What is your favorite thing about school? 3. What do you not like about school? 4. What kind of support do you receive at home with your schoolwork? Is there anyone who helps you when the schoolwork is difficult? 5. Can anyone tell me about the times when you are supposed to wash your hands? How do you know this, who taught you? [If they mention washing hands before meals], are you able to do this every time you eat? What about before eating lunch at school? Are you able to wash your hands every day? Let’s talk a little bit about eating at school. 6. What do you think of the food items served at school? Does it taste good to you? How is it usually prepared? What type or kind of food would you like to see in the school meal? 7. How does eating the school meal make you feel? [Probe to see if it is enough food to make them full, and if it helps them to concentrate in class] 8. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today? Thank you all very much for your sharing with me today, your feedback is much appreciated. LEAPS II – Final Impact Evaluation Report  February 2020 J5 Focus Group Discussion: Village Education Development Committee (VEDC) 1. What motivated you to become a VEDC member? 2. What VEDC responsibility do you like the most? Why? 3. What VEDC responsibility do you like the least? Why? What is the most challenging aspect? 4. Who in your group is primarily responsible for the decision making? Do men and women participate equally in your VEDC? 5. Have you been a part of or seen any successful activities related to education or school through your work as a VEDC member? How did it make you feel? What aspects of the activity(ies) do you feel were the most successful? 6. Were there any activities the VEDC tried to do this year or last year related to education or school which were not successful? Why were they not successful? What needs do you think remain unmet, and what can be done to address them? 7. What kinds of training have you received as a VEDC member? (Probe: Who provided them? What did you learn? Did you feel the training was sufficient for your role?) Now, I am going to ask you about the school meals program in your community. 8. How has your VEDC been involved with the program? What kind of support have you provided? 9. Can you tell me about your interactions with CRS? (Probe: Do you know how to share information with them? Do you feel like they are considering your opinions? How have you learned from them?) 10. In what ways, if any, has the program improved your community? What about the capacity of the VEDC? Are there any other things about the community that has changed during the time of the program? 11. In your opinion, is there anything about the project that could be strengthened or done differently? 12. What support do you think CRS could provide to your VEDC so that you can adequately continue school feeding after the program ends? Do you have any recommendations to ensure the sustainability of the program? Thank you. I want to ask you now a few questions about the water, sanitation, and hygiene (WASH) interventions. 13. What has been your involvement in sustaining/maintaining school latrines, water points, and handwashing facilities? How has your role changed as a result of the WASH interventions, or have they stayed the same? 14. Can you describe your interaction, if any, with the Water User Committee? How do they support your work, and how do you ensure a cohesive effort to achieve goals? 15. To what extent have these interventions, in your opinion, increased schools’ access to clean water and sanitation services? Has this access changed health and hygiene practices? In what ways if so? If not, what steps could be taken to improve health and hygiene practices? 16. How do you think VEDCs could continue to play a role in ensuring access to clean water and sanitation services beyond the end of the program? 17. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today? J6 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Thank you all very much for your sharing with me today, your feedback is much appreciated. LEAPS II – Final Impact Evaluation Report  February 2020 J7 Focus Group Discussion: Cooks 1. Can you tell me about your responsibilities as a cook? 2. What motivated you to become a cook? How long have you been a cook with this school? Did the promise of take-home rations impact your decision to become a cook? 3. What is a responsibility of your work that you like most? Why? 4. What is a responsibility of your work that you like least? What is it about this that you find to be most challenging? 5. What do you find to be most rewarding in your work? 6. What is your connection to the school? Do you have any children who attend? What role, if any, does the school play for your community? 7. Do you think attending school is important for children? How do children benefit from school? (Probe for both immediate benefits and future benefits) Now, I want to ask you a few questions about the program and its different activities. 8. What do you think of the trainings that the program has offered? What have you learned? Do you feel that the trainings have increased your knowledge of safe food preparation and storage practices? What kinds of things have you learned? 9. How do you think the trainings could be more effective? What else do you think it could address to meet your needs? 10. How have you incorporated locally-available foods into school meals? What kind of support have you received from the community and/or VEDC in doing so? 11. How have the WASH interventions contributed to changes in health and dietary practices? (Probe: Are there new procedures in place? Are children eating different kinds of food? Do you see children washing their hands more frequently?) 12. What are some challenges to ensuring appropriate WASH practices during food preparation? 13. When this project ends, would you continue to serve as a cook? What would encourage you to continue with your work? 14. Have you been a part of or seen any successful activities through your work as a cook or storekeeper? How did it make you feel? What aspects of the activity(ies) do you feel were the most successful? 15. Were there any activities as part of your job which were not successful? Which were the least successful? What needs do you think remain unmet? 16. In your opinion, do you believe that the availability of school meals has encouraged students to attend school? Why? Why not? Have you seen a change in students (e.g. attitude, behaviour, drop-out) as a result of the program? 17. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! J8 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Focus Group Discussion: Teachers 1. Let’s start by talking a little bit about your background – how long have you been teachers? How long have you been at this school? 2. Are there families in this village who face challenges in sending their children to primary school and supporting their education? If yes, can you give some examples of the challenges people face? Who makes the decision about whether a child attends school or not? (Probe to understand if there are different reasons that prevent boys and girls from going to school, and if there are different challenges by age group, i.e. in early primary school vs later) 3. When we surveyed students before the program started, they mentioned missing school because of sickness and other health issues. How commonly are students absent because of an illness? Does this issue represent an area of concern? 4. What are the most important things you would like to see improve with regard to education in this village? 5. Now I want to ask you a few questions about the LEAPS project. What do you think of the teacher and administrator training programs organized? (Probe to learn more about whether the topics addressed are relevant to beneficiary needs and whether these trainings support the challenges that teachers and administrators face in their jobs) 6. To what extent do you feel these trainings have improved your ability to conduct your job? Can you tell me an example of something that has changed because of these trainings? If nothing has changed, why not? What could make the program more effective? 7. I want to talk a bit now about the school feeding component of the program. How have the take-home rations benefited you? Is there anything else that encourages or motivates you to come to school, or stay engaged? 8. How do you think the school feeding program has affected students and their behavior? (Probe if there has been any effect on student attendance, parent inclination to send students to school, attentiveness in the classroom, etc.) 9. What about the cooks and storekeepers? Do you think take home rations have encouraged them to come to school? Is there anything else that encourages or motivates them to come to school, or stay engaged? 10. Have you witnessed any changes in your school’s access to clean water and sanitation services? If so, how has this improvement affected health and hygiene practices? If not, what kind of support would help to achieve better these kinds of practices? 11. What do you think of the training that you have received regarding improved WASH practices? How successful do you consider these messages, and have you and/or your students applied them? 12. I would like to ask a few questions now about the VEDC in your community. What kind of support have you received from them, and what do you think of the current interaction with them? a. How could the VEDC further support schools, including school meals? b. Do you think the LEAPS project has helped to enhance the capacity of the VEDC? If so, how? If not, why not, and what could be improved? 13. Overall, what do you think have been the greatest impacts as a result of the LEAPS project? LEAPS II – Final Impact Evaluation Report  February 2020 J9 14. Have there been any unexpected impacts? (Probe to ask about anything they have seen change within the community or school which they find surprising) 15. How have these impacts varied for different types of people (e.g. boys vs girls, ethnic vs. Lao communities)? (Probe to ask about effect on literacy and drop-out rates) 16. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! J10 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Key Informant Interviews: Community Mobilizers 1. Let us start by talking a little bit about your background – when did you become a community mobilizer? In your role, how do you engage with the community and the school? (Probe to see how they interact with the VEDC and school teachers/principals.) 2. What do you find most rewarding and enjoyable about this job? What are the difficulties? 3. Given your job requires coordinating with multiple stakeholders, can you tell me about these interactions and your experience communicating with these different individuals? What have you learned? Have there been any challenges? If so, can you describe them? 4. Have you received any kind of training as a community mobilizer? If yes, what kind of trainings have you received and how have they contributed to your ability to perform your job? (Probe if they have not received training to ask what kind of training they would like to receive.) 5. Can you describe how the community currently supports the progress of the school? Do you feel the current level of support is sufficient? What else do you think the community could do? 6. What do you think should be priorities for the schools moving forward? How do you think the program could be more effective to achieve those priorities? 7. How do you think the school could better engage the community? 8. Based on the work that you have done and the progress you have witnessed in your schools, which activities have been the most successful in your view and why? Which activities have had little success? Why do you think they have not been as successful? 9. Have you witnessed any changes in your school’s access to clean water and sanitation services? If so, how has this improvement affected health and hygiene practices? If not, what kind of support would help to achieve better these kinds of practices? 10. Overall, what do you think have been the greatest impacts as a result of the LEAPS project? How have these impacts varied for different types of people (e.g. boys vs girls, ethnic vs. Lao communities)? (Probe to ask about effect on literacy and drop-out rates.) 11. Is there anything that I did not ask about that you would like to share with me? LEAPS II – Final Impact Evaluation Report  February 2020 J11 Key Informant Interview: Principals 1. Let’s start by talking a little bit about your background – how long have you been a principal? How long have you been at this school? 2. What are your priorities as a leader in terms of developing the school? 3. Are there families in this village who face challenges in sending their children to primary school and supporting their education? If yes, can you give some examples of the challenges people face? Who makes the decision about whether a child attends school or not? (Probe to understand if there are different reasons that prevent boys and girls from going to school, and if there are different challenges by age group, i.e. in early primary school vs later) 4. When we surveyed students before the program started, they mentioned missing school because of sickness and other health issues. How commonly are students absent because of an illness? Does this issue represent an area of concern? 5. What are the most important things you would like to see improve with regard to education in this village? 6. Now I want to ask you a few questions about the LEAPS project. What do you think of the teacher and administrator training programs organized? (Probe to learn more about whether the topics addressed are relevant to beneficiary needs and whether these trainings support the challenges that administrators face in their jobs) 7. To what extent do you feel these trainings have improved your ability to conduct your job? Can you tell me an example of something that has changed because of these trainings? If nothing has changed, why not? What could make the program more effective? 8. What do you think about the summer reading camps organized as part of this program? In what ways were these camps effective? How could they be improved? 9. I want to talk a bit now about the school feeding component of the program. How have take-home rations changed teacher behavior? Is there anything else that encourages or motivates teachers to come to school, or stay engaged? 10. How do you think the school feeding program has affected students and their behavior? (Probe if there has been any effect on student attendance, parent inclination to send students to school, attentiveness in the classroom, etc.) 11. What about the cooks and storekeepers? Do you think take-home rations have encouraged them to come to school? Is there anything else that encourages or motivates them to come to school, or stay engaged? 12. Have you witnessed any changes in your school’s access to clean water and sanitation services? If so, how has this improvement affected health and hygiene practices? If not, what kind of support would help to achieve better these kinds of practices? 13. I would like to ask a few questions now about the VEDC in your community. What kind of support have you received from them, and what do you think of the current interaction with them? a. How could the VEDC further support schools, including school meals? b. Do you think the LEAPS project has helped to enhance the capacity of the VEDC? If so, how? If not, why not, and what could be improved? J12 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report 14. Overall, what do you think have been the greatest impacts as a result of the LEAPS project? How have these impacts varied for different types of people (e.g. boys vs girls, ethnic vs. Lao communities)? (Probe to ask about effect on literacy and drop-out rates.) 15. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! LEAPS II – Final Impact Evaluation Report  February 2020 J13 Key Informant Interview: Government Stakeholders (PESS and DESB representatives) 1. Can you tell me about your responsibilities as [PESS or DESB] representative? How do you coordinate and work with the MoES? 2. How have been you involved with the LEAPS II project? What kind of role do you play? 3. What have been some of the most successful elements of the project from your perspective? What has been the most challenging? 4. How do you feel the program aligns with the Government of Laos’ priorities and policies? If they contrast, why? 5. In your opinion, and from the feedback that you have gotten from the communities and the schools, do you feel LEAPS II has improved literacy outcomes for children? What about drop-out rates? 6. What ideas do you have to increase the effectiveness of school feeding so that it can better boost literacy outcomes? What do you see as critical elements to retain after the project ends? How should MoES staff continue to support and scale up literacy education? 7. What kinds of actions or steps could LEAPS II take now to ensure that MoES staff have the necessary training to continue the activity? 8. How has the program integrated your opinions and considered your needs? If you feel they have not, what steps could they take to increase this flow of communication? 9. What kinds of lessons learned or best practices have you acquired from the project that you feel is useful to your job? If you feel you have gained limited knowledge, what types of learning mechanisms do you think should be in place to more effectively facilitate learning? 10. In your opinion, how do you think the program has helped to improve the capacity of [PESS or DESB]? If you think it has not done much, why? What steps could be taken so that the program can more strongly improve capacity? 11. What kind of support and/or interactions do you have with the community and/or VEDC members? How do you engage with them? 12. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! J14 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Key Informant Interview: Pedagogical Advisors (PAs) 1. Can you tell me about your role and responsibilities? 2. What do you think are the biggest gaps in improving the quality of classroom instruction? 3. How have you seen teaching practices evolve as a result of the training? If they have not changed, why not? What about the trainings could be made more effective? 4. Can you tell me any specific anecdotes from classroom observations or trainings that best illustrate how the training is improving teacher quality? Or conversely, any anecdotes of these trainings not achieving its intended goal? 5. What kind of general feedback have you gotten directly from teachers and principals regarding the trainings? 6. How well do you think the trainings align with the goals and priorities of the Government of Laos? If they contrast, why? 7. In your opinion, what factors account for students struggling with attentiveness in the classroom? Has the school feeding program affected this attentiveness at all? If so, in what ways? If not, why not? Have you observed any changes in child performance in classroom based on your classroom observations and monitoring visits? 8. What do you think of the current monitoring system and classroom observation tools used to track school performance? How easy have they been to learn/use? In what ways could be they made more effective? 9. Can you tell me about your experience working with project staff to conduct these trainings and school visits? Have there been any challenges? What lessons have you learned from them? 10. How well-equipped do you feel to independently conduct these training sessions and monitoring visits after the program ends? Why? Why not? In which areas do you think you will require further support? 11. Can you describe your interactions, if any, with the community and/or VEDC? 12. What are the biggest challenges you face in your role as a PA? 13. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! LEAPS II – Final Impact Evaluation Report  February 2020 J15 Key Informant Interview: Community Literacy Volunteers (CLVs) 1. What motivated you to become a CLV? 2. Can you tell me about your role and responsibilities? 3. What do you like about being a CLV? What do you dislike? 4. What do you think are the biggest barriers to improved child literacy? 5. How effectively do you think the literacy activities undertaken as part of the program addresses these deficits? If not effectively, what more can be done? 6. What are the biggest challenges in implementing these literacy activities? What has worked well? Why do you think those activities have been successful? 7. How do you think these literacy activities complement school instruction? 8. From your experience and memory, can you think of any specific examples from the Reading Camps that best illustrates how its activities have contributed to improving student literacy outcomes? 9. Can you tell me your experience working with program staff? (Probe: What have you enjoyed? Have there been any challenges? What have you learned?) 10. Can you tell me your experience working with the community and/or VEDC (Probe: What have you enjoyed? Have there been any challenges? What have you learned?) 11. Based on your interactions with parents and the community, what has been their feedback regarding the program and the literacy activities? 12. From your experience working with parents, how have their attitudes and opinions toward schooling evolved since you have begun activities with them? Have you seen any impact of these literacy activities on their awareness of the importance of education and sending their children to school? 13. Do you feel that you have received adequate training to carry out your job? If yes, in what ways has this training helped? If no, what could be more effective? 14. Based on your experience, what other activities might be helpful to enhance child literacy? 15. What kinds of changes have you seen in the community as a result of this program? After the program ends, what kind of support do you think the community could provide to continue these types of literacy activities? 16. Is there anything that I did not ask about that you would like to share with me? Thank you for taking the time to speak with us today, your feedback is very valuable! J16 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Key Informant Interview: Project staff 1. What is your title? How long have you been with [organization]? What are your main responsibilities in your position, particularly related to the FFE project? How long have you been involved with this project? 2. Were you involved or consulted in the design phase? If so, in what ways? What are the strengths of the project’s design? What are its weaknesses? 3. Do you think the planning and organization of the FFE project were well-planned and realistic in terms of its objectives, desired outcomes, targets, and timeframe? Why or why not? 4. In what ways/how well do you think the program’s goals fit with the government’s educational and health priorities (at the national, regional, and/or local level)? 5. Since the project began, have beneficiaries’ needs changed over time in a way that has affected the project? If so, how has the project responded to changing needs? 6. Are the project’s planned activities proceeding on schedule? What has helped stay on schedule and what has made it difficult? 7. What outputs or outcomes has the project achieved so far? Are expected results occurring as planned? To what extent will the objectives be achieved? 8. Are the implementation of some activities more successful than others? If so, which ones? Why? 9. How effective is the monitoring strategy for collecting regular and reliable data on the work? What are the strengths and gaps in the monitoring system? 10. Overall, what are the successes and challenges experienced in the implementation process? How can they be addressed for better achievements in the future? 11. Based on your knowledge, to what extent are the planned activities being implemented according to budget? 12. What obstacles have arisen in terms of allocating budget resources? How were they overcome and at what cost? 13. Do you think the project is implemented in the most efficient way? In what ways can it be more efficient? Are objectives being achieved on time? Why or why not? 14. What are the impacts, if any so far, of the activities on communities where the FFE project is being implemented? Which activities do you think have the greatest impacts? Why? 15. Is there a difference in the way that the activities are affecting boys and girls? 16. What has been the impact of the project’s activities on the communities in terms of: a. Building organizational capacities (VEDCs) b. Raising awareness in parents/teacher/students (e.g. on the importance of education, promoting attendance for students and teachers) 17. How are community-based structures (e.g. schools, VEDCs) supporting project implementation? Are they on track to assume ownership of key activities beyond the life of the project? Are they satisfied with their participation? How might they be encouraged and/or supported to participate more? LEAPS II – Final Impact Evaluation Report  February 2020 J17 18. How have capacity building activities for VEDCs improved their capacities? What obstacles persist? What more should be done to ensure they will have the capacity to manage the school canteens beyond the life of the project? 19. What innovations, lessons learned, and good practices can be documented so far? 20. From your perspective, which activities and processes will be sustainable beyond project funding and which will not be sustainable? Please explain. What are the biggest challenges to sustainability? Is the project prepared to overcome these challenges? 21. How has the project engaged other stakeholders (e.g. government and local organizations) to sustain the project activities and outcomes after the project funding ends? What will be their role in sustaining the outcomes of the project at the district and community levels? What systems do you think they have in place to sustain the outcomes? 22. Do any sociocultural or political aspects endanger the sustainability of the project and what actions are being taken to sensitize local institutions and target groups to these issues? 23. What measures, and which specific efforts, have been undertaken already to prepare for the phase out of the project’s funding? 24. Do you have any recommendations to help ensure sustainability of the project? J18 IMPAQ International, LLC  LEAPS II – Final Impact Evaluation Report Key Informant Interview: USDA staff 1. Can you tell me a bit about what your role has been in the LEAPS II project? 2. From your perspective, what are the most critical aspects for CRS/Laos to focus on to make sure that this project is a success? 3. Based on your knowledge of LEAPS II as well as other FFE projects, what are the strengths and weaknesses of the design of LEAPS II? 4. How well does LEAPS II align with USDA’s priorities and trends? 5. How effective/efficient do the management structures seem to be? (Probe for specific examples) 6. How do you see the evaluation design – as approved – aligning with USDA’s overall interest in evaluation, as well as USDA’s learning agenda? 7. With respect to sustainability, what should CRS/Laos focus on in LEAPS II to improve sustainability and a successful transition from donor support? 8. Please tell me about any specific factors that you think might affect the LEAPS II project’s chances to succeed, now and in the future? Please explain why and how you think this factor/these factors could influence LEAPS II in the future? As you know, we are complementing the performance evaluation by collecting qualitative data. At baseline, our questions focused on verifying project design assumptions and identifying potential threats to implementation, including formative feedback on planned activities. At midterm, questions will focus on learning what occurred to date, including a formative assessment and suggestions for program process improvement. At endline, questions will focus on learning more about the potential for program sustainability and promising practices and lessons learned. 9. We would like to get USDA’s feedback on our qualitative data collection. What elements are you most interested in that we are unable to capture in our quantitative analysis? 10. Is there anything else that you’d like to share with me today? Thank you for your time and comments. LEAPS II Midterm Performance Evaluation Report  February 2020 KL19 APPENDIX K. MIDTERM EVALUATION SOW AND TOR Redacted