This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by NRMC. Foreign Agricultural Service, United States Department of Agriculture Lao – PDR McGovern-Dole Food for Education Project Final Evaluation October 2018 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Decentralized Evaluation for USDA McGovern Dole FY14 End￾Line Evaluation in Lao PDR (FY14-16) The United States Department of Agriculture (USDA) McGovern-Dole Food for Education Program in Lao PDR was implemented by World Food Programme from 2014-2018. The aim of the program is to improve the literacy, health, and dietary practices of primary-school-aged children. Agreement Number: FFE-439-2014/049-00 Project Duration: 2014-2018 Implemented by: World Food Programme Evaluation Authored by: NRMC Evaluation Manager: Utomo Tjipto Team Leader: Bhaskar Goswami Team Members: Malay Das, Ankita Singh, Prapti Barooah Prepared by Team Leader: Bhaskar Goswami Team Member: Malay Das, Ankita Singh, Prapti Barooah October 2018 WFP Country Office Lao PDR Mr. Utomo Tjipto, WFP Lao PDR Evaluation Manager USDA Mc-GOVERN DOLE FY14END-LINE EVALUATION IN LAO PDR [FY 14-16] Report of End line Evaluation Decentralized Evaluation Decentralized evaluation for evidence-based decision making Acknowledgements The NRMC evaluation team wishes to acknowledge the guidance, support, and cooperation received from all the participants in the evaluation. NRMC takes this opportunity to extend sincere thanks to the distinguished Government officials at the Ministries, provincial level and district level for their time and precious inputs. The NRMC Evaluation Team expresses its gratitude to Ada Ihenachor and Tracy Johnson from USDA for their valuable suggestions and guidance for the evaluation. We would like to thank the members of RBB, Felicity Chard and Yumiko Kanemitsu, for sharing their useful insights. Their suggestions have immensely helped in enhancing the design of the evaluation. We would also like to thank the staff of international development agencies (The World Bank, UNICEF, UNFPA, Australian DFAT, Plan International, Big Brother Mouse, SNV, and GIZ) who kindly took the time to meet us and give us their views on the school feeding programme in Lao PDR. NRMC wishes to sincerely acknowledge the support and guidance received from the staff of the WFP Lao PDR Country Office, for assisting with the planning of and facilitating the evaluation mission, and for supplying documentation, especially to Sarah Gordon Gibson, Nanna Skau, Utomo Tjipto, Yangxia Lee, Serena Mithbaokar, Air Sensomphone, Khizar Ashraf, OuthaiSihalath, Phetsamone Southalack, Kevin Howley, Thai Thao, and the WFP staff from the Pakse and LuangNamtha sub-offices. We are grateful to the team from Geo-Sys for their partnership with the NRMC evaluation team throughout the period of evaluation, particularly for their untiring efforts for data collection. Last but not the least, the evaluation team wishes to acknowledge the cooperation received from all informants, including school children, school head, teachers, parents, cooks, storekeepers and VEDC members, during the primary survey. Disclaimer The opinions expressed in this report are those of the Evaluation Team and do not necessarily reflect those of the World Food Programme. Responsibility for the opinions expressed in this report rests solely with the authors. Publication of this document does not imply endorsement by WFP of the opinions expressed. The designation employed and the presentation of material in maps do no imply the expression of any opinion whatsoever on the part of WFP concerning the legal or constitutional status of any country, territory or sea area, or concerning the delimitation of frontiers. USDA Mc-Govern Dole FY 14 End-line Evaluation in Lao PDR Final Report i Table of Contents Acknowledgements List of Acronyms Executive Summary 1. Introduction ...................................................................................................................................... 1 1.1. The subject of the Evaluation ......................................................................................................................... 1 1.2. Context ....................................................................................................................................................................... 3 1.3. Evaluation Methodology ................................................................................................................................... 5 2. Evaluation Findings ......................................................................................................................... 9 2.1. The relevance of MGD SFP FY14 in Lao PDR .......................................................................................10 2.2. The effectiveness of MGD-SFP FY14 (based on performance indicators) ...........................12 2.3. Impact of MGD-SFP FY14 ..............................................................................................................................26 2.4. The efficiency of MGD-SFP FY14 ...............................................................................................................28 2.5. Sustainability of MGD-SFP FY14 ................................................................................................................30 2.6. Theory of Change for MGD-SFP FY14 .....................................................................................................33 3. Conclusions and Recommendations ........................................................................................ 33 3.1. Overall Assessment/ Conclusions.............................................................................................................33 3.2. Good practices and Lessons Learned ......................................................................................................36 3.3. Recommendations ............................................................................................................................................38 Annexes .................................................................................................................................................... 42 Annex A: Scope of Work for End-Line Evaluation ......................................................................................42 Annex B: Stakeholders and Report Users .......................................................................................................44 Annex C: Map of the Intervention Area ..........................................................................................................48 Annex D: List of Districts under MGD FY14 ...................................................................................................49 Annex E: Planned Outcomes ..................................................................................................................................50 Annex F: Break-up of Contribution from Non-USDA Funding for WFP School Feeding Programme ..........................................................................................................................................................................59 Annex G: Project-Level Results Framework ..................................................................................................60 Annex H: Theory of Change ....................................................................................................................................64 USDA Mc-Govern Dole FY 14 End-line Evaluation in Lao PDR Final Report ii Annex I: Role of WFP's partners in MGD SFP ..............................................................................................70 Annex J: Status of Poverty, Nutrition and Education in Lao PDR, 2012 ........................................71 Annex K: ODA Assistance to Lao PDR ...............................................................................................................74 Annex L: Evaluation Mission Schedule ............................................................................................................78 Annex M: Evaluation Matrix ...................................................................................................................................85 Annex N: Evaluation Approach and Methodology ......................................................................................98 Annex O: Documents Gathered .........................................................................................................................107 Annex P: List of Sample Schools .......................................................................................................................111 Annex Q: End-line Values of Key Indicators ...............................................................................................115 Annex R: End-line Values of Other Key Performance Indicators ....................................................121 Annex S: Results from EGRA sub-tests .........................................................................................................131 Annex T: Baseline and MTE recommendations and follow up .........................................................134 List of Tables Table 1: Target beneficiaries disaggregated by sex ................................................................................................. 2 Table 2: Summary of the evaluation methodology .................................................................................................. 7 Table 3: Gender disaggregated data for the proportion of students with at least 75 percent comprehension of Lao language .....................................................................................................................................16 Table 4: Capacity building activities undertaken by MGD-SFP .......................................................................25 Table 5: Comparison of different school meal programmes ............................................................................30 Table 6: Recommendations, rationale and proposed actions (for FY17 award) ...................................38 Table 7: Scope of Work for End-Line Evaluation ...................................................................................................42 Table 8: Stakeholder mapping .........................................................................................................................................44 Table 9: SFP’s coverage by provinces and districts ..............................................................................................49 Table 10: Strategic objectives, expected outcomes, and activities under the MGD supported SFP ..........................................................................................................................................................................................................50 Table 11: Planned and updated performance indicators (Source: Performance Monitoring Plan, WFP 2017)..................................................................................................................................................................................52 Table 12: Assumptions and risks....................................................................................................................................66 Table 13: Comparison of Planned and Actual ODA disbursements from 2011 to 2015....................74 Table 14: Disbursements by Development Partners ...........................................................................................74 Table 15: ODA projects by SDGs .....................................................................................................................................75 Table 16: Evaluation Mission Schedule .......................................................................................................................78 Table 17: Evaluation Matrix ..............................................................................................................................................85 Table 18: Summary of the evaluation methodology..........................................................................................100 Table 19: Sample covered during quantitative survey Phase 1 ..................................................................103 Table 20: Sample covered during quantitative survey Phase 2 ..................................................................104 Table 21: Sample covered during qualitative survey Phase 1......................................................................104 Table 22: Sample covered during qualitative survey Phase 2......................................................................105 USDA Mc-Govern Dole FY 14 End-line Evaluation in Lao PDR Final Report iii Table 23: Documents Gathered ....................................................................................................................................107 Table 24: End-line values of MGD-SFP (FY14) performance indicators, disaggregated by sex and geographical location .........................................................................................................................................................115 Table 25: EGRA sub-tests and their measurement approaches ..................................................................132 Table 26: Performance of students on other EGRA sub-tests, baseline-end-line comparison ....132 Table 27: Baseline and MTE recommendations and follow up ...................................................................134 List of Figures Figure 1: Ethnicity of students .........................................................................................................................................15 Figure 2: Reading comprehension categorization based on EGRA (Baseline-End line Comparison) ..............................................................................................................................................................................16 Figure 3: Categorising Dietary Diversity Score, by baseline and end-line ................................................18 Figure 4: Issues faced by cooks in preparing school lunch ...............................................................................21 Figure 5: Harvest from school gardens in the last one year, by geography .............................................22 Figure 6: MGD Laos Implicit Theory of Change.......................................................................................................64 Figure 7: ODA disbursements by Sector Working Groups ................................................................................76 List of Boxes Box 1: Status on partial absenteeism ...........................................................................................................................15 Box 2: Setting up Model Schools: Key to boost effectiveness and utilisation of school gardens ...23 USDA Mc-Govern Dole FY 14 End-line Evaluation in Lao PDR Draft Report iv List of Acronyms BBM Big Brother Mouse (Local NGO) BEQUAL Basic Education Quality and Access in Lao PDR CO Country Office CP Country Programme CPE Country Programme Evaluation CSP Country Strategic Plan CRF WFP Corporate Results Framework CRS Catholic Relief Services CSB Corn Soya Blend CSO Civil Society Organization DAC Development Assistance Committee (of the OECD) DAFO District Agriculture and Forestry Office DDS Dietary Diversity Score DEQAS Decentralized Evaluation Quality Assurance System (of WFP) DESB District Education and Sports Bureau DFAT Department of Foreign Affairs and Trade (Australia) DP Development Partner EB Executive Board (of WFP) ECE Early Childhood Education ED Executive Director EDF Education Development Fund EGRA Early Grade Reading Assessment EM Evaluation Manager EMIS Education Management Information System EMOP Emergency Operation EP Evaluation Plan EQ Evaluation Question EQAS Evaluation quality assurance system (of WFP) ER Evaluation Report ESDF Education Sector Development Framework ESDP Education Sector Development Plan ESWG Education Sector Working Group ET Evaluation Team EU European Union FAD Food Assistance Division USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report v FAO Food and Agriculture Organization FFA Food assistance For Assets FFE Food for Education FGS Gravity Fed Water Supply Schemes FFR Food for Relief FFT Food for Training GAIN Global Alliance for Improved Nutrition GDI Gender Development Index GDP Gross Domestic Product GEEW Gender Equality and Empowerment of Women GFD General Food Distribution GIF Global Innovation Fund GIZ Deutsche Gesselschaft fur Internationale Zusammenarbeit GM Gender Marker GNR Global Nutrition Report GoL Government of Lao PDR GPE Global Partnership for Education HDI Human Development Index HGSF Home-Grown School Feeding HQ Headquarters IB Informal Boarders IEC Information, Education and Communication IFAD International Fund for Agricultural Development IFPRI International Food Policy Research Institute ILO International Labour Organization INGO International non-governmental organization IR Inception Report IVR Interactive Voice Recording LBW Low birth weight LDC Least developed Country LEAPS Learning and Engaging All in Primary School LIC Low Income Country LMIC Low Middle Income Country LRP Local and Regional Procurement – USDA LWU Lao Women’s Union MA Monitoring Assistants USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report vi MAF Ministry of Agriculture and Forestry MDG Millennium Development Goal MGD Mc-Govern Dole MMS Mid-Morning Snack MoES Ministry of Education and Sports MoH Ministry of Health MPI Ministry of Planning and Investment MT Metric Ton MTE Mid-Term Evaluation NAN National Plan of Action on Nutrition NER Net Enrolment Rate NNS National Nutrition Strategy NNSPA National Nutrition Strategy and Plan of Action NSEDP National Socio-Economic Development Plan NSMP National School Meals Programme ODI Overseas Development Institute ODA Overseas Development Assistance OECD Organization for Economic Cooperation and Development OEV WFP Office of Evaluation OLA UN Office for Legal Affairs OpEv Operation Evaluation PDR People’s Democratic Republic – Lao PI Plan International PLW Pregnant Lactating Women PESS Provincial Education and Sports Services – sub national ministry RBB Regional Bureau Bangkok RC United Nations Resident Coordinator REACH Renewed Effort against Child Hunger and Undernutrition SABER Systems Approach for Better Education Results SDG Sustainable Development Goal SEAMEO Southeast Asian Ministers of Education Organisation SFP School Feeding Programme SMAP School Meals Action Plan SNF Specialized Nutritious Food SNV SNV Netherlands Development Organisation SPR Standard Project Report USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report vii SRF Strategic Results Framework STH Soil-Transmitted Helminthiasis SUN Scaling-up Nutrition Movement THR Take Home Rations TOC Theory of Change TOR Terms of Reference TWG Technical Working Group UNDP United Nations Development Programme UNEG United Nations Evaluation Group UNESCO United Nations Educational, Scientific and Cultural Organization UNICEF United Nations Children's Fund UNFPA United Nations Population Fund USAID United States Agency for International Development USDA United States Department of Agriculture VEDC Village Education Development Committee WASH Water Sanitation and Hygiene WATSAN Water and Sanitation WFP World Food Programme WHO World Health Organisation ZHC Zero Hunger Challenge USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report i Executive Summary Introduction 1 The end-line evaluation for United States Department of Agriculture (USDA) Mc-Govern Dole Food for Education Grant (MGD) supported School Feeding Programme (SFP) FY14-16 award cycle (hereafter referred to as FY14) in Lao PDR was commissioned by WFP country office Lao PDR (WFP CO) between February-June 2018. The objective of the evaluation was to assess the implementation and performance of the FY14 award and generate recommendations that will strengthen and inform the operational and strategic decision-making for the FY17 award rollout. The evaluation served the dual purpose of accountability and learning. It assessed the performance and results of the implementation and determined the causalities towards achieving or missing the results. The evaluation has provided an evidence-based independent assessment of the performance of the operation that would enable WFP and program partners to take informed operational and strategic decisions for the FY17 award. 2 The primary stakeholders and users of this evaluation are a) WFP (WFP CO, Regional Bureau Bangkok, WFP-HQ, and Office of Evaluation), b) USDA, c) Ministry of Education and Sports (MoES), Lao PDR, f) Partners such as Big Brother Mouse, Plan International, SNV, Australian Department of Foreign Affairs and Trade (DFAT), UNICEF, and The World Bank are interested in the results of the evaluation. 3 Despite, the socio-economic progress made by Lao PDR, the Human Development Index of the country was estimated at 0.56 (2015). While the net enrolment rate is 98 percent, primary school dropout rate as a percentage of primary school cohort is high (22%)1. As per the Global Hunger Index 2015, about 44 percent of children under the age of five are malnourished2. In 2015, 35.6 percent of children under five years of age were stunted (37.6% boys; 33.6% girls), 25.5 percent were underweight and 9.6 percent were wasted. Under-5 mortality rate (U5MR) was 673 in 20154. Poor nutrition causes 45 percent of the U5MR5 and also leads to stunting and delays in physical development (HDR 2016). 4 The Government of Laos (GoL) aims at moving from Least Developed Country status to Middle Income country by 2020. In alignment with the Sustainable Development Goals (SDGs), WFP with support from MGD implemented the SFP FY14 across seven provinces6, covering 30 districts characterized by poverty, malnutrition, and low literacy rates. The US$ 27 million award commenced from Sep 2014 and continued till Aug 2016. The programme aimed to provide meals through the 175-day school year to 405,758 students7 (of which approximately 200,039 are girls) in 1,446 schools (WFP ToR, 2017). 5 WFP had established strategic partnerships and leveraged support from government institutions, UN agencies, donors, and NGOs towards achieving the MGD goals. In particular, WFP worked in close coordination with MoES, MPI, MAF, UNICEF, The World Bank, WHO, DFAT, BBM, PI, EDF, and CRS. Methodology 6 The evaluation methodology was guided by the ToR underpinned by the results framework of the FY14 programme. The difference in status of indicators over the programme period was examined by comparing the baseline data with that of the end-line through desk review and primary survey. Data from the primary survey was triangulated to assess its reliability. Gender Equality and Empowerment of Women (GEEW) was mainstreamed by disaggregating all school-level data by sex, and the sampling of students was done to ensure an equal representation of boys and girls. 7 The sample was spread across seven provinces covering ten districts. 60 schools were sampled covering 1155 students from Grade 1 to 5. It also included 984 parents, 58 cooks and storekeepers, 57 school heads, and 58 teachers. Additionally, 88 qualitative discussions were also held with parents, cooks, storekeepers, school head, teachers, and Village Education Development Committees (VEDC). The survey tools were translated to Lao language for easy administration of the tools. 1Human Development Report, 2016: Lao PDR http://hdr.undp.org/en/countries/profiles/LAO 2 Nutrition Country Profile (2015), Lao PDR International Food Policy Research Institute. (Source: UNICEF, WHO, WB 2015) 3http://www.la.one.un.org/sdgs/sdg-3-health accessed on 16 April 2018 4 U5MR for 2016 is slightly lower at 66.7 (HDR 2016) 5UNDP (2016). Human development report 2016: Human development for everyone. USA: UNDP 6Attapeu, Saravane, and Sekong provinces located in the south, and LuangNamtha, LuangPrabang, Oudomxay, Phongsaly provinces in the north 7 Study ToR USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report ii 8 Two significant limitations of this study area) unavailability of all the data points (including baseline data) for all the indicators, and b) method of assessing attentiveness. Attentiveness of a student is a function of different variables one of which is short-term hunger. Others may include poor teaching methods, lack of interest in the subject matter being taught in class, and incidents at home. The evaluation measures the attentiveness of children based on opinion of their teachers which may be biased based on the teacher’s perception of the child. Key findings 9 Relevance: The Mid-morning Snacks, and subsequently the school lunch, were implemented in the most vulnerable and food insecure areas in the country. By targeting school going children, the programme has contributed to the overall objective of improving the education and nutrition status in Lao. Through community contribution for school meals, the programme promoted ownership among the communities which will contribute to sustaining the programme after the exit of MGD in 2021. The programme was coherent with School Meals Action Plan (SMAP 2016-2020), Education Sector Development Plan (ESDP 2016-20), and National Nutrition Strategy to 2025 and Plan of Action 2016-20 (NNSPA) through which the government is combating malnutrition and attempting to improve the enrolment and attendance, and reduce dropout and grade repetitions of school children. 10 The programme has encouraged participation of the community which has translated to building ownership among the community. The transfer of technical knowledge on nutrition and school gardens have helped communities to diversify food production and their food basket. 11 Promotion of literacy activities at the community-level with participation of girls and mothers led to an improved awareness about the importance of education and nutrition. The establishment of a library and book corners in the schools created an enabling environment for children to study. Capacity building at the operational and strategic levels for Naiban, teachers, cooks, and storekeepers, officials at the centre, provinces, and districts was undertaken for better implementation of SFP. 12 MGD-SFP FY14 had emphasised on improving enrolment, attendance, and retention of both boys and girls. The gender components included the sharing of tasks in school gardens between boys and girls without defining tasks by sex, strengthening of nutrition education, and mainstreaming of gender equality considerations into field-level activities by using new literacy materials to challenge traditional gender roles. 13 Effectiveness: MGD-SFP reached out to 455,024 individuals directly and another 252,894 individuals indirectly. This included students, parents, farmers, and other community members. 14 The evidence indicates that the program has contributed towards improving dietary practices, creating awareness among parents about the importance of education, and improving the learning environment of children at school and at home. Cumulatively, these have contributed towards achieving the objectives of improved literacy of school-age children (Strategic Objective 1) and increased use of health and dietary practices (Strategic Objective 2). 15 There has been considerable improvement with respect to children’s reading and comprehension abilities. The proportion of Grade 2 students who showed at least 75 percent comprehension of the Lao language has increased to 6.9 percent from 1.9 percent. An interesting finding is that the comprehension ability of Lao language was considerably higher among Grade 3 students (26% demonstrated at least 75% comprehension) as compared to Grade 2 students. Overall, it was found that the level of comprehension is better among boys than girls. 16 No significant improvement was observed in the attendance and enrolment rates of children over the program period. The baseline reported considerably high rates for these parameters thereby leaving little scope for improvement. It was observed that by providing support to children for learning and improving their learning environment, the program has contributed to improving the proportion of students who complete primary schooling from 60 to 80 percent. 17 There has been significant improvement in children’s home environment for pursuing studies. A 40 percent increase in the proportion of students taking extra classes beyond school hours was reported. Further, a 35 percent increase in students having someone at home to help them with their studies and around 47 percent increase in the proportion of students reading extracurricular books were USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report iii reported. Also, a considerably higher proportion of parents during the end-line (96%) could name at least three benefits of primary education as compared to the baseline (45%). 18 A 10 percent improvement in the proportion of teachers regularly attending school for at least 90 percent of the school days was reported. The proportion of staff trained in teaching techniques doubled over the programme period from 23 percent during the baseline to 48 percent presently. 19 Significant improvement could be seen with respect to the nutrition intake among children and diversification of their food basket. The mean Dietary Diversity Score (DDS) increased from a low of 5during the baseline to 7.6 on a scale of 1-10. The consumption of legumes and beans, nuts and seeds, dairy products, and eggs have increased significantly as compared to the baseline. 20 Improving knowledge on health and nutrition among students, parents, and other community members who are part of the programme requires attention during the next award cycle. In the case of WASH, availability of water needs to be addressed for promoting the adoption of hygienic practices and bring about a sustained behavioural change. 21 School gardens established under the project have contributed to the provisioning of school lunch and improving knowledge on farming techniques among children. According to monitoring reports, quite a high number of school gardens have been set up/rehabilitated through MGD assistance (1406 against a target of 768; Annex R). However, these school gardens also ought to be used to impart nutrition education to students and other community members. Water scarcity and dependence on seasonal water sources for irrigation are affecting the usefulness of these gardens. 22 The Lao Women’s Union provided voluntary services for cooking school lunch. They were also responsible for collecting firewood, vegetables, and fetching water which considerably increased their workload. They felt that the incentives provided are not commensurate with their efforts. Given that cooks play a very important role in the provisioning of school meals, it is imperative that they are adequately incentivised. The community’s contribution of vegetables to the school is affected during the winter months; this contribution is also contingent on the family’s economic status and availability of a surplus. The lack of water in several schools is also affecting the cooking of school lunch. 23 Overall, capacity building of cooks, storekeepers and teachers emerged as a pressing need for effective implementation of the programme. While training sessions on safe food preparation, storage practices, and warehouse management have been conducted, there still remains a sizeable proportion of cooks, storekeepers and teachers who have not yet been trained. 24 Access to improved sanitation facilities, improved water source, and handwashing facilities is critical for improving the health and hygiene conditions which needs to be strengthened across most schools. Handwashing stations were observed in very few schools and less than one-third of the schools had access to improved water source. On the other hand, more than two-third of the sample schools had access to improved sanitation facilities. 25 As may be concluded from above, the availability of water is a critical factor for sustaining the programme as it is closely linked to school gardens, cooking, health, and hygiene. Currently, water shortage is a serious concern in the Northern provinces. 26 Although parents have contributed in the terms of labour for construction of school infrastructure, providing vegetables and cash, community ownership towards provisioning of school lunch needs to be further strengthened. For this, capacity building of the community, VEDC members, teachers, and other support staff on health and nutrition, programme management, record keeping, improved farming techniques, commodity management, safe food preparation, and storage is critical. The role of VEDC needs to be strengthened further in order to enhance community participation. 27 The field-level evidence indicated that the robustness of monitoring mechanisms under MGD-SFP needs strengthening. It was found that schools were not maintaining records of monthly food utilisation, the number of meals served, and the number of students receiving the meals despite being trained and provided with registers. The formats for reporting against performance indicators of the programme also need to be simplified for ease of comprehension and reporting by the field staff. 28 Close coordination between WFP-CO, government departments (MoES, MAF, MPI), bilateral and multilateral organisations (DFAT, BEQUAL, The World Bank, UNICEF), NGOs (PI, BBM, CRS), and local government recognized groups (VEDC, LWU) contributed to the programme results. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report iv 29 Recommendations from FY14 baseline study and FY14 mid-term evaluation have been successfully integrated into the programme to a large extent over the programme period (Annex T). This includes strengthening partnership with MoES, strengthening community contribution through capacity building, improving access to WASH facilities in the schools, improving capacities of stakeholders, improving knowledge on nutrition, good health and hygienic practices, and improving mechanisms for monitoring and record keeping. Based on the evaluation team’s qualitative discussions with the stakeholders and findings from the primary survey, additional areas of improvement for strengthening the programme have been included as part of the recommendations. 30 Impact: For villages where a higher number of families support the SFP, the cost to each family was found to be lower as compared to villages with fewer families supporting it. This can be used as an example to incentivise communities to contribute to the programme. Some parents (mostly mothers) also felt that the SFP enables them to spend longer hours at work which improves their economic productivity. Moreover, they also have additional time for leisure, recreation and other activities, as they no longer are required to cook lunch for their children. 31 The project’s focus on nutrition awareness also led to positive changes in dietary practices at the household level. Discussions with stakeholders indicated that while these changes are in the infancy, it has the potential to benefit the community in terms of improved nutrition. 32 With respect to gender, the programme has been able to achieve improvement in education and nutrition outcomes for girls and boys alike. All activities of the programme were implemented to suit the needs of male as well as female students and both sexes have been equally focussed upon. 33 Efficiency: The programme was efficient in terms of designing and implementing activities that conformed to the educational and nutritional needs of the community. By targeting schools from the furthest behind districts of the country, the programme created a platform for the physical and mental development of the children. Through collaborations with other donors and NGOs, WFP has ensured the availability of technical expertise in implementing the complementary and supplementary activities under the programme. The annual per capita expenditure per student is estimated to be US$ 16.8 which is almost the same as that allocated under the National School Meals Programme (NSMP). 34 Sustainability: MGD-SFP has created an effective platform for gender equality, creating knowledge on health and nutrition, and advocating for the value of education. 35 The sustainability of the programme depends upon GoL’s ability to finance, maintain and operate it. Critical also are community partners’ support towards sustaining the benefits of the programme. Summary Conclusions 36 Overall assessment: The evaluation team’ overall assessment is that the project was positive in terms of design and implementation. The targeting of the most deserving group under the programme and aligning with the government’s policies and strategies reflects a strong understanding of the country context. The programme has been effective in improving the reading ability among children as well as their dietary practices. Engaging with teachers and parents has helped in improving the learning environment for children. The programme has the potential to generate economic incentives for parents by means of reduced expenditure on food. While educational and nutritional outcomes have improved for boys and girls alike, the programme’s impact on broader issues of gender equality has been limited and evident mostly with respect to parents and VEDCs, and attributable to advocacy/sensitisation. 37 There are a few areas that deserve WFP’s attention to strengthen the FY17 programme. Most important among these is the monitoring system of the programme. Other areas of concern are knowledge about nutrition education which is currently weak and needs to be strengthened. Also, access to water is critical for WASH, functioning of school gardens, and cooking of school lunch. 38 Good practices: Building community ownership through participation in programme activities along with communication on the importance of education and nutrition are the highlights of the initiative. This has been complemented by aligning the program with the government’s policy on education and NSMP, thus ensuring greater institutionalisation and thereby the sustainability. The participatory approach to development and collaborations with multiple actors are good practices that offer valuable lessons for the FY17 programme. 39 The areas that deserve WFP’s attention have been presented below: USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report v Sl. No. Recommendations Proposed actions 1 Children should not be leaving school during lunch break Similar to the policy on retaining children during lunch time in NSMP supported schools, a government order needs to be issued directing schools to keep children back in schools after consuming their meals. Currently the programme mandates provisioning of school lunch after which children still have another hour of break when both teachers and children return home and the classrooms are locked. To prevent this, WFP in collaboration with BEQUAL, PI, and BBM may design extra-curricular activities to ensure that teachers and students stay back in school. 2 Improving access to water in schools Under SFP-FY17 award, WFP will support UNICEF to install 25 FGS and 25 boreholes and improve access to water. This however will fall short of the requirements for all the schools. For schools which are drawing water from the surface, pipelines with attached pump-sets should be provided for supplying water to the school garden, kitchen, and WASH facilities. WFP with support from donors can install rainwater harvesting units to compensate for the groundwater drawn for school gardens. 3 Training of a group of cooks in each school By adopting a tree-structure approach, master trainers from each village should be trained in cooking. They in turn should train multiple groups in their villages. The VEDC should be responsible for maintaining a roster for cooking and informing the groups. In addition, groups will also remind the group responsible for cooking meals the next day. The cooks ought to be given adequate incentives for their voluntary services. 4 Strengthening gender dimensions of the project- cooking Both girls and boys should be encouraged to volunteer for the SFP activities such as cooking, gardening, fetching water, and collecting firewood. This will have three immediate benefits: breaking down the stereotyping of gender roles, lowering the burden on the cooks, and bring in values of shared responsibility. The Lao Youth Union can be engaged along with LWU for leading this initiative. 5 Strengthening gender dimensions of the project- managing commodities It is important that warehouses are safely accessible. The technical specifications for constructing them needs to be circulated among the schools. After overcoming the infrastructural barrier, schools should encourage equal participation from both men and women for handling operational activities. 6 Convergence between the horizontal and vertical structures of the government A joint workshop of all the three ministries should be conducted by WFP and the way forward should be presented by MoES for successful transition from SFP to NSMP. The expectations of each ministry should be clearly articulated. The ministries in turn should conduct workshops for their departments in the provinces and districts on the expected roles and responsibilities. The third step would be workshops at the provincial and district levels where officials from these departments should work out the course of action. A successful transition to NSMP is also dependent on the availability of funds for the programme which needs to be ensured. 7 Harmonisation of MAF’s technical support to school gardens and crop diversification for farmers MAF is providing technical inputs to ‘School Garden Training Guidelines’ and is expected to conduct district and school level training on school gardens and greenhouses. DAFO officers are expected to visit communities on a regular basis. Additionally, WFP and MoES should collaborate with MAF and identify schools which are keen to adopt the good practices. In such schools, pilot agriculture projects should be initiated. This will highlight the importance of school gardens and enable cross learning between the stakeholders. 8 Capacity building for all stakeholders to deepen their understanding around community led school feeding programme. For a successful handover of the SFP, it will be essential that all stakeholders are aligned with the programme objectives and have clarity on their roles and responsibilities. There is a need to train all the stakeholders as part of the handover process: 1) Officials from MoES, PESS and DESB who will be responsible for school lunch should be trained together for at least two sessions for clarity of roles and responsibilities, 2) DESB members should then train VEDC members, school teachers, cooks and storekeepers on implementing the programme effectively, and 3) DESB members with support from VEDC should train the community. In addition, teachers, cooks and communities should be trained repeatedly on issues like hygiene, education, health, and nutrition. 9 Strengthening of M&E systems Maintaining school records on enrolment, attendance, dropout, and repetitions should be made mandatory and feed into government data systems. Currently, these records are rarely maintained. A template to capture teachers’ attendance and details of school lunch served should be provided to school who in turn ought to submit the data to MoES on a monthly basis. This information should be collected by DESB. PESS should compile all DESB data and submit it to MoES. It also needs to be ensured that the data being submitted by schools is validated by the community/ LWU/ or Lao Front for National Development. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 1 1. Introduction 1 The end-line evaluation for United States Department of Agriculture (USDA) Mc-Govern Dole Food for Education Grant (MGD) supported School Feeding Programme (SFP) FY14-16 award cycle (hereafter referred to as FY14) in Lao PDR was commissioned by WFP country office Lao PDR (WFP CO) between February-June 2018. The evaluation covers the period from September 2015 (when the food arrived) till March 2018. The objective of the evaluation was to assess the implementation and performance of the FY14 award and generate recommendations that will strengthen and inform the operational and strategic decision-making for the FY17 award rollout. The evaluation served the dual purpose of accountability and learning. It assessed the performance and results of the implementation and determined the causalities towards achieving or missing the results. The evaluation has provided an evidence-based independent assessment of the performance of the operation that would enable WFP and program partners to take informed operational and strategic decisions for the FY17 award and for designing similar SFPs. 2 The scope of the Evaluation (details in Annex A): It covers the MGD-SFP FY14 award, including all activities and processes related to its formulation, implementation, resourcing, monitoring, evaluation, and reporting that were relevant to answer the evaluation questions. The evaluation has provided an evidence-based, independent assessment of the performance of the operation. 3 Primary Users of the Report(details in Annex B): The primary stakeholders and users of this evaluation are a) WFP (WFP CO, Regional Bureau Bangkok, WFP-HQ, and Office of Evaluation), b) USDA, c) Ministry of Education and Sports (MoES), Government of Lao(GoL), f) Partners such as Big Brother Mouse (BBM), Plan International (PI), SNV, Australian Department of Foreign Affairs and Trade (DFAT), UNICEF, and The World Bank are interested in the results of the evaluation. 1.1. The subject of the Evaluation 4 In line with the ToR, the subject and scope of the end-line evaluation were the WFP Lao School Feeding USDA McGovern-Dole Grant FFE-439- 2014/049-00. The two strategic objectives of MGD-SFP FY14were (a)to improve the literacy of school-age children (SO1), and (b) increase the use of health and dietary practices (SO2). Central to these objectives, were four foundational results that contributed towards attaining sustainability of SFP. These were linked to the strengthening of the external ecosystem vis-à-vis policy framework and capacities of the government. Further, they also related to the increased engagement of the local organisations and community groups, thereby building community’s ownership of the programme. Thus, it can be understood that MGD-SFP FY14 has a very broad encompassing approach, including food, infrastructure, training, and education, making it a very holistic intervention. 5 Programme Geography (details in Annex C & D): The MGD-SFP was implemented in seven provinces: Attapeu, Saravane, Sekong (south), and LuangNamtha, LuangPrabang, Oudomxay, Phongsaly (north). It covered around 145,000 primary and pre-primary students (of which half were girls) in 1,446 schools across 30 select districts characterized by poverty, malnutrition and low literacy rates (Annex J). The programme provided lunch for school children along with complementary resources (school gardens, training of teachers and communities, provision of books and educational material), and built capacity towards a gradual handover to the Government by 2021. 6 Programme Timeline: In 2014, USDA issued the MGD commitment letter allocating US$27 million for donations of food, transportation, and financial assistance for a three-year period (2014-2016) to the WFP Country Programme (CP) for Lao PDR. The implementation commenced in September 2015 with the arrival of the first tranche of food commodities. In October 2016, USDA approved an amendment to the original grant from Mid-Morning Snack (MMS) to school lunch, including a shift from the provision of Corn Soya Blend (CSB) to rice and lentils for the lunch programme. 7 Amendments to the original design: GoL adopted the National School Meals Policy (NSMP) in 2014 focussing on provisions for school lunch. To align WFP’s activities to the national policy, WFP in a gradual manner between 2015-2017shifted from MMS to school lunch. Since, rice was the staple food and accounted for 77 percent of the daily energy needs, WFP introduced it as part of school lunch. Further, to ensure dietary diversity, lentils was also introduced and thus CSB was replaced with rice and lentils. For schools with MMS, WFP also provided Take Home Rations (THR) consisting of 40kg of USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 2 rice twice a year to Informal Boarders (IB)8. Due to an overlap of support from ADB, GoL had requested WFP to stop THR to IBs. In late 2015 this support was suspended. Further, with implementation of school lunch, WFP was cognisant of the support activities that would be required by the schools for effective implementation of MGD-SFP. Hence, it promoted construction of dining room facilities, encouraged activities around hygiene, and supported improved access to water in schools. 8 Due to unforeseen reasons, WFP did not receive the desired support from the identified partner for implementing the literacy program. Hence, the original approach of establishing a literacy baseline followed by a literacy campaign was amended. Underpinned by the MGD-SFP baseline findings, WFP partnered with PI and BBM for improving the literacy and access to reading material. Also, with MoES’ focus shifting from enrolment (which was 97%) to increased retention and lowering of dropout, WFP also reduced funds for enrolment activities and provisioned more for the literacy activities. All these underline WFP’s commitment to realign its interventions with the GoL’s priorities. 9 Planned Outputs and Beneficiaries: Consisting of 100g of rice, 10g of oil and 40g of lentils, the school lunch was cooked and provided to every student within school premises. At the design stage, WFP planned to provide school meals (MMS and school lunch) to a total of 405,758 school children (205,719 boys and 200,039 girls). Initially, the commodity requirements were 6,000mt of CSB, 5,500mt of rice, and 1,140mt of vegetable oil9,10,11. The planned beneficiaries and revised targets of the intervention disaggregated by sex are presented in Table 112. Table 1: Target beneficiaries disaggregated by sex Gender 2015 2016 2017 Revised target for 2017 Boys 76,355 72,097 57,267 53,645 Girls 74,247 70,107 55,658 52,527 Total 150,602 142,204 112,952 106,172 Source: USDA Third Year Operational Plan for Agreement #FFE-439-2014/049-00-C Laos, 2016 10 The THRs for informal boarder was suspended in late 2015 and assistance to 63 poor performing schools was stopped. This led to a revision of the beneficiary numbers for 2017, bringing the total number of children assisted to 398,978. From September 2017, WFP added lentils into the programme. The commodity requirements in the third year of implementation changed to 1060mt of lentils, 3900mt of rice and 230mt of vegetable oil. No cash or vouchers were provided under the programme except under the pilot project in Beng and Nalae districts. Further, 13,600 cooks and storekeepers were also entitled to THRs under the programme13. 11 MGD-SFP also targeted at building/ rehabilitating 1700 kitchens, 1700 storerooms/ warehouses and establishing 550 school gardens. Distribution of 5000 posters on the importance of education, 50 enrolment campaigns, distribution of 500,000 books and supplementary materials, training of 14,000 cooks and storekeepers on food preparation and storage practices, training of 482 provincial and district school meals committee members on good health and nutrition practices were the other targets envisaged for the MGD-SFP award. 12 Planned Outcomes: Details of the planned outcomes have been provided in Annex E. This has been drawn from the USDA approval of an amendment to FY14-16 wherein WFP Laos had sought to update several activities and partners and provided an update to indicators. 13 Programme financing: USDA had allocated $27 million to MGD-SFP. Additional support was provided by Australia, Cuba, Japan Association for the World Food Programme and Yum! Brands Inc. As per the figures in the MTE report, contributions received for the overall MGD-SFP as of May 2016 was US$ 45,958,344, which is 100 percent against the total requirements. This evaluation covers the USDA’s funding for the programme. A break-up of the contributions is provided in Annex F. 8Informal Boarders were students who did not have access to schooling in their home villages and were therefore living on their own close to the school premises in small huts 9Mid-Term Evaluation of McGovern-Dole School Feeding in Laos 2015–2016 Evaluation Report 10 USDA Commitment letter for agreement #FFE-439-2014/049-00 Laos, 2014 11USDA Third Year Operational Plan for Agreement #FFE-439-2014/049-00-C Laos, 2016 12According to LaoEduInfo database 2017, the total number of children enrolled in primary school in the country was 808,705. Of this, boys are higher in number (419,379) as compared to the number of girls (389, 326). Correspondingly, the programme reached out to a higher number of boys as compared to girls. 13FY2014 MGD Proposal USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 3 14 Logical Framework: The design of the programme can be understood via the updated MGD project level results framework14 (Annex G). A key aspect of the MGD-SFP was the in-kind contribution (vegetables, salt, sugar, firewood, and labour) by parents, the effectiveness of which was dependent on the mobilisation of the community by the VEDC. Other important aspects that emanated from the design of the programme and affected its effectiveness included the suitability of food commodities provided by USDA and the Government’s preferred mode of procurement at the local level15. Certain other assumptions underlying the achievements of results of the MGD-SFP such as the Government’s commitment to funding the SFP, convergence of different Government departments (education, health and nutrition, and agriculture) for future implementation of the programme, willingness and capacities of the community, particularly the VEDC, to perform the roles envisaged by the programme, were examined and assessed to arrive at recommendations. The evaluation also examined the Theory of Change (ToC), developed during the MTE (Annex H). 15 Partners: WFP had established strategic partnerships and leveraged support from government institutions, UN agencies, other donors, and NGOs towards achieving the MGD goals. For implementing the FY 14 programme, WFP closely coordinated with MoES, MPI, MAF, UNICEF, World Bank, WHO, DFAT, BBM, PI, EDF, and CRS. This approach intended to provide school lunch, impart training, improve school infrastructure, increase access to reading and instruction materials, and develop school curriculum. The detailed roles of different partners are presented in Annex I. 16 Gender Dimensions of the Intervention: Stunting rates are four times higher among children of uneducated women than among children of educated mothers16. This perhaps stems from lower awareness and understanding about nutrition. In this context, WFP promoted knowledge around nutrition and the importance of education among all, particularly caregivers, and encouraged them to participate in programme activities. The WFP School Feeding Policy (2013) and Gender Policy for 2015-2020 emphasised recognizing and including specific needs of young girls, ethnic and religious minorities, and children with special needs. WFP advocated the importance of education (particularly attendance) at the community level through the distribution of posters that emphasised on the importance of regular attendance for boys and girls. 17 Although no gender analysis was available for the FY14 programme during the baseline, WFP-CO had integrated gender components in its implementation through the MGD-SFP as stated in the Mid-Term Evaluation (MTE) report, 2017. This included measures like sharing of tasks between boys and girls without defining tasks by gender, strengthening of nutrition education, and mainstreaming of gender into field-level activities by using new literacy materials to challenge unsuitable gender roles. 1.2. Context 18 Indicators and Trend Data: Lao People’s Democratic Republic (Lao PDR) with a population of 6.9 million people is inhabited by 47 officially recognised ethnic groups. The economic growth of the country has aided in improving the HDI from 0.4 to 0.56 during 1990-2015. However, Lao PDR still ranks 141 out of 188 countries (2015) and is a least developed country (LDC)17. 19 Education: There has been a significant improvement in the status of children’s education in the recent years. The net enrolment rate (NER) for primary school has increased from 65 percent in 1990 to 98 percent in 201418. However, it is not uniform across the country and inter-province differences exists as per the 2015 census report (Annex J). Gender parity is observed in enrolment with 98 percent of girls of age 6-10 years being enrolled in primary school (2014)19. The mean years of schooling have increased to 5.6 for boys and to 4.5 for girls (2015)20. Primary school dropout rate has decreased from 7.65 percent in 201021 to 5.40 percent in 2014 (girls-4.90; boys-5.90)22. 14USDA Third Year Operational Plan for Agreement #FFE-439-2014/049-00-C Laos, 2016 15The local staple diet is glutinous/ sticky rice whereas the SFP provides long-grain rice to the schools. MTE also points out WFPs objections to both the lack of nutrition and the higher cost of sticky rice. 16 Save the Children, 2017. Stolen childhoods: End of childhood report. 17http://hdr.undp.org/en/countries/profiles/LAO# 18World Bank, 2016. Reducing Early Grade Drop Out and Low Learning Achievement in Lao PDR. 19LAO Eduinfo database 20http://hdr.undp.org/en/countries/profiles/LAO# 21 Lao-PDR: Root Causes of Early School Leaving; World Bank 22LAO Eduinfo database USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 4 20 Food and Nutrition: Lao PDR with a score of 27.5 falls in the “serious” category as per 2017 Global Hunger Index. Overall, about 14 percent of the rural population in the country is food-insecure. As per the Global Hunger Index 2015, about 44 percent of children under the age of five23 are malnourished. In 2015, 35.6 percent of children under five years of age were stunted (37.6% boys and 33.6% girls), 25.5 percent were underweight and 9.6 percent were wasted. The inter-district disparity in food and nutrition security is presented in Annex J. 21 Health: Under-5 mortality rate (U5MR) was 6724 in 201525. Although there had been a 59 percent decline in U5MR from 1990, it fell short of the Millennium Development Goal (MDG) target for child mortality of 54. According to the HDR 2016, poor nutrition causes 45 percent of the deaths among children under the age of five years26 and also leads to stunting and delays in physical development. 22 Water, Sanitation, and Hygiene: Lack of good hygiene practices is a major reason for health-related absenteeism among students in the country. Unhygienic practices also lead to high levels of STH and thus increasing the need for deworming. The problem is further accentuated by the lack of WASH facilities as only half of the schools in Lao have access to appropriate WASH facilities27. 23 Government Strategy, Policies and Programmes: Government of Laos (GoL) aims to move from LDC status to that of a Middle-Income country by 2020. GoL through the 8th National Socio-Economic Development Plan (NSEDP) 2016-2020 and other policy instruments is striving towards sustainable economic growth and equitable social development28. The NSEDP includes sectoral plans of various departments including School Meals Action Plan (SMAP) 2016-2020. Complimenting this plan is the Agriculture Development Strategy (2025), through which GoL intends to combat malnutrition by promoting dietary diversity. 24 Towards, achieving universal access to primary education, GoL has made it free and compulsory. The Education Sector Development Plan (ESDP, 2016-2020) stresses the need for expanding and maintaining school feeding programs to encourage disadvantaged children (ethnic communities, children with disabilities, those in remote and impoverished circumstances) in lower primary grades to remain in school29. The National Nutrition Strategy (2016-2025) and Plan of Action (2016-2020) (NNSPA) aims at promoting equality in gender roles, placing emphasis on women’s access to health services, to nutrition and food security information, and to food. In 2012, GoL launched the National School Meals Programme (NSMP)30 with support from Global Partnership for Education (GPE). Under GPE-II (2015-2019), The World Bank has allocated US$ 28 million for five-year ECE project, of which US$ 8.5 million will be utilised towards school lunch. 25 Gender Dimensions: The Gender Development Index (GDI) reflects gender inequalities in achievement in the three dimensions of the HDI: health (measured by female and male life expectancy at birth), education (measured by female and male expected years of schooling for children and mean years for adults aged 25 years and older); and command over economic resources (measured by female and male estimated GNI per capita). As per the GDI categorization, with a value of 0.924, Lao PDR fell under group 4, which indicates medium-low equality in HDI achievements. 26 In terms of education, there exist gender-based differences in years of schooling and overall education level. According to HDR 2016, the number of expected years of schooling for an average female is 10.4 years while that for male is 11.2 years. Several studies suggest that the dropout rate for girls after completion of primary school is higher than that of boys. One of the causes is the higher burden of household work shared by the girls (HDR 2016). 27 Development Assistance in Lao: Official Development Assistance (ODA) is the official contribution of donor government agencies for supporting development and welfare in underdeveloped and developing countries. Economic support received under ODA helps in achieving the development 23 Nutrition Country Profile (2015), Lao PDR International Food Policy Research Institute. (Source: UNICEF, WHO, WB 2015) 24http://www.la.one.un.org/sdgs/sdg-3-health accessed on 16 April 2018 25 U5MR for 2016 is slightly lower at 66.7 (HDR 2016) 26UNDP (2016). Human development report 2016: Human development for everyone. USA: UNDP 27WFP FY17 MGD Proposal 28 Lao PDR, National Plan of Action 2003-2015 29http://moe.gov.la/data/publications/ESDF%20English%20version.pdf 30 The GoL’s NSMP currently operates in five provinces and is funded by GPE, a World Bank managed trust fund. None of GoL’s NSMP covered schools are covered by WFP with entirely separate district coverage. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 5 targets envisaged by the governments. Development Partners have supported Lao PDR with almost US$ 1,970 million from 2011 to 201531. Of this, international financing institution provided US$ 577 million, bilateral donors from Asia Pacific regions provided US$ 566 million, and the European Union provided US$ 363 million. Infrastructure, health, education, natural resource management and Environment, and agriculture and rural development were the top five sectors that had received support from ODA. The details are presented in Annex K. 28 For the school lunch programme, WFP is one of the three main providers of school meals in Laos covering 1446 schools across 30 districts in seven provinces. GoL provided lunch in 312 schools in 10 districts across five provinces and CRS supported 40,000 children in 360 schools across seven districts. In addition, the Education Development Fund (EDF) provided meals to four schools and Humana People to People supports two schools32. 29 WFP has been working with PI and BBM emphasising on the inclusion of disadvantaged children, increased engagement with parents, and equipping teachers with improved instruction materials. Targeting the 65 educationally disadvantaged districts, the BEQUAL program, funded by Australia and the EU, was the largest single donor funded education program in the country. WFP has been a member of the Education Sector Working Group (ESWG) chaired by the Minister of Education and co￾chaired by Australia and EU. The subgroups are required to ensure a coordinated approach between all stakeholders towards achieving inclusive education. WFP co-chairs the School Meals technical working group which is a subgroup under the focal group 1 of the ESWG. An integrated nutrition and WASH project (NURTURE) is being supported by USAID and implemented by Save the Children in collaboration with SNV Netherlands. Implementation Constraints 30 Institutional and financial capacity: Despite the strong policy framework established by GoL for attaining the SDGs, the Systems Approach for Better Education Results (SABER) results of 2016 raises issues towards Government’s financial and institutional capacity towards sustaining the SFP. The absence of specific budget line for school meals puts the entire onus of running the SFP on communities. Limited capacities, weaknesses in government structures and competing demands of districts levels officials highlight the lack of coordination and similar vision of school meals at district level. Further, the national strategic review of food and nutrition security (2015/2016) clearly highlighted the weaknesses in the government structures for addressing the complex and cross-sector issues of food and nutrition security. The report also emphasised the need for building capacities of government officials to translate strategies and plans into practice. Insufficient budget, limited social protection and safety nets, persistent cultural taboos in combination with poor nutrition knowledge were the other challenges highlighted in the report. 31 Cultural diversity: The low level of understanding of spoken and written Lao language among the non-Lao speaking community is one of the factors for the slow realisation of the SFP outcomes. Language and local cultural norms influence the behaviours and thus it is necessary that the interventions are contextualised to the local settings for increased outreach and acceptability among the various ethnic groups. 32 Natural disasters: The northern province of the country has experienced earthquakes and landslides that led to crop losses. Depending on the establishment of green houses as an addition to school gardens will determine the success of SFP in the schools of Northern provinces. 1.3. Evaluation Methodology 33 The evaluation was conducted during Feb-Aug 2018 (Mission schedule presented in Annex L). The tenure under evaluation was from Sep 2015-Mar 2018. The data was collected during Mar-May 2018 due to which information on certain data points such as attendance, attentiveness, and food utilisation catered to an academic year from Sep-Mar 2018. The evaluation team undertook a five-day scoping mission to Lao during 12-16 Feb 2018 for a better understanding of the program and finalise the evaluation approach and methodology in consultation with the WFP-CO. 31 Foreign Aid Implementation Report (FAIR) Fiscal Year 2015/16, Lao PDR 32WFP FY17 MGD Proposal USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 6 34 The evaluation complied with the ToR in the application of the international evaluation criteria of relevance, effectiveness, efficiency, impact, and sustainability with gender equity and inclusion mainstreamed throughout (Annex M). Wherever appropriate, gender dimensions were factored into the sub-questions, judgement criteria, and indicators for each evaluation question. 35 Under the criteria of relevance, the evaluation assessed the alignment of MGD-SFP design with GoL’s priorities and WFP's Country Strategic Plan (CSP) 2014-16. In addition, the evaluation examined whether the design and implementation of the programme were relevant to the local needs of the most vulnerable groups and whether it aligned with gender policy of the GoL and that of WFP. 36 The effectiveness of the programme was assessed by comparing the end-line values for literacy (MGD 1.1), attendance (MGD 1.3), attentiveness (MGD 1.2), and health-related absence (MGD 1.2 and 2.4) with the baseline values. Further, the effect of provisioning school meals, supporting school infrastructure, improving access to quality learning materials in improving retention and reducing dropouts were examined. Also, the role of school gardens as perceived by the beneficiaries and school authorities were studied to understand their importance as a learning tool on agriculture and nutrition and the adequateness of the current resources to sustain the gardens. 37 To measure the sustainability of the programme, GoL’s willingness and capacity to integrate MGD-SFP with NSMP was examined. The processes set-up by WFP to strengthen capacities at all levels and the design of the exit plan including the plan delineating responsibilities of the stakeholders were assessed. The rationale of the recommendations made by MTE and the extent to which these recommendations have been integrated into the programme were also assessed. 38 The end-line evaluation was a non-experimental, cross-section study which followed a pre-and-post study design given the absence of counterfactuals in the 2014 baseline study. Since the evaluation was unable to attribute changes directly to the programme, it looked at the programme’s influence by comparing the observed primary results with evidence from past evaluations and studies. 39 The study methodology developed for the evaluation was guided by the ToR and underpinned by the results framework of the FY14 programme. The evaluation adopted an approach (detailed in Annex N) that included key thematic and functional areas that the interventions would have worked upon. It analysed cross cutting factors of gender and WASH that influenced the actors and structures in the programme and determined the extent of the effectiveness of the programme. 40 Drawing upon the risks and assumptions of the ToC developed during the MTE of the programme, the methodology examined the causality implied in programme design and, by identifying the validity of the underlying assumptions in the context of the programme, determined key factors that explained the extent to which the programme has achieved its objectives. 41 The methodology entailed desk review (documents listed in Annex O) and primary survey. Data from the primary survey was triangulated to assess its reliability and validity. Triangulation was mainly undertaken within methods by comparison of perspectives of different stakeholders on the same issue. In order to mainstream GEEW throughout the analysis, all school-level data have been disaggregated by sex, and sampling of students was done in a way so as to ensure equal representation of boys and girls. 42 A mixed method approach was employed to answer the questions specified in the evaluation criteria using structured interviews for quantitative data and semi-structured interviews and discussions for qualitative data. While qualitative interviews focused on assessing the factors affecting literacy, attendance, and attentiveness in school, quantitative assessment analysed the extent of the changes. The findings were disaggregated based on ethnicity and gender to understand the results of school feeding on different groups. A summary of the methodology is presented in Table 2. 43 The evaluation sample was spread across seven programme provinces covering nine districts. 60 schools were sampled for the evaluation of which 45 are common with the end-line evaluation FY14. The schools were selected using simple random sampling. The sample distribution is detailed in Annex N and the list of sample schools is in Annex P. Overall, 414 boys, 452 girls, 583 men33 and 443 women34 were interviewed for the evaluation. 33Includes school heads, teachers, cooks, storekeepers, VEDC members and parents 34Includes school heads, teachers, cooks, storekeepers, VEDC members and parents USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 7 44 Integration of Gender into the Methodology: As indicated in the evaluation matrix, gender dimensions have been integrated in its design. The evaluation questions (Annex M) 2, 3, 4, and 5 under the criteria of relevance aimed to understand a) whether the programme aligned with the priority of women, men, girls, and boys, b) whether the programme was based on a sound gender analysis, c) whether the design and implementation were gender sensitive, and d) whether the programme had made any impact on gender relations. Evaluation questions 7, 8, and 9 under the criteria of effectiveness provided sex-disaggregated data on the attainment of programme outcomes related to literacy, attendance, and attentiveness. Question 19 specifically responds to influence of the programme on gender and the gender-specific impacts. 45 The evaluation examined the role and nature of participation of men and women in the VEDC. It also captured the perspectives of men, women, boys, and girls. The gender-balanced data collection team was deployed after adequate training to ensure that the views of all key groups are considered, reflected, and triangulated with due attention to issues focussing on gender. For quantitative interviews, parents (men and women), boys and girls were interviewed separately. FGD participants included both men and women in equal numbers, to the extent possible, and questions to assess their views on gender issues were included in the checklist. Further, the qualitative data collection team included male and female moderators to enable participation of women in the discussion. Table 2: Summary of the evaluation methodology Methods of data collection Primary Survey Secondary Review Quantitative (Structured interviews) Qualitative (Focus group discussions, key informant interviews) Target Group/ Source of Information  Children  Parents  School head  School teachers  School cook  School storekeeper  School observation  Children  Parents  School head  School teachers  School cook  School storekeeper  School observation  Network Partners  Government officials of MAF, MoES, and MoH  VEDC and farmer groups  A literature review of available public sources  Project documents and results framework  Standard project reports and semi-annual reports  Community strength assessments  Baseline and mid-term reports  Impact review on school meals  Monitoring data Indicative outputs of data collection  Established end-line values of key indicators  Assessed change from baseline  Examined the effectiveness of the programme  Change in enrolment and attentiveness (from direct and indirect influence)  Change in availability and access to food supplies and other school materials Appropriate gender disaggregated analysis provided  Children and parents: Quantity and quality of receipt and consumption of school lunch, dietary diversity, regularity to school, change in attentiveness, reasons for change (disaggregated by sex), economic incentives  School: observed changes, infrastructural support, school gardens, gender considerations in implementation, perceived risks Others: Implementation modalities, changes and reasons, policy perspectives, province and district level initiatives, resources and partnerships, innovation and learning  Understand context and relevance of the programme  Established alignment of the programme with government initiatives and WFP’s country strategy  Resources utilised  Key design elements that contributed to direct and indirect change  Pathways created for sustainability; scope of replication; key considerations for recommendations 46 Data Analysis: The questionnaires were developed in line with the baseline to ensure comparability of datasets. The baseline data has been used for reference in the analysis phase, and information on implementation indicators were taken from WFP monitoring reports. The indicators specified in the USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 8 ToR were mapped with the tools, respondent groups, and questions which facilitated the identification of methods for data triangulation and assessed the reliability and validity of the data collected. 47 The quantitative analysis included applying sex-disaggregated data to compare the baseline with the end-line to highlight the change in the intermediate and immediate outcomes. The evaluation also used the findings from the MTE and the Impact Assessment Review to supplement the results of analysis. The qualitative data focused on identifying the reasons for the observed change. Data on attainment of outputs and outcomes were disaggregated by sex and age. An assessment of qualitative data on reasons that led to drop-out or affected the attendance of boys and girls in schools was conducted. Variables such as ethnicity and distance from the school were also used to contextualise the variations in evaluation findings for both boys and girls. Limitations and Risks 48 The end-line evaluation was a non-experimental, cross-section study which followed a pre-and-post study design given the absence of counterfactuals in the baseline conducted in 2014. The evaluation, therefore, was not able to attribute any changes to the programme. However, the evaluation sought to understand how the programme has influenced the observed results highlighted in the evaluation questions using evidence from past evaluations and primary data from the current evaluation. 49 Discussions with the WFP CO and the observations from the scoping visit had highlighted the unavailability of past year student attendance data in several schools. To address this issue, the questionnaire included measuring the attendance by counting the total number of boys and girls present in each class in the school on the day of the survey. Additionally, student self-reported their absence from school in the past week. 50 The method of data collection to capture attentiveness also raises questions on the reliability of data. Attentiveness of a student is a function of different variables one of which is short-term hunger. Others may include poor teaching methods, lack of interest in the subject matter being taught in class, and incidents at home. The evaluation measured the attentiveness of children based on the opinion of their school teachers which may be biased based on the teacher’s perception of the child. 51 FY14 Baseline used the Learning Boost tool developed by Save the Children to assess students’ learning levels. This tool is very similar to the EGRA tool which was used for this evaluation. The EGRA tool used Lao as the language for reading and writing as was done in the baseline. However, two sections of this tool were not available and therefore could not be adapted in the EGRA tool for baseline FY17. Thus, those specific sections were reconstructed in consultation with WFP CO which had no implications for the results of EGRA when compared to the baseline FY14. WFP has been aiding remote rural areas with diverse ethnic compositions where Lao is not spoken at home. The pace of learning by children is thus challenged by the limited exposure to Lao at home and is heavily dependent on teaching at the school. The evaluation findings on learning are to be read in light of the above context. 52 Another aspect that needs to be borne in mind while comparing the results is that while baseline FY14 administered the test to Grade 3 children, it was administered to students in Grade 2 during the baseline FY17. This is due to the timing of the survey. The EGRA tool requires that it is administered to students who are either ending Grade 2 or beginning Grade 3. Baseline FY14 was conducted in September which marks the commencement of the semester in Lao. Therefore, the EGRA tool was administered to students who were beginning Grade 3, while FY14 evaluation was conducted during the end of an academic year and therefore administered to Grade-2 students. 53 The primary survey was conducted during Mar-May 2018 due to which enrolment and attendance data for the current school year was limited to Sep-Feb. However, this did not affect the analysis. 54 In absence of the availability of a student list from the selected schools prior to the survey, the students were sampled on the day of the survey. Few of their parents were not available on the same day as they were engaged in their daily works. In some cases, the data collection team revisited the school site for conducting the interviews with parents and, on some occasions, they visited the farms to meet the parents. However, some parents could not be contacted and the sample fell short than what was envisaged. However, given the buffer in the sample, the analysis was not affected. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 9 55 Two schools35 in Ngoy district in LuangPrabang were closed during the time of the survey. Given the poor road and poor weather conditions, the team could not revisit the schools in this district which resulted in a reduction in the sample size for programme schools from 60 to 58. However, since the required sample was 45 schools, the dropping of these two schools did not affect the results. 56 Since children from grades one to five were to be interviewed, there was a risk that they may not be able to provide reliable data due to low recall. To address this, support from class teachers were sought to corroborate the data. Further, parents of the interviews children were consulted to validate the information collected. Ensuring Quality 57 WFP’s DEQAS defines the quality standards expected from this evaluation and lays down processes with in-built steps for quality assurance, templates for evaluation products, and checklists for their review. DEQAS is closely aligned to the WFP’s evaluation quality assurance system (EQAS) and is based on the UNEG norms and standards and good practice of the international evaluation community. It aims to ensure that the evaluation process and products conform to best practice. DEQAS was systematically applied to this evaluation guiding the evaluation team at NRMC. 58 With a rich experience of working with UN agencies, NRMC has a deep understanding of the UNEG norms, standards, and ethical guidelines. Further, NRMC’s internal quality protocols were integrated with the process for information collection, collation, analysis, and delivery. 59 The data collection team comprised of Lao-based personnel who were well versed with the local language and had prior experience of collecting and collating field-level information. The team included members from both northern and southern provinces to ensure that the unbiased information is collected. Further, a gender-balanced team was deployed to gather the perspectives of boys, girls, men, and women. Separate teams were deployed for quantitative and qualitative surveys. Discussions with the Government officials, WFP field offices, and partners were conducted by NRMC core team. A translator’s assistance was used whenever required. 60 A three-day training session on field ethics and data collection tools was conducted for the data collection team by NRMC. Sessions on ethical issues faced during data collection and integrity were conducted. Mock sessions were conducted on interacting with children. The team was provided with translated tools to overcome language barriers. A pilot was also conducted prior to the actual survey. 61 As part of quality control and to ensure timeliness of data collection, NRMC developed detailed field movement plans prior to the survey. A daily team movement plan was shared well in advance with the team. At least one member of the core evaluation team of NRMC was present in the field during the entire period of data collection accompanying the qualitative and quantitative interviewers. 2. Evaluation Findings 62 This section presents the field findings and assesses the relevance, effectiveness, efficiency, impact, and sustainability of the programme with gender equity and inclusion mainstreamed throughout. The findings have been reported based on data collected from both primary and secondary sources (Annex M). For each evaluation question, Table 17 presents the key information areas, the data collection methods, and the data source. Further Table 18 (Annex M) presents the end-line evaluation questions and the paragraph numbers presenting the study findings corresponding to each of the evaluation questions and sub-questions36. For the primary survey, quantitative and qualitative findings from the stakeholders were triangulated before arriving at the overall findings. Wherever appropriate, gender dimensions were factored into the sub-questions, judgement criteria, and the indicators for each evaluation question. 35Houaypoung (EMIS code- 10606052) and Nongkham (EMIS code- 10606025) 36The key questions and sub-questions of the evaluation matrix have been clubbed and categorized into broader themes (based on programme activities and inputs) so as to avoid repetition and enhance readability of the findings. The key themes related to evaluation findings are as follows: a) attentiveness of school-age children, b) enrolment of school age-children, c) attendance of school-age children, d) status on literacy and health, e) impact of the programme on literacy instruction, e) school garden, f) school infrastructure, g) engagement of local organisations and community groups, and h) how have the recommendations made during FY14-16 baseline and mid-term evaluation been integrated into FY14-16 programme implementation. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 10 2.1. The relevance of MGD SFP FY14 in Lao PDR 63 The relevance of FY14 programme has been examined by looking at the needs that the programme intended to support, the coherence of the programme with government policies and strategies, coherence with WFP’s country strategy (2012-15) and the choice of activities implemented through the programme, including gender responsiveness. Need in Lao PDR 64 As highlighted in section 1, despite significant improvements in the status of education over the years, primary school dropouts continue to be very high (22%; HDR 2016). Moreover, the country faces high levels of hunger and malnutrition due to food insecurity and a lack of food diversity. With a score of 27.5, Lao falls in the “serious” category as per the Global Hunger Index 2017. Overall, about 14 percent of the rural population in the country is food-insecure. As per the HDR 2016, poor nutrition is the cause of nearly half of the U5MR. It also leads to stunting and delays in physical development which affects children’s learning ability and the development of cognitive skills. 65 In light of the country context, MGD-SFP was designed to meet the nutritional and educational needs of the country. The programme targeted reaching to schools located in difficult terrain. The school lunches comprising 100g rice, 40g lentils and 10g vegetable oil fortified with Vitamin A and D were able to meet 37 percent of the caloric needs of children. The balance was fulfilled by the contribution of vegetables, meat, eggs etc. from the community. Overall, due to MGD-SFP, children had access to diverse foods. The establishment of school gardens has contributed towards diversifying the food basket and also generating awareness among children on agri-nutrition systems (based on findings from the primary survey and figures reported in the monitoring reports). 66 MGD FY14 promoted literacy activities with an emphasis on girls and mothers. This aided in communicating the importance of education and nutrition among the community. Community-level discussions highlighted the prevalence of social taboos around food and nutrition for pregnant and lactating women. As part of the nutrition education, a need for dietary diversity and its contribution towards reducing stunting in the country was communicated. The findings do show that apart from school, home meals of many families are now more diverse. 67 To reduce health-related absenteeism, WFP collaborated with UNICEF to improve access to water, construct separate toilets for boys and girls, and impart training on hygiene. WFP supported MoH for de-worming of children in schools and carrying out hand washing campaigns. Coherence with Government Policies 68 The MGD-SFP FY14 was coherent with the relevant national policies and strategies for improving the education and nutrition sectors. GoL has been combating malnutrition by promoting dietary diversity both at the household and community levels by implementing a SMAP 2016-2020 and Agriculture Development Strategy (2025). The MGD-SFP was aligned with these strategies and has contributed to GoL’s goal of achieving nutritional security by promoting dietary diversity. 69 MoES is the nodal agency for the implementation of nation-wide school meals in Lao. It has setup an IEC unit for oversight and scaling up of NSMP. WFP closely coordinated with MoES for implementing the MGD-SFP which aided in aligning SFP with NSMP with joint action on addressing the challenges. 70 The NNSPA aims at promoting gender roles, placing emphasis on women’s access to health services, to nutrition and food security information, and to food. Women and girls are to be supported with education and training and enabled to earn an income and participate in household and community decision making. Aligning with the government objectives, MGD also promoted gender and social equality by giving equal opportunity to volunteer for SFP. THRs were given to cooks and storekeepers as compensation for volunteering and encouraging them to continue. Training of department staff and communities were designed to be inclusive and equitable. 71 Towards achieving universal access to primary education, GoL has made primary education free and compulsory. For implementing the activities under ESDP (2016-2020), MGD-SFP activities were aligned with it to provide school meals and literacy and education support in the highest priority districts. Further, the MGD supported districts are among the 65 most disadvantage districts identified by BEQUAL. While MoES was mandated with the construction of school infrastructure, MGD support has been utilised for constructing kitchens, dining area, warehouses, and school gardens. In the model USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 11 schools, fishponds were constructed. WFP has been collaborating with MAF and MoES for formulating guidelines for school gardens and following up with training and establishment of the gardens. 72 WFP has engaged with GoL functionaries in meetings chaired by MPI for programme formulation with ODA support. It has been the co-chair of the coordination group on school meals and has been supporting MoES on finalisation of a handover policy from SFP to NSMP. Underlining the significance of sanitation and deworming, WFP actively supported these activities despite their not being direct activities under MGD, which has contributed to improved outcomes. It has supported MoH with the national de-worming campaign and partnered with FAO on developing curriculum on nutrition and school gardens. 73 WFP has supported VEDCs by imparting training to oversee implementation of the MGD-SFP. LWU made a significant contribution to recipe development and trained cooks at the village level. The Naibans are members of the VEDC and have been trained by WFP in implementing the MGD-SFP. These training programmes have helped in sustaining the momentum of SFP. Coherence with WFP’s Country Programme 74 WFP Country Programme (CP 200242) aimed at supporting government efforts to reduce wasting, stunting, and micronutrient deficiencies. It set a target of reaching 751,000 beneficiaries through five components: 1) emergency preparedness and response, 2) mother-and-child health and nutrition, 3) school meals, 4) livelihood initiatives for nutrition, and 5) food fortification and marketing. While component 1 focused on strengthening the government’s preparedness at national, provincial, and district levels to respond to emergencies, component 2, 3, and 4 focused on reducing wasting and stunting. Under component 5, fortified food was provided to overcome micronutrient deficiencies. 75 MGD-SFP was in alignment with all the five components of CP 200242. The importance of building capacities in the government for making it responsive to community needs is critical for the success of a social safety programme. MoES officials participated in workshops and undertook exposure visits. This helped in creating awareness on the contribution of the school meals to better nutrition and education. Training of VEDC members helped in a better implementation of the SFP. These capacity building activities conform to component 1 of CP 200242. 76 MGD-SFP activities on nutrition awareness (school gardens and fish ponds) contributed to component 2 of CP 200242. 77 Supplementation, diet diversification, and fortification are three types of intervention through which additional nutrition can be provided. MGD-SFP contributed to component 3 by providing fortified oil for overcoming Vitamins A and D deficiencies. This is in addition to the diet diversification through the establishment of school gardens and communities providing vegetables for school meals. 78 Handing over of SFP to government has been articulated in CP 200242. The design of MGD-SFP also included the handover of the programme as a key component. As per the programme design, the handing over has been planned as part of the next round of MGD-SFP FY17 implementation. WFP has engaged with GoL for the preparation and signing of the school meals handing-over plan. Gender 79 MGD-SFP FY14 had emphasised on increasing enrolment, attendance, and retention for both boys and girls. Given a high correlation between stunting rates and women’s low levels of education, WFP encouraged girls and mothers to participate in literacy activities. The WFP School Feeding Policy (2013) and Gender Policy (2015-20) emphasised recognizing and including specific needs of young girls, ethnic and religious minorities, and children with disabilities. WFP advocated the importance of education at the community level through the distribution of posters that emphasised on these. 80 WFP-CO’s 2009 Gender Policy was prevailing in effect during the initial phase of the MGD-SFP. It committed to gender across four dimensions - capacity development, accountability, partnerships, advocacy and research, and operational mainstreaming. The Gender Policy (2015-20) advocated a ‘gender-focused’ rather than a ‘women-focused’ approach. Under this, the strategy was to a) adapt food assistance to the different needs of men and women, b) pursue equal participation, c) empower women and girls in decision making regarding their food security and nutrition, and d) ensure the protection of men and women. Following the recommendations of the MTE 2017, WFP-CO reported integrating gender components in its implementation of the MGD-SFP. This included sharing of tasks USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 12 in school gardens between boys and girls without defining tasks by gender, strengthening of nutrition education, and mainstreaming of gender into field-level activities by using new literacy materials to challenge unsuitable gender roles and improve sensitivity. 81 Despite the integration of gender components in the programme, social norms on the division of labour played a role in terms of cooking and managing commodities for SFP. Field observations indicate that cooking of school lunch has been entrusted with women. Also, the training is provided to LWU thus restricting the role to women. 82 For following safe storage practices, warehouses were constructed on stilts to protect food materials from rodent infestations. However, the access to the warehouses was unsafe. The person managing the commodities was required to climb using a wooden log or a bamboo ladder without any support and in some instances had to jump. This constrained women from taking up the responsibility of storekeepers. In cases, where the woman teacher was given the responsibility of storekeeper, she was dependent on a male member for drawing the commodities from the warehouse. 2.2. The effectiveness of MGD-SFP FY14 (based on performance indicators) 83 This section presents the findings from the end-line evaluation related to the MGD-SFP indicators. Annexes Q and R present the detailed results. Indicator: Number of individuals benefitting directly from USDA-funded interventions 84 Over the project period, 455,024 individuals have benefitted directly from USDA funded MGD-SFP (Annex R), of which 234,465 are male and 220,559 are female. This falls short of the overall target by around 45,000 individuals. Indicator: Number of individuals benefitting indirectly from USDA-funded interventions 85 Over the project period, 252,894 individuals have benefitted indirectly (Annex R). Indicator: Number of students enrolled in schools receiving USDA assistance 86 The target of the program was to reach out to 502,460 students enrolled in schools throughout the country. It has been able to cover 465,426 students (232,713 boys and 217,178 girls - Annex R). Attentiveness of school-age children Indicator: Percentage of children in classrooms identified as inattentive by their teachers 87 Around 21 percent of students in project schools were reported as inattentive by their teachers while the corresponding figure for baseline was 19 percent. While there has been no statistically significant change in terms of the overall proportion of inattentive students over the project period, a significant difference was noted when the indicator was disaggregated by sex37. Inattentiveness was reported to be higher amongst female students (24%) as compared to the male students (17.4%). However, for baseline, a reverse trend was observed with a higher proportion of boys being identified as inattentive (22%) as compared to girls (16%) (Annex Q). Indicator: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance 88 471,650 social assistance beneficiaries are participating in productive safety nets. This includes 235,825 boys and 224,014 girls. While the program falls short by 40,000 beneficiaries as per the target for this indicator, the number of beneficiaries has doubled since the baseline (Annex R). Provisioning of school lunch enhances the social protection on access to food and nutrition by these children. Indicator: Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance 89 According to monitoring reports, 34,427,022 daily school meals have been provided to students (Annex R), which falls short of the target of 67,405,628. A plausible explanation for this could be the delayed commencement of the school lunch program. The first tranche of food supplies had arrived at 37Attentiveness of a student is a function of different variables one of which is short-term hunger; others may include poor teaching methods, lack of interest in the subject matter being taught in class and incidents at home. The evaluation measured the attentiveness of children based on information from their school teachers which may be biased based on the teacher’s perception of the child. Thus, the data for inattentiveness was not considered very reliable and therefore, rationale for the difference among boys and girls could not be explored. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 13 the beginning of the first semester in 2015. Sample schools had incomplete records of lunch meals served which was not found to be reliable and hence has not been considered. Indicator: Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance 90 On provision of school meals, against a target of 405,758 children, monitoring reports indicate that 402,403 children (206,640 boys and 195,763 girls – Annex R) received daily lunch. Discussions with field-level stakeholders indicate that all the children in a school receive lunch on the days it is served. Indicator: Number of schools providing school lunch every day for the past two weeks 91 The primary survey findings show poor performance of the school lunch program in terms of regularity. Nearly half of the schools in the sample (52%) reported that they have not been able to provide school lunch every day for the past 2 weeks (Annex R). In the absence of formal records, information around regularity of school lunch was collected from school head, teachers, cooks, storekeepers, students, and their parents for triangulation. Given that the survey was conducted between end-March to early-May, it is possible that lack of water and availability of vegetables in the school garden may have resulted in the irregularity in the provisioning of lunch in these schools. This needs to be addressed in order to ensure the effectiveness of the program as well as continue to maintain interest among students and goodwill among parents and the community at large. Indicator: Number of take-home rations provided as a result of USDA assistance 92 The MGD Grant was amended in October 12 201638 and THRs for IBs in secondary schools were stopped. 232,668 THRs were provided, well exceeding the target of 183,936 (Annex R). Monitoring reports did not state the number of THRs provided to cooks and storekeepers. Records for this were also not available in any of the schools. Discussions with cooks, storekeepers and school authorities revealed that the food left at the end of the semester is distributed among the cooks and storekeepers. According to the program guidelines, cooks are entitled to receive 50kg of rice per semester. However, this was meant for one cook. With cooking groups being formed, there are a number of cooks in a village who reported receiving very small proportions of rice as incentive for volunteering as cooks. Indicator: Number of individuals receiving take-home rations as a result of USDA assistance 93 According to the monitoring reports, 209,581 individuals have received THRs as a result of USDA assistance. The target was much lower at 78,468 individuals (Annex R). The higher coverage can be explained by the fact that there are 15-20 volunteers in every village now who are working as cooks, whereas the original design envisaged 1-2 cooks per village. Enrolment of school-age children 94 The evaluation team aimed to collect enrolment data from school records for the past five academic years in order to examine trends in student enrolment in WFP-supported schools. Around two-third of the schools did not maintain enrolment records at the time of the survey. About 40 percent of schools had records for at least one of the years, and 34 percent (20 out of 58 schools) had complete enrolment records for all five years. Indicator: Average percentage change in school enrolment 95 Percent change in school enrolment in the current academic year (2017-18) was calculated by comparing the number of students in 2017-18 to the previous academic year (2016-17). There has been a minuscule increase in enrolment (1%) in project schools since the last academic year. However, it was noted that there has been a decrease in enrolment of girls by a percentage point over the last one year. There appears no statistically significant difference in percent change in enrolment rate in schools in Northern provinces vis-à-vis southern provinces, as well as that for enrolment of boys and girls. Further, no statistically significant changes were observed in terms of the percentage change in enrolment during the baseline vis-à-vis end-line (Annex Q). 96 Given that the enrolment rate for primary school is already high (98%), there is little scope for improvement. Studies and evaluations also confirm that there are no gender-based differences in enrolment in primary schools. Further, prior evaluations in Lao PDR as well as elsewhere have established that enrolment is hardly an issue in ensuring quality education. 38MGD commitment letter dated 12 October 2016 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 14 Attendance of school age children 97 Complete information on monthly student attendance over the last academic year was available from 77 percent of the project schools, and an additional 7 percent of the schools had partial data. Overall, data for only 690 students in project schools (out of 1162 in the sample) were available for calculating average student attendance and percentage of students attending school regularly. 98 Student attendance was measured in two ways: the monthly school attendance records of fifteen students per school for the last academic semester (Sep 2017 - Feb 2018) were used to compute average and regular school attendance (defined as greater than or equal to 80% attendance), and the average school attendance on the day of the survey was computed from the number of students present at school on the day of the survey to the number of students enrolled. Although just a snapshot of the day, this indicator was used to triangulate the reliability of school records. Indicator: Average student attendance 99 Average student attendance is high (98%) in the project schools. It is similar for both sexes and geographical location (Annex Q). The baseline figures also show a high attendance rate of 97 percent. During the baseline, average attendance was found to be higher (statistically significant) in the north (99%) as compared to the south (94%). However, schools across both the regions were found to be on a similar plane during the end-line. No statistically significant difference was found between average student attendances during baseline vis-à-vis end-line. Indicator: Percent of students regularly attending school (at least 80 percent of the school days) 100 Nearly all students (98%), across schools in northern and southern provinces, were reported attending school regularly. There is no variation in terms of regular attendance among both the sexes (Annex Q). Baseline figures indicate a similar pattern. No statistically significant difference was found between the percentage of students regularly attending school during baseline vis-à-vis end-line. 101 As per monitoring data, a greater number of students are attending school regularly (269,894) as compared to the target (252,307) (Annex R). There appears no gender-based disparity in regular attendance among students. The target for girls attending school was exceeded by 15 percent. Indicator: Student attendance on the day of the survey 102 Student attendance on the day of the survey was slightly lower than the average attendance and regular attendance, at 91 percent in project schools. There is no significant variation across sex39 (Annex Q). Overall, there has been no significant change in attendance among primary school children over the project period. It may be noted that this value is determined from all children at the schools, and not information for only the sampled students. 103 While poverty remains a prime cause of discontinuation of education, lack of interest among boys and engaging in domestic chores by girls cause absenteeism. Absenteeism among both boys and girls is high during peak agricultural seasons of sowing and harvesting. The banana plantations are harvesting during April and May in the Northern provinces and families along stay there for weeks. Indicator: Percent of students having reduced absenteeism due to USDA support 104 Overall, in sample schools, 10.5 percent of students were reported to be absent for the whole day in the last one week from the date of the survey. There are statistically significant differences across geography with seven percent of students reported to be absent in the north while in the south it was around 21 percent. The mean number of days for which a child was absent was two, with no variation across sex. The mean absent days in North was one, which is not statistically significant. 105 While the primary reason for absenteeism was illness, about 13 percent of students in project schools also reported that they were absent because their parents asked them to stay at home and help out with chores. Disaggregating by sex reveals an equal proportion of boys and girls reported this (12.5%). Status on literacy and health Literacy 39 World Bank (2016) report on “Reducing early grade dropout and low learning achievement in Lao PDR: Root causes and possible interventions”, gender difference in attendance and enrollment starts to be significant after age 14, with early drop out being more common for girls than for boys. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 15 Children’s background 106 Comparing the end-line sample of students with that of the baseline, it appears that it was largely similar in terms of ethnicity. Mon￾Khmer is the most predominant ethnic group in the end-line sample (43%), followed by Lao-Tai (34%), Sino-Tibetan (12%), and Hmong-Lu Mien (6%). Mon-Khmer is the predominant ethnic group in the north while in the south it is Lao-Tai. 107 The mean age of students during the baseline was around 9, which reduced to years during the end-line with no statistically significant variations across sex and geographical location. The sample includes 52 percent of boys, and 48 percent of girls. Detailed description on ethnic group composition is provided in Annex S. 108 Children from the Lao-Tai ethnic group speak Lao at home and are less likely to face problems in terms of the medium of instruction. Those belonging to other ethnic groups are learning to read in an unfamiliar language and hence may be disadvantaged in following what is being taught in school. Children’s home literacy background 109The proportion of students taking extra classes beyond school hours has increased sharply by 40 percent (statistically significant) during the program period. Majority students (93%; statistically significant) reported having someone at home to help them with their studies; during baseline it was 58 percent. Further, the proportion of students reporting the reading of any kind of extracurricular book has gone up from 21 percent during baseline to around 68 percent during the end-line evaluation (statistically significant). For details, please refer to Annex S. Indicator: Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of the grade-level text. 110 Teaching materials and medium of instruction used at schools in Lao PDR follow the Lao language. However, given the wide ethnic and linguistic diversity in the country, exposure to the Lao language at home is limited for the vast majority of children. The pace of learning for such children is thus challenged. WFP with support from partners PI and BBM has been providing assistance to remote rural areas with a diverse ethnic composition where the Lao language is not spoken at home to improve learning abilities among school-age children. 6% 43% 34% 12% 6% Hmong-Eiw Mien Mon-Khmer Lao-Tai Sino-Tibetan Others Figure 1: Ethnicity of students N=866 Box 1: Status on partial absenteeism The school lunch program also aimed to ensure that children stay back in school post lunch. Children going back home during lunch was recognised as a security issue for children as they would be crossing roads unescorted while there is high speed moving traffic. Moreover, it also meant that children sometimes would not come back for the latter half of school thus impacting their education. However, the evaluation team observed that children are still going back home after having lunch and sometimes they did not come back resulting in a fall in attendance during the afternoon classes. Quantitatively, around 40 percent students reported that they were absent in the afternoon after lunch in the last one week. Mean number of days for such absenteeism is one day. Discussions with parents reveal that since all the teachers go back home during lunch, there is no one to look after the children and hence they are asked to go back home too. Furthermore, around 64 percent parents reported that school lunch was served on all school days in the last one week. For the remaining parents, a very high proportion (90%) shared that children came home to eat lunch on days when school lunch was not served. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 16 111 To assess the learning levels among school-age children, the EGRA tool was administered to 175 children studying in Grade 2 in project schools. Additionally, a sample of 148 Grade 3 students in project schools was covered. The EGRA tool used Lao as the language for reading and writing. It used six sub-tests for assessing the learning levels among children: letter recognition, familiar word recognition, decoding words (invented word reading), fluency, reading comprehension, and listening comprehension. In this section, reading comprehension abilities of children are focussed upon. Details of the other sub-tests are presented in Annex S. 112 According to baseline FY14, only 1.9 percent of students demonstrated that they can read and comprehend the meaning of the grade-level text. It is encouraging to observe that the proportion of Grade 2 students who showed at least 75 percent comprehension of the Lao language has increased to about 6.9 percent over the project period (statistically significant) over the baseline (Annex Q). According to project documents, the target for such students is 25 percent40. There emerges no statistically significant variation by sex or geographical location. 113 With respect to ethnic variations, findings from EGRA shows that literacy among Lao-Tai students is considerably higher (statistically significant) as compared to students from other non-Lao-Tai ethnic groups. For instance, around 12 percent of Lao-Tai students in Grade 2 demonstrated at least 75 percent comprehension of Lao language while for non-Lao Tai students, this was only 4 percent. 114 An interesting aspect of the EGRA findings for both Grades 2 and 3 students is that comprehension of Lao language is quite high among the latter. Around 26 percent of Grade 3 students demonstrated at least 75 percent comprehension of Lao language (41% Lao-Tai and 20% non-Lao Tai), which is at par with the target of 25 percent students in this category. This indicates that comprehension of Lao language is better among Grade 3 students which can be interpreted that children perhaps take about three years of schooling to grasp and acquire the language. 115 Disaggregating the above data by gender (Table 3) shows that the level of comprehension is better among boys than girls across both Grades41. As pointed out in the Investment Design Document of BEQUAL (2014), “Girls, particularly those from ethnic minority groups, often experience later starts in education, delaying their Lao language acquisition and creating a life-long disadvantage through dropout and low levels of literacy”. Table 3: Gender disaggregated data for students with at least 75 percent comprehension of Lao language Grade Male students with at least 75% comprehension of Lao language Female students with at least 75% comprehension of Lao language Grade 2 8.7% 4.8% Grade 3 32.5% 19.7% Indicator: Repetition rate 116 Children repeating their grade were recorded during survey data collection. Of the 20 schools for which records were available, it was found that overall, on an average, 8 percent of children were repeating the same grade during 2016-17. This is similar to the repetition rate during the baseline 40June 2018. Semi-Annual report for MGD-SFP 41The difference between girls and boys are not statistically significant at 95% confidence level (Total number of students in the sample for grade 2=175, 92 male and 83 female; total number of students in the sample for grade 3=148, 77 male and 71 female). 90% 3% 7% 93% 5% 2% 0% 20% 40% 60% 80% 100% Emergent (<50% comprehension) Beginning (50-75% comprehension) Reading with Comprehension (>75 % comprehension) BL FY14 EL FY14 N: BL=830 EL= 175 Figure 2: Reading comprehension categorization based on EGRA (Baseline-End line Comparison) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 17 (7.8%). Further, as observed during the baseline, end-line findings also show that the repetition rate is higher among boys (11%) as compared to girls (5%) (Annex Q). The Education for All: Global Monitoring Report (2013-14) also mentions that while gender disparity is not significant with respect to repetition, the male repetition rate is higher than female repetition rate. 117 The secondary literature42 further underlines a strong correlation between dropout and repetition rates. In order to reduce the dropout rate, it is critical to reduce the repetition rate. Indicator: Average students dropout rate 118 The overall dropout rate for 2016-17 was found to be 2 percent, which is lower than the national average of 5.5 percent (UNESCO, MoES 2014). No statistically significant changes were observed in terms of overall dropout rate during the baseline vis-à-vis end-line (Annex Q). 119 The primary school dropout rate for girls and boys was found to be similar (2%) (Annex Q). According to LaoEduInfo database (2014), the national primary school dropout rate for girls and boys was 4.9 and 5.9, respectively. Similarly, there emerged no statistically significant difference between the dropout rate in north and south. 120 The World Bank (2016) report Reducing early grade dropout and low learning achievement in Lao PDR: Root causes and possible interventions states that early dropout is more common for girls than boys. It acknowledges that early grade dropout is a result of household’s socio-economic and cultural factors that lead to a child being pulled out of school. Children are taken out of school when the child is needed at home, or in the field to help around, or when the family has limited financial resources. In a society with strong patriarchal norms, education of a girl child is valued less in comparison to a boy and hence is the first choice to be pulled out of school. The opportunity cost of educating girls is also perceived to be higher by their parents. Therefore, when a girl does not do well or has to repeat a class, they usually dropout. Boys are more likely to be encouraged to repeat a class thus offering a plausible explanation for the higher repetition rate among boys. However, data from the primary survey did not reflect a higher dropout rate for girls as compared to boys. 121 Discussions with the community highlighted that there has been considerable improvement in terms of parents’ perception about educating their daughters. However, given the challenging economic situations that they often face, daughters often get pulled out of schools to help in the household work. In certain ethnic groups like Akha and Hmong, girls are married off at a very early age (after grade 5). Indicator: Percent of students in target schools who start grade one and complete the last grade of primary school (cohort survival rate) as a result of USDA assistance 122 The baseline for the program showed that 60 percent of the students were completing their primary school education. The target of the program was to increase this to 70 percent. The end-line figures show that 80 percent of students are now completing their primary education (Annex R). Increased awareness among community members and improved learning environments at school and at home may have contributed towards this improvement. Indicator: Percent of parents in programme schools who can name at least three benefits of primary education 123 It is encouraging to note that a very high proportion of parents (96 percent in end-line as compared to 45 percent in the baseline) could name at least three benefits of primary education (Annex Q). This increase was across sex and geographical locations. The three most common responses from parents for benefits of primary education were (i) helps children’s skill development, (ii) improves literacy rate, and (iii) improves future opportunities of work for children. Given the poor perception about relevance and quality of education among parents that has historically been a factor keeping children out of school43, this is a positive development. 42http://unesdoc.unesco.org/images/0022/002266/226662e.pdf accessed on 5.07.2018 43The World Bank (2016). Reducing early grade dropout and low learning achievement in Lao PDR: Root causes and possible interventions. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 18 Health Indicator: Health related absence 124 The primary reason for absenteeism among students in the last one week (from the date of survey) was health related (67%). The most common illnesses were reported to be fever, headache, cold and cough. Disaggregating data by geography, a considerably higher proportion of students (statistically significant) reported a health￾related absence in the north (83%) as compared to the south (53%). Indicator: Average dietary diversity score (DDS) of school-aged children 125 The dietary diversity was assessed by collecting complete information on food and drink consumed by each child during 24 hours prior to the interview for 706 school children through interviews with parents. 126 The mean DDS for children in project schools was high at around 7.6 (out of a maximum score of 1044); the corresponding figure during baseline was 5 (Annex Q). This shows that there has been an improvement in DDS although not statistically significant. The mean DDS for male and female students were largely similar, as observed during baseline. While baseline observed significant differences between mean DDS in north vis-à-vis south (north being higher), that difference seems to have neutralised with both faring similarly. 127 Students’ food intake status was measured by categorising individual DDS into different classes. One categorizes DDS in terms of high, medium and low dietary diversity scores, whereas another classifies students into two groups: those that consumed more or less than 5 food items. As shown in Figure 3, only one in every five children exhibited high dietary diversity during the baseline which increased to nearly 50 percent during end-line. 128 58 percent of the students in the baseline sample were consuming five or more food groups in the 24 hours preceding the survey. This has increased to 80 percent during the end-line. Further, a higher proportion of students (statistically significant) in the northern region (72%) are consuming five or more food groups as compared to students in the south (57%). There is no significant variation by sex. 129 Thus, the above findings illustrate that there has been a diversification in the food basket to a large extent over the project period which is a very critical and positive result of the programme. Overall, the food groups that were commonly consumed by students during the baseline included (a) grains, roots and tubers (92%), (b) flesh food including organ meat and miscellaneous small animal protein (82%), (c) vitamin A rich green leafy vegetables (73%), and (e) other vegetables (72%). The end-line evaluation notes the addition of the following food groups in the food basket - (a) legumes and beans (67% over 27% during baseline), (b) nuts and seeds (56% over 24%), (c) dairy products (61% over 20%), and (d) eggs (79% over 46%). 130 The increase in consumption of legumes and beans, nuts and seeds, along with dairy products and eggs indicate an increase in consumption of proteins, fats, and vitamins. The reason for this increase in consumption is that although milk from cows and goats is not widely consumed in Laos, soy milk is gaining popularity. Condensed milk is also being consumed in rural areas. 131 Lentils were being provided under the program to address protein deficiencies. However, the evaluation team observed that the consumption of lentil was rather low and cooks reported that children were wasting lentils. Moreover, since lentils are not a part of the usual Lao diet, cooks also do not know how to cook it and they generally mix it with the rice and boil hem together. Children do not like this preparation and end up wasting rice as well. 44 Feed the Future Guidance (2014) suggests grouping food items into 10 groups. Although classification of food as “other fruits” and “other vegetable” was not clear during EL FY14, the scale has been kept similar to the baseline in order to facilitate comparison. 22% 57% 21% 12% 38% 51% 0% 20% 40% 60% Low Dietary Diversity Score (1-3) Medium Dietary Diversity Score (4-6) High Dietary Diveristy Score (7-10) Endline Baseline Figure 3: Categorising Dietary Diversity Score, by baseline and end-line N: BL=810 EL= 706 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 19 132 Discussions with WFP-CO highlighted that they recognize that getting the Lao population to increase the intake of lentils will take time. However, since animal protein is more expensive than the plant￾based protein, the possibility of switching to lentils is high. There has been an increase in access to lentils and beans (including soy beans), and it is expected that increasing income, better awareness of nutrition, and improved dietary practices will boost their demand. Indicator: Number of individuals trained in child health and nutrition as a result of USDA assistance 133 Monitoring reports indicate that 19,197 individuals (11,508 males and 7,689 females) have been trained in child health and nutrition as a result of USDA assistance. There is a need to increase the coverage substantially to meet the target of 31,999 individuals (Annex R). 134 Primary survey findings point out that 40 percent of teachers (64 male and 56 female teachers) have been trained in child health and nutrition through USDA assistance (Annex R). The figure during baseline was 15 percent. Access to water and sanitation facilities Indicator: Number of schools using an improved water source 135 Of the 58 project schools, only 16 schools (28%) were using an improved water source (Annex R). Only 16 percent of schools in the north were using an improved water source while the figure was 56 percent in the south. In order to improve health and hygiene conditions, it would be important to have access to improved water sources. Indicator: Number of schools with improved sanitation facilities 136 More than two-third of project schools (72%) have access to improved sanitation facilities (Annex R). An equal proportion of schools, with no statistically significant variation, in the north and south, have access to improved sanitation facilities. To give further impetus to behavioural change with respect to sanitation, this coverage needs to be expanded to cover all schools. Availability of water and financial support would be critical towards this. It was found that scarcity of water is currently affecting the regular usage of toilets by students. 137 Only 42 percent of the schools in the sample had separate toilets for boys and girls (Annex R). Hand-washing practices 138 Presence of functioning hand-washing stations could be observed in only a few sample schools in project areas. Also, water scarcity was affecting their functioning. The evaluation team observed that children were washing their hands before and after meals. However, the use of soap was not universal. Also, children were often washing their hands and utensils by dipping them in a bucket of water. Such unhygienic practices need to be discouraged. Safe food preparation and storage practices Indicator: Percent of food preparers in target school who achieve a passing score on a test of safe food preparation and storage practices 139 58 cooks from the sampled project schools were tested on their knowledge and attitude regarding safe food preparation and storage practices. They were served questions on whether they practice the following: (i) use of apron or uniform in the kitchen, (ii) clean the kitchen before and after food preparation, (iii) clean pots and utensils before and after cooking, (iv) use of soap to wash pots and utensils, (v) cleaning vegetables and rice before cooking, (vi) washing hands before and during food preparation with water and soap, (vii) checking for the following in the food before cooking: expiry date, packaging, colour of the food, presence of pests, and (viii) storing cooked food in covered cooking pots in a clean, safe place before serving the students. 140 88 percent of cooks, mostly women, scored at least 80 percent, indicating a high knowledge about safe food preparation and storage practices. The corresponding figure during the baseline was eight percent, thus highlighting the significant improvement over the project period (Annex Q). Indicator: Number of cooks trained in safe food preparation and storage practices as a result of USDA assistance 141 It is to be noted that only around 55 percent of cooks (32 numbers) in the sample have been trained in safe food preparation and storage practices (Annex R). The original program design included one- USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 20 to-two cooks per village who were trained in safe food preparation and storage techniques. However, these were insufficient and a number of cooking groups were formed in the villages with support from the Naiban and Lao Women Union. Most women members in a village then started contributing voluntarily towards the programme. These new cooks have not been provided any formal training on safe food preparation and storage practices. This perhaps explains why around 45 percent of the cooks (26 cooks) in the sample reported that they had not received any training in safe food preparation and storage practices Indicator: Number of storekeepers trained in safe food preparation and storage practices as a result of USDA assistance 142 38 out of the 58 storekeepers (65%) interviewed were trained in safe food preparation and storage practices as a result of USDA assistance. Of this, a higher proportion were in the north (70%) as compared to the south (50%) (Annex R). Indicator: Number of teachers trained in safe food preparation and storage practices as a result of USDA assistance 143 Only about 30 percent of the teachers covered in the sample have been trained in safe food preparation and storage practices. A (statistically) significant higher proportion of teachers in the sample in the south (43%) were trained as compared to the north (24%), although the overall proportion of trained teachers is low (Annex R). Indicator: Percent of storekeepers received a book about warehouse management in the Lao language within the last 12 months 144 Half of the storekeepers reported receiving a book on warehouse management in Lao language (Annex R). The low figures suggest low outreach of such communication material as well as low recall about it among storekeepers. In either scenario, there emerges the need for a wider dissemination and training on using it in order to maintain the hygiene standards. Indicator: Increased access to requisite food preparation and storage tools and equipment 145 Smoke reducing stoves were piloted in 2014 to reduce smoke exposure and the risk of lung diseases. However, most schools (98%) reported not receiving these (Annex R). 146 In terms of infrastructure, nearly all schools (93%) in the sample reported having a dedicated room as a kitchen. 97 percent of the cooks reported that the kitchen is well-ventilated. The program guidelines had specified construction of raised pallets for cooking. Although in around half of the schools food was being cooked off the ground, the majority of them (83%) were using raised pallets for cooking, adhering to the guidelines. In terms of storage, 74 percent of schools reported storing food off the ground to maintain hygiene. 147 Around 76 percent of schools in project areas have access to water for cooking purposes and 67 percent have access to water for washing (Annex R). This has implications on running the SFP. While it is encouraging that 28 percent of the schools had access to piped water for cleaning and cooking food, a majority (38%) were dependent on water from river or streams. These go dry during the winter months impacting the cooking of school meals. Further, it also puts an additional burden of collecting water on the cooks. Figure 4 shows that lack of water for cooking is an issue for over a quarter of the cooks in the sample. 148 Cookbooks have been circulated by WFP recently in the Northern provinces. This provides a variety of recipes for nutritious food that can be cooked. DESB in Oudomxay and LuangNamtha shared that they distributed cookbooks and provided instructions on its use. However, its recall among cooks was found to be very low. Only a fifth (21%) of the cooks in the sample reported receiving this cookbook. With cooks not being adequately trained and not having access to such cookbooks, they end up cooking the same recipe almost daily. The monotony leads to children rejecting and wasting food. Many parents shared that the children have complained about this. Also, all the cooks in the sample expressed the desire to have a varied menu. Moreover, it is important that nutritious food is served in schools. If the cooks are not made aware of the nutrition content of various food items, there are chances of falling short of achieving the health and nutrition goals. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 21 149 Another issue that came up during the FY14 program was that the recipes in the current cook book were more suited for the local food habits in the southern provinces. Quite a few of the vegetables mentioned in the recipes were not produced or consumed in the Northern provinces. The cookbook is being revised by GoL to suit local food habits. 150 The cooks are of the opinion that if eggs and meat were made available for the school lunch, children would prefer them (Figure 4). Lack of cooking utensils is a challenge for 41 percent of the cooks while another 36 percent reported shortage of vegetables as an issue. Impact of the MGD-SFP FY14 on literacy instruction Indicator: Average teacher attendance rates 151 Teacher attendance was assessed using monthly school records and the number of school days per month in the academic semester September 2017-February 2018. The average teacher attendance rate for the previous semester was high at around 97 percent in project schools. The corresponding figure during baseline was around 94 percent, thus there is no statistically significant difference in the average teacher attendance rates over the project period (Annex Q). There is no statistically significant difference between the attendance of female teachers and male teachers, both during baseline and end-line. The baseline observed higher (statistically significant) attendance rate among teachers in the north as compared to the south. The end-line, however, reports similar attendance rates across both regions. Indicator: Percent of teachers attending at least 90 percent of the school days 152 Regular teacher attendance is a critical variable that affects project outcomes. It is defined in this case as those attending at least 90 percent of the school days. With around 94 percent of teachers regularly attending school in the last academic semester (Sep 2017-Feb 2018), there has been an improvement by 10 percent over the project period. The corresponding figure during the baseline was 84 percent (Annex Q). There is no significant difference among male and female teachers, both during baseline and end-line. However, as reported during baseline, evaluation findings report that the proportion of teachers regularly attending school is considerably lower (statistically significant) in schools in the south (83%) as compared to 97% in the north. Indicator: Number of textbooks and other teaching and learning materials provided as a result of USDA assistance 153 While the target was to provide 30,000 textbooks and other teaching and learning materials, so far 71,563 such materials have been provided to schools with USDA assistance, which is encouraging (Annex R). Partnerships with PI and BBM helped achieve such results. According to monitoring report for October 2017 to March 2018, PI and BBM had planned to distribute 60,000 books to teachers and students during the period Oct 2017- March 2018. Further, around 45 percent of the sample schools (26 out of 58) reported receiving such material as part of MGD-SFP. Indicator: Percent of teachers/educators/teaching assistants trained or certified in teaching techniques during the last one year 154 Around 48 percent of teaching staff reported receiving training in teaching techniques over the last one year. There is no significant difference across sex or geographical location. This is an improvement from baseline when 23 percent of teaching staff were trained. It may be noted that all surveyed schools reported that at least one teacher has been trained in teaching techniques during the last one year (Annex Q). 8.6% 36.2% 55.2% 27.6% 8.6% 25.9% 41.4% 19.0% 0% 10% 20% 30% 40% 50% 60% Lack of rice Lack of vegetables Lack of egg/meat Lack of water for cooking No availability of kitchen Unclean kitchen Lack of cooking utensils Limited involvement of community Figure 4: Issues faced by cooks in preparing school lunch N= 58 cooks USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 22 Indicator: Percent of teachers/educators/teaching assistants in target schools who demonstrate the use of new and quality teaching techniques or tools as identified by their supervisor/mentor/coach 155 For teachers who had completed such training in the last one year, responses from the school heads were used to assess their practise. All school heads except one reported that teachers were demonstrating the use of new and quality teaching techniques or tools (Annex Q). Indicator: Number of teacher instruction sets/manuals, guidance, teaching materials and books distributed as a result of USDA assistance 156 School heads reported that they do not maintain records for these. This information was also not available in the monitoring report. School garden 157 School gardens have been globally recognised as an effective means of encouraging a nutritious diet, nutrition education, and development of livelihood skills among students and their families. Indicator: Number of school gardens constructed or rehabilitated as a result of USDA assistance 158 Two-third of the sampled project schools reported having a school garden. According to monitoring reports, quite a high number of school gardens have been constructed/ rehabilitated as a result of USDA assistance (1406 as compared to the target of 768 - Annex R). Discussions with stakeholders indicated that a number of schools who were not under the initial design of the programme were inspired from other schools and demanded the construction of school gardens. Indicator: Number of school gardens able to contribute at least two times harvest to school lunch in a month 159 About 64 percent of the schools were able to contribute to the school lunch at least twice a month in semester 1. This reduces to a low of 36 percent during the second semester (dry season). This is typical in schools in northern provinces with 31 percent reporting an ability to contribute twice a month in the second semester as compared to 50 percent schools in the south (Annex R). 160 Availability of water is a critical factor in the effectiveness and utility of school gardens. It was found that majority of the school gardens were non-operational during the dry season, owing to water scarcity. Figure 5 shows that the proportion of school gardens that do not provide vegetables for the school lunch is considerably higher in the second semester (dry season). Nearly half of the school gardens in the Northern provinces (46%), do not provide vegetables at all in the second semester. While the proportion of school gardens providing vegetables for school lunch during the second semester also goes down in the southern provinces, the decline is slightly less than the northern schools. Given the mountainous terrain in the North, water scarcity inhibits the sustenance of school gardens during the dry season. 161 Rivers, streams, and canals are the most important source of water for irrigating school gardens in the north (32%) which usually go dry during the winter months. School gardens in the south receive gravity-feed water (45%). Overall, around 73 percent school gardens in the southern provinces have access to either a borehole or gravity feed water for irrigation, while the corresponding figure for the north is 50 percent. Access to piped water or a pond within the school premises is negligible across both the provinces. 0% 20% 40% 60% 80% 100% 1st semester 2nd semester 1st semester 2nd semester North South 1-2 times a month 3-4 times a month More than 4 times a month Does not provide Figure 5: Harvest from school gardens in the last one year, by geography USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 23 162 Given the limited access to public irrigation sources, the access depends on the households’/ schools’ economic status. Well-off households have installed bore-wells/ pumps, and schools with better financial status are buying water from these households for irrigation. 163 During the survey (end March to early May), nearly 50 percent of the cooks in sample schools reported that they have not been able to use vegetables from the school garden in the last one month. Thus, it is clear that ensuring availability of water is critical towards utilizing school gardens and providing regular school meals. School infrastructure 164 The program has also focussed on improving school infrastructure. It worked around construction/rehabilitation of kitchen, dining room, storeroom, school garden, and improving access to water and sanitation facilities. Indicator: Number of kitchens constructed or rehabilitated as a result of USDA assistance 165 The target for the program was to construct/ rehabilitate 1339 kitchens. However, around half of the target has been achieved so far (711). Of the sample schools (58), 39 schools reported the construction of kitchen as a result of USDA assistance (67%). Of this 30 (out of 42) were in the North and 9 (out of 16) in the South (Annex R). Indicator: Number of storerooms constructed or rehabilitated as a result of USDA assistance 166 The target for the program was to construct/ rehabilitate 1500 warehouses of which 725 have been completed. Of the 58 sample schools, 36 reported the construction of storerooms (warehouse) using USDA assistance (62%). Of this 25 (out of 42) were in North and 11 in South (out of 16) (Annex R). Nearly all schools in the sample (98%) reported having dedicated storerooms. Indicator: Percentage of schools with access to water for cooking purposes 167 A quarter of the sampled schools do not have access to water for cooking (Annex R). This has implications on the regularity and continuity of the SFP. The situation is particularly worse during the Box 2: Setting up Model Schools: Key to boost effectiveness and utilisation of school gardens MoES, with technical support from WFP have been working towards setting up model schools in targeted districts in order to demonstrate the utility of school gardens. MAF is providing technical support to this. This can be further scaled up to cover all districts so as to promote community knowledge and awareness on these issues. The evaluation team also witnessed some exemplary model schools that have used innovative ways of sustaining school gardens and the overall school lunch program. Depending on the existing resources (water, finance, etc.), they have adopted various methods to ensure adequate resources for continuing the program. In Kokhai village of Sekong district, school staff had planted fruit trees (guava, banana and papaya) along with vegetables in the school garden. They made a good profit by selling the produce from these fruit trees, which they used to provide for vegetables and meat to children in school. They also set up a fish pond in the school premises towards an effective step to supplement protein availability to the children. The school head also expressed a desire to start poultry farming soon. Ban Thiao village in LuangNamtha presents another very inspiring story. With government support, they set up a greenhouse. This boosted their vegetable production throughout the year. Last year, they managed to sell vegetables worth 2,343,000 kips within a span of seven months. They used this money to buy food items to complement the school lunch. However, the green house was not found to be in good shape during the evaluation team’s visit. The school head shared that the plastic used for the green house lasts for around two years. Exposure to strong winds and heavy rainfall had destroyed it and they do not have adequate money to replace them. They hoped that the government would provide financial support for this. Such examples put forth the idea of creating “multiple models of school gardens” depending on the prevalent circumstances. In areas where there is acute water shortage, water resistant cash crops such as Casava can be promoted to complement the earnings of the school. Similarly, green houses can be promoted in areas prone to flooding. This will provide greater incentive to school authorities to maintain the school gardens and also help ensure continuity of school lunch program. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 24 second semester (dry season) when the survey was conducted. The numbers do not vary by geographical location. Indicator: Percentage of schools with access to water for washing purposes 168 A third of the sampled schools reported not having access to water for washing purposes (Annex R). This has implications on the health and hygiene standards mandated under the program. Indicator: Number of schools having well-functioning and clean dining facilities 169 Around 40 percent of schools reported the construction of dining facilities as a result of USDA assistance (15 in the North and 8 in the South - Annex R). More than half of the sampled schools (55%) reported having well-functioning and clean dining facilities (Annex R). Around 70 percent of the schools in the south had access to such facilities while the corresponding figure in the north is 50 percent. Given that the program had commenced later in the south, it is possible that the facilities are better there because they are newer. Indicator: Percentage of schools with library facilities 170 Around 40 percent of the sampled schools reported having library facilities (Annex R). Engagement of local organisations and community groups 171 Nutrition: Under the activity of supporting school agriculture, the SFP advocated for an increased engagement of the community for ensuring the functioning of the school gardens and greenhouses. About 93 percent (54 schools) reported that communities have been contributing to school lunch through the supply of vegetables, eggs, and meat, and by cooking. WFP, through USDA funding, has constructed greenhouses in a few schools on a pilot basis which can be used for demonstrating the benefits to communities across the country and thereby generate knowledge on dietary/crop diversity, climate change, and low-cost agricultural solution among students and communities. To improve access to animal protein, MGD-SFP FY14 had also supported the establishment of fishponds in model schools. 172 Safe food preparation and storage: The program has engaged community members for construction and rehabilitation of kitchen, dining area, storage rooms and hand-washing stations in schools. The VEDC is responsible for overseeing this construction work undertaken by the community. Towards construction of school infrastructure, WFP provided nails, zinc sheets, and hinges while community contributed timber and labour. Under the FY14 award, monitoring records indicate that kitchens have been constructed in all 1146 schools. Around 300 dining spaces have also been constructed. Additionally, communities (women through Lao Women Union) have also been encouraged to volunteer for cooking the meals in schools and contribute vegetables or cash. 173 As per the primary survey, about 90 percent of parents have contributed to the school meals, either in cash or in kind (which includes labour). These were largely in the form of labour for cooking the school meal (72%) and providing vegetables (68%). Other forms of contribution included labour for constructing kitchen, dining space, storage, school garden (17%), and contributing cash (25%). On an average, household contributed between 27,000 Kips (reported by school heads) and 33,000 Kips (reported by parents). 174 The above indicates that the community is actively involved in the school lunch program. There has been a tangible change in people’s perception of the benefits of education. With their contributions in the form of cooking meals, labour for construction, and maintaining school gardens, they visit the school premises more often than they earlier would. This has created a sense of ownership over the SFP activities. Several parents shared that they now have a better awareness of their children’s learning environment and perception about the importance of education. 175 While no formal partnerships were forged with farmer groups, monitoring reports indicate that 948 farmer-parents supplied food to schools (Annex R). 176 Capacity building: Several training sessions were organised on commodity management, safe food preparation and storage, health and hygiene, and nutrition. A number of initiatives aimed at building capacities of VEDC members and officials of the education department towards effective program management, implementation, and monitoring were organised. However, the evaluation findings point towards the need for refresher training as recall among cooks and storekeepers was quite low. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 25 Moreover, it was also found that a number of cooks and storekeepers who were trained are no longer working in that capacity. The replacements reported not receiving any formal training. 177 Capacity building activities undertaken by MGD-SFP exceeded the program targets as can be seen below and in Annex R. Table 4: Capacity building activities undertaken by MGD-SFP Planned Organised National school meal sustainability workshop 1 8 Overseas exposure study tour 1 2 National school meal review workshop 1 2 Exchange visit within Lao 1 6 Awareness raising posters 5,000 26,016 178 However, three nationwide TV and radio programs for awareness creation and three enrolment campaigns were not carried out. 179 Literacy: BBM, with its objective of “making literacy fun”, held half-day book parties for distributing books and promoting literacy among primary school children. At these events, BBM volunteers read aloud stories, played games, sang songs about books, and taught caring for books. At the end of the event, every child received a book. Around 50-80 additional books were given to the school. Reading materials were also shared with parents and other members of the village during these events for community’s access to reading materials and books. To build community ownership, interested individuals were trained and engaged as volunteers for conducting book parties. 180 PI, as part of the BEQUAL consortium, worked towards enhancing Lao language proficiency and promoting reading habits among primary school students. It also worked with teachers to improve their Lao language skills and familiarise them with improved teaching techniques. Interactive teaching materials were provided to aid better learning among students. It also aided caregivers to support their children with reading at home by promoting healthy home and community environment for student’s learning. Towards this, reading festivals and camps were organised in the villages for promoting reading habit among children. Caregivers were further engaged through workshops on reading awareness, creation of reading materials, and establishment of book banks. Awareness sessions were also held in the villages for caregivers to help them understand the importance of education and nutrition for their children. VEDCs were also trained to support and take forward this agenda. 181 As noted earlier in the sub-section on literacy, the above mentioned activities undertaken by both BBM and PI have helped in improving Lao language comprehension skills among children from Grade 2, improved the quality of teaching instruction, and improved the home environment for children to study. The indicators show considerable improvement over the baseline. 182 As per the monitoring reports, 15,454 students have benefitted from literacy campaigns undertaken as a result of USDA assistance against a target of 30,000. Integrating recommendations from FY14 baseline and mid-term evaluation: Findings from the end-line evaluation 183 The FY14 baseline study recommended that WFP should maintain a strong partnership with MoES, as the GoL eventually will take over and run the programme. Building on this and subsequent recommendations made during the FY14 MTE, WFP and MoES signed a joint School Feeding Handover Plan in May 2018. The critical role played by MAF and MPI in achieving the targets has also been recognised. The FY14 end-line evaluation also reiterates the need for strong convergence between WFP, MoES, MAF, and MPI. 184 Community contribution is a key enabler of the school meals programme. This has been highlighted in FY14 baseline study and FY14 MTE. The latter recognised the pivotal role played by VEDCs in achieving this. It also recommended a pilot using cash-based, local procurement models of school feeding to ensure sustainability. The end-line findings indicate considerable community contribution (90%) in the form of cash or in kind for the school meals programme. 80 percent of the VEDCs in the sample schools have been trained. A pilot on cash-based transfers has been initiated in Beng district of Oudomxay province and Nalae district in LuangNamtha. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 26 185 WASH facilities were identified as another critical aspect for the success of the programme. MTE suggested that WFP should work with UNICEF to meet infrastructure requirements and advocate for WASH activities. WFP collaborated with UNICEF for constructing toilets and GIZ for constructing hand washing stations. Good coverage of toilet facilities was observed. However, lack of availability of water is a serious concern affecting usage of WASH facilities. 186 Capacity building of stakeholders, particularly cooks and storekeepers, was identified for strengthening during the FY14 baseline study. The MTE reiterated this need along with capacity building of government officials and community members. The end-line evaluation findings show that the training programmes and exchange visits for government staff have been completed and more are planned. Capacity development activities for VEDCs are also being conducted to deepen the understanding of the SFP. However, training coverage for cooks and storekeepers needs further improvement. Refresher training is also desirable. 187 Other areas of improvement identified by the FY14 baseline study and FY14 MTE are a) improving student knowledge on nutrition, good health, and hygiene practices, and b) strengthening school-level record keeping and maintenance of quality data for enrolment and food receipts. For the former, WFP is working towards introducing nutrition and school gardens in the primary education curriculum. Exchange visits and workshops are also being conducted to facilitate peer and cross learning. 188 With respect to monitoring systems, limited records were found to be maintained in schools during the FY14 end-line evaluation too. Following the recommendations of the MTE, reporting formats have been simplified. Insufficient staffing was recognised as a problem and WFP has signed a MoU with Lao Front for National Development to support monitoring activities. 189 The FY14 baseline study did not present any gender analysis of the MGD-SFP FY14 programme. However, FY14 MTE provided useful recommendations to mainstream gender into field activities by using new literacy materials to challenge traditional gender roles. It was found during the FY14 end￾line evaluation that such literacy material have been developed and circulated. However, gender equality is yet to be achieved in certain activities like cooking and commodity management. The end￾line evaluation finds that women are responsible for activities such as collecting firewood and vegetables, fetching water, and cleaning as most cooks are women; which is likely to contribute towards women drudgery. Details on integrating the recommendations from FY14 baseline study and FY14 MTE are in Annex T. 2.3. Impact of MGD-SFP FY14 Increased economic incentives 190 For villages where a higher number of families support the school feeding programme, the cost for each family was found to be lower. This finding further highlights the need for boosting community contribution. It can be shared as an example to incentivise communities to contribute at a lower cost. 191 By provisioning school lunch, MGD-SFP aimed to keep children back in the school during and after lunch. This also acts as an economic incentive for parents who shared that usually mothers had to return from the field during lunch to feed the children. However, now that children are fed at the school, they can spend longer hours at work thus enhancing their productivity. Furthermore, this additional time is also a relief to the women of the household (mostly mothers). Not having to prepare lunch for their children and be at home around that time gives them the opportunity to do other things or be at leisure, given that they are largely responsible for care/home duties. However, it is to be noted that a smaller proportion of parents feel this way because in majority cases, children are still coming home after having lunch. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 27 Impact of school gardens on increasing knowledge on nutrition 192 As per the findings of the primary survey, school gardens had helped in increasing knowledge among students on school gardening. Their involvement in sowing, irrigating, tending, and harvesting had helped in increasing awareness about farming techniques. In an agrarian economy, this is also helping students build livelihood skills. However, evidence for increased awareness around nutritional value of vegetables was found to be weak among students. Many students also did not know the vegetables that were growing in the garden. 193 Nutrition education through school gardens has improved the knowledge among the community. This, in turn, has led to positive changes in dietary patterns at the household level. While the comparison of DDS over the project period provided evidence to this finding, discussions with stakeholders indicated that these changes are at a preliminary stage. Gender-specific impact of MGD-SFP FY14 Alignment of the program with the priorities of women, men, girls, and boys 194 Perusal of secondary literature indicates that “women’s literacy is lower than men’s, especially in rural areas without road access, amongst the poorest group and ethnic groups living in remote areas” (MTE FY1445). Moreover, it has been found that stunting rates are four times higher among children of uneducated women than among children of mothers with secondary or higher education. 195 The causes for absenteeism among boys and girls were found to be different. For boys, it was primarily due to engagement in farming and a lack of interest in studies, while for girls the primary cause of absenteeism was taking care of household chores and their younger siblings. This conforms to the conventional gender roles of women being the “care giver” and men as the “bread earner” in a family. Further, it was reported that girls from certain ethnic groups such as Akha and Hmong were married off at an early age leading to dropouts after completion of primary education. 196 In order to address these issues and change the perception of gender roles in society, WFP intends to encourage girls and mothers to participate in literacy activities to increase the number of girls attending school. The WFP School Feeding Policy (2013) and Gender Policy for 2015-2020 lay emphasis on recognizing and including specific needs of young girls, ethnic and religious minorities, and children with disabilities. In order to promote inclusion and equity in education, WFP has supported the Government’s efforts to increase enrolment and attendance of schoolgirls. WFP advocates the importance of education at the village and household levels through conducting parenting awareness sessions, creating book banks containing age appropriate books, and distribution of poster that emphasises on the importance of regular attendance for boys and girls. Gender sensitive design and implementation of the program 197 The MGD FY14 programme design is gender sensitive and has integrated gender-related recommendations of the MTE FY14. This includes the promotion of sharing of tasks in school gardens between boys and girls without defining tasks by gender, strengthening of nutrition education, and mainstreaming of gender into field-level activities by using new literacy materials. 198 The number of beneficiaries reached through the program is gender-balanced, reaching out to an equal proportion of boys and girls (Annex R). MGD-SFP has maintained gender parity in promoting enrolment rates and addressing drop-outs among both boys and girls. The awareness-building and community sensitisation activities focussed adequately on the importance of education for both sexes (Annex Q and R). With respect to school gardens, the evaluation team witnessed equal participation of boys and girls in activities such as sowing, watering, and cleaning. 45As stated in the 8th Five Year National Socio-Economic Development Plan “Involvement in gardening improves children’s attitude to the natural resources and the environment. They develop a practical understanding of the environment and its ecosystem. Given the current scenario around global warming, this is important”. DESB, Thateng, Sekong Province USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 28 199 In order to prevent increased workload for women cooks in fetching water for school lunch, the program advocates sharing of responsibilities. However, given that women are the ones engaged in cooking, the responsibility lies on them, further adding to their work burden, especially in school in water scarce areas. These female cooks also are responsible for collecting firewood and vegetables for school lunch. Thus, the workload of women in cooking school lunch- both physical and economic is way higher than that on men, leading to further drudgery among them. At times, they are unable to spare the time to collect firewood or vegetables and school lunch is given a miss. Therefore, it desirable that the project increases its efforts to ensure sharing of these responsibilities between men and women to reduce women drudgery as well as ensure regularity of school lunch. 200 Smoke reducing stoves were piloted for cooking school lunch which was not found to have been implemented fully. The target was to supply these stoves to 768 schools, of which only 48 schools in one pilot district in the north have been covered so far as part of the first roll-out (Annex R). Only one school in the primary survey sample reported receiving it. Under MGD FY17 award, WFP plans to spend about US$ 147,000 towards installing such stoves in partnership with The World Bank. Impact on gender roles in the medium and long-term 201 MGD-SFP has also encouraged VEDCs to have both male and female cooks for school meals. However, discussions with the communities indicated that the opinion is biased against men’s involvement in cooking as it is regarded as a feminine chore. Nonetheless, encouraging instances of husbands cooking in schools on behalf of their wives were noted. VEDCs were encouraged to share photographs of men cooking in order to encourage more numbers to men to cook school lunches. 202 Another relevant finding was that a majority of the storekeepers (67%) in the sample schools were men. Yet again, this reinforces conventional gender roles wherein men are “supposedly” in charge of procuring/ receiving materials, maintaining records, and managing the stock that reiterates their role as ‘providers’. There is a felt need to maintain gender balance in the allocation of cooking and storekeeping responsibilities. Also, the storeroom designs need to be made safe and women-friendly. They are usually constructed in stilts with storekeepers usually jumping or using wooden planks to access it, thereby making access an issue for the women. Role and nature of participation of women in the program, specifically VEDC, Lao Women Union, and other local school feeding management and support committees 203 By forging partnerships between VEDC and Lao Women Union, women’s participation has been promoted and they have been encouraged to take up leadership roles in school management committees. However, evidence of women in leadership roles in VEDC was found to be weak. Sustaining the gender-based outcomes 204 It was observed that GEEW has been mainstreamed throughout the program activities involving school aged boys and girls. The program design for MGD-SFP and WFP-CO’s implementation and monitoring plan have shown considerable attention to gender issues and conform to the national gender commitments and policies. The program was found to benefit boys and girls in a similar manner and shows responsiveness to gender-based needs and differences. It would be imperative to continue to mainstream a gender focus in order to achieve sustained results. To do so, it would be critical to enhance the participation of women in leadership roles in VEDCs and changing conventional perceptions around gender roles. It would also be crucial to mainstream gender in activities such as cooking of school meals and storekeeping. The existing drudgery of women resulting from SFP needs to be addressed through improved sharing of responsibilities and access to improved infrastructure. 2.4. The efficiency of MGD-SFP FY14 205 Two dimensions - allocation and financial– have been applied to assess efficiency. While allocation efficiency examines the choice of the interventions and their usage at an optimal level, financial efficiency measures the ratio of expenditure to that of the intervention. Additional dimensions for measuring efficiency could be spatial targeting (interventions in the hard to reach pockets), administrative efficiency (selection of resources) and management efficiency (processes adopted). Within the current evaluation, these have been grouped under two questions: a) was the targeting under the programme done through an efficient allocation of resources, and b) whether MGD-SFP FY14 was implemented in the most efficient way as compared to alternatives? USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 29 Efficient allocation of resources under MGD-SFP FY14 206 As elaborated in the relevance section, the MGD-SFP FY14 responded to the needs for addressing issues related to children’s education and nutrition in Lao. The programme supported GoL’s agenda for increasing retention in school, reducing drop-out and absenteeism, improving the quality of education, and reducing stunting in the country. Towards achieving these goals, WFP directly supported nutrition improvement and, with partner’s assistance, addressed literacy. 207 Secondary research from developing countries has shown that school meals are an efficient approach to improve enrolment and retention, and for children’s physical and mental development. This becomes more crucial if the targeted population are economically backward, reside in remote locations, and are food and nutrition insecure. The program was implemented in districts which ranked low on socio-economic indicators of BEQUAL and had presence of ethnic minorities. 208 The MMS encouraged student’s enrolment and attendance, and also reduced hunger. However, children do eat breakfast at home before coming to school. Therefore, rather than MMS, school lunch was a more appropriate intervention. With this transition, students are not required to go back home during lunch46. Further, with MGD-supplied food materials, vegetables from school gardens, and community contribution of meat and eggs ensured that students were eating nutritious food. 209 WFP had identified hygiene as a core area of intervention during the transition from MMS to school lunch. To facilitate safe hygienic practices, the programme supported construction of the kitchen area with raised platforms, clean dining area and warehouses for safe storage of commodities. WFP collaborated with UNICEF for constructing toilets and GIZ for hand washing stations. Engaging with the community in constructing school infrastructure and supporting the school lunch, has also improved bonding between the schools and communities. This was an effective and an efficient response mechanism towards building ownership among communities for sustaining the school meals programme. 210 Despite the initial setback due to non-participation by the local partner for promoting literacy activities, WFP improvised by partnering with PI and BBM. The program supported the joint literacy approach being implemented by BEQUAL in 10 schools of Nga district in Oudomxay in partnership with PI. The partnership with BBM led to improved access to reading material and literacy in 99 schools of Phongthon and Ngoi districts. These partnerships helped in establishing libraries and book corners in school, creating community book banks, and supplying books and reading materials. Discussions with teachers revealed these interventions have increased reading habits. Teachers who had received the teaching materials were highly appreciative of the same. 211 WFP worked closely with MoES at the national, PESS at the provincial, and DESB at the district levels for implementing the programme. WFP has trained GoL officials and VEDC members on monitoring the programme and building their implementation capacities. WFP participated in the coordination meetings of MPI for providing technical support to the government for implementing the programme after WFP’s exit from implementation. WFP has worked towards building a network of donors, NGOs, and GoL officials whose contributions have aided in maximising the impact of the activities and returns on the investments made. 212 The total expense for FGD-SFP FY14 was US$ 27 million47. Of this amount, about 85 percent (US$ 23 million) was spent on providing food materials for 454,889 beneficiaries (including students, cooks, and storekeepers) over three years. Thus, the annual per capita expenditure per student is estimated to be US$ 16.8. Under NSMP, cash transfer at the rate of US$ 0.1 per student per day was being made. Assuming that the schools were open for 165 days, an allocation of US$ 16.5 per student was made under NSMP for purchasing of food materials. The comparison of these rates suggests that the MGD￾SFP allocation of financial resources were appropriate and matched with GoL and World Bank supported school meals programme. Further, the programme team transitioned gradually from MMS to school lunch to ensure that the schools, VEDC members, teachers, school heads, cooks, and storekeepers were adequately equipped to deal with the transition. 46 A number of students do go home for reasons (and solutions) discussed earlier. 47 Commitment letter for Agreement #FFE-439-2014/049-00 Laos USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 30 Efficient implementation of MGD-SFP FY14 213 Apart from MGD-SFP, the MGD-LEAPS I (2013-2016) implemented by Catholic Relief Services (CRS) 48and Education for All (EFA) (2009-2015) implemented by The World Bank were being executed in parallel but in different provinces. The subsequent phase for both the programmes was launched on completion of the first phase as with MGD-SFP. EFA has now been adopted as NSMP. 214 Comparison of the three programmes reveals that these were implemented in the poorest of the districts across the country without any overlapping of the geographical areas. The objectives of all the three programmes were similar: a) ensuring equitable access to education and nutrition services, b) improving the quality and efficiency of education and nutrition services, and (c) improving sector￾wide governance. Underpinning these objectives was provisioning of school meals to increase enrolment and regular attendance, reduce hunger, and improve literacy. Complementary activities like procuring local food (home-grown school feeding- HGSF), training village level institutions like VEDCs, and training of teachers were also undertaken under all three programmes. 215 The table below presents the number of direct beneficiaries of the programme and the cost of school meals programme: Table 5: Comparison of different school meal programmes Programme Period Implemented by Direct project beneficiaries Cost of school meal component Per capita investment MGD-SFP 2014-17 WFP 454,889 US$ 27 million US$ 59.35 MGD-LEAPS I 2013-16 CRS 144,000 US$ 12.3 million US$ 85.41 EFA (NSMP) 2009-15 World Bank and DFAT 330,000 US$ 25.15 million* US$ 76.21 *- Excluding school construction cost49 216 The per-capita beneficiary investment is a broad indicator and has to be read in context with the extent of support provided on various activities by each implementing agency. For instance, World Bank supported EFA spent about a fourth of the funds (US$ 6.58 million) on stewardship of education sector at the local, provincial and national levels. This is an important investment as the government capacities are being built for efficient implementation of NSMP throughout the country. 217 An important aspect emerging from the comparison of these programmes was that MGD-SFP FY14 was in alignment with other similar programmes and the needs of the education and nutrition sector in the country. Through the establishment of school gardens, fishponds, and livestock development, it has attempted to ensure a continuous supply of vegetables and meat. Similar to other two programmes MGD-SFP is also building capacities of VEDCs and teachers. To ensure, that government officials from the education department are able to continue to implement MGD-SFP as NSMP, WFP has been closely coordinating with MoES, PESS, and DESB officials. 218 It can thus be concluded that MGD-SFP FY14 was efficiently designed and implemented. It aligned with the Government’s objectives and adapted to the changing environment by switching over from MMS to lunch in a phased manner without any extra financial implications on USDA. 2.5. Sustainability of MGD-SFP FY14 219 This section seeks to assess the extent to which the outcomes achieved under the program can be sustained after withdrawal of USDA support, and the critical conditions for ensuring sustainability. 220 Sustainability of program outcomes: MGD-SFP FY14 has demonstrated an effective model for improving learning abilities and nutrition among children. It has created an effective platform for creating knowledge on health, nutrition, and the importance of education. 221 Sustainability is driven by a combination of stakeholder capacities, role clarity, resource availability, ownership, and the intent of stakeholders. These elements play a critical role in ensuring the continuity of a programme after the exit of the external agency. In the long run, communities on the 48 The Learning and Engaging All in Primary School (LEAPS) Program was a three-year (2013-2016) food for education (FFE) program funded by USDA. Implemented by CRS, the aim of the program was to promote primary education and reduce the incidence of hunger for approximately 36,000 primary school-age children in over 300 schools each year across six educationally￾disadvantaged districts in Laos. 49 http://documents.worldbank.org/curated/en/754651468047761710/pdf/ICR32860P114600C0disclosed020270150.pdf USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 31 demand side and government as the supplier have to work in close coordination and shared understanding to sustain and also improve the outcomes. 222 Overall, MGD FY14 has engaged with all critical stakeholders through various programmatic activities from both the demand as well as the supply extremities. This is likely to help in sustaining the programme after withdrawal of USDA-WFP support. It lays emphasis on the roles of government, school, community, aid partners, and involving NGOs. 223 WFP has been training the communities on the importance of school gardens for improved nutrition. The end-line evaluation shows that there has been a change in the dietary diversity of children and, to a certain extent, at the household level as well. The shift towards a diverse food basket can contribute to the sustainability of school meals as it is expected that families will support school meals by providing vegetables and other materials. 224 Sustaining school gardens requires support to access water especially during the dry season. Similarly, communities need to be aware of their role for the continuity of the program. Policy makers and GoL officials have to be further trained. In order to sustain the nutrition related outcomes, it would be imperative that all stakeholders have adequate knowledge of the nutritional value of various foods. 225 One of the challenges to sustainability is the growing number of private companies engaged in large scale banana and tobacco plantations. Being cash crops, farmers have started planting these for enhancing their income. Due to this, they may not be willing to grow vegetables contribution to school lunch. People are also being hired in these plantations as wage labourers and families are migrating during the peak season which can increase absenteeism and dropouts from schools. 226 Cooks are central to the success of MGD-SFP and the NSMP. Hence it is imperative that a structured approach is adopted towards training and ensuring regularity of cooks to sustain the school meals programme. The concept of multiple groups (each with 3-4 persons) cooking on a rotational basis is a practical approach to reduces the burden on a fewer numbers. Rostering needs to be adopted and communicated to all the groups. At the same time, each group can inform and remind the group responsible for the next day’s cooking. Under NSMP, cooks would be responsible for purchasing the cooking ingredients and will burden a single person which may impact the cooking efficiency. All those who are involved with cooking need to be trained on safe cooking methods. 227 Given the prevalent gender inequality in engaging cooks, equal sharing of responsibilities between men and women should be encouraged for providing school meals. This will help reduce women drudgery and encouraging everyone to contribute will help ensure sustainability. Through this approach, youths especially boys can be persuaded to participate in the school meals programme. These in turn will impact gender relations in the medium and long term. 228 By design, MGD has also attempted to promote animal-based protein. For this, the programme can expand the establishment of fishponds and poultry. 229 Alignment with GoL policies: As presented in section 2.1, the SFP is well aligned with government policies on attaining nutritional security and improving the quality of education in the country. 230 Monitoring of the programme: Concurrent monitoring is essential and a strong M&E system is desirable for sustaining the programme and assess performance. It was observed that records on attendance and school feeding are rarely maintained by the schools. Information on enrolment, attendance, dropout and repetition, number of meals served, students benefitted, THR provided, utilisation of food supply, etc. were limited. Maintaining such records is critical for monitoring performance and identifying areas that require strengthening. Issues need to be identified and resolved at the local level and in case it cannot be resolved at the local level, it needs to be scaled up. It was also learnt by the evaluation team that most of the storekeepers were not aware about how to fill the record book and therefore they were not maintaining these records. Hence, further trainings on these aspects become critical. 231 WFP has initiated a mobile based IVR system for the community to reach programme staff. Discussions with WFP staff at the Pakse field office indicated that very few people have used this service. They share that communities instead prefer to meet the WFP official in-person in order to raise their issues; the perception is that their grievances would be better heard then. However, given the limited number of staff available in the field offices, it is not possible for them to visit the schools regularly and as a result, lot of issues are reaching late to WFP staff to act upon. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 32 232 The format for reporting against performance indicators was revised based on the MTE recommendations. However, there is scope to further improvement. For instance, whether the yearly achievements are cumulative or yearly numbers is unclear. At times, the data and the corresponding comments are inconsistent. For example, while the value for the indicator “number of individuals benefiting directly from USDA-funded interventions (male)” varies between two semesters in the Lao￾Semi-Annual Report Oct 16-Mar 17, the comments remains the same. 233 Moving ahead, WFP is now engaging Lao Front for National Development for monitoring of the SFP. Since the members from this group are present in all the villages, it will be convenient for them to monitor and report. After MGD’s exit, this structure can support MoES on monitoring and reporting for NSMP. 234 MoES also has Education Management Information System (EMIS) though which it collects information on the condition of schools, number of classrooms, student enrolment data, repetition data, ethnicity disaggregated grade graduating data, and a list of teachers. While assessing the reliability of the EMIS data is beyond the purview of the end-line evaluation, data on student enrolment, dropout, and repetition were difficult to access. It will be useful to include school meals programme data in EMIS. At the same time, it will be helpful, if union members who are going to monitor the SFP, also provide additional information on students, teachers and school infrastructure and literacy activities undertaken in the school. This will also aid in tracking attendance of girls, the participation of women in literacy activities, temporary dropout of boys during sowing and harvesting season etc. 235 Preparedness among stakeholders to continue the program: Discussions with PESS and DESB officials and schools in the Northern Province indicated that they are aware of the transition of school meals from WFP to GoL. However, they are not completely aware of their roles and responsibilities. Support and guidance needs to be provided to the officials at the provincial and district levels. Discussions with WFP indicated that the roles and responsibilities will be agreed upon with MoES around June 2018 and communicated to PESS and DESB. 236 While the schools were confident that they will be able to continue the programme, concerns regarding timely release of funds were voiced. Another concern was the amount (800 kips per student per day) that will be transferred for school lunch. With the cost of one egg at 1000 kip, there are reasonable doubts on the sufficiency of the amount that the government is proposing to transfer. For a successful transition and sustaining the school lunch, training support needs to be provided to school officials and VEDC members on the balanced utilisation of the combination of resources- financial from government and material from the community. 237 Planned measures for ensuring sustainability: For a smooth transition of schools from SFP to NSMP, WFP and MoES have jointly drawn up a School Feeding Handover Plan with a phase-wise approach of the transition and the transfer to MoES. This plan was approved and signed in May 2018. Workshops are planned for the National Assembly Members to sensitise them on the handover. The plan also lays emphasis on MoES to include the school meal expenditure of current WFP supported schools in the national budget. Private sector participation is also envisaged. On complete handover of all schools to MoES, WFP plans to continue monitoring, supervising and providing technical support to these schools. 238 National and international exchange visits for government staff have been carried out. WFP has facilitated an exchange programme for MPI, MoES, and MAF officials to Brazil. This has encouraged the GoL in detailing out a clearer oversight of roles and responsibilities at the national, provincial, and district levels. WFP, in collaboration with MoES, is also attempting to set up a School Meals Centre of Excellence. Joint training programmes are planned to build capacities of GoL officials on school agriculture and nutrition, education, hand-washing, and hygiene, using a ToT model. 239 To ensure sustainability of SFP, WFP has drawn up a strategy that focusses on building capacities of VEDCs. To empower the VEDC members, capacity development activities like training, workshops and exchange programmes have been included. Exchange visits for VEDC members have been conducted in the last quarter of 2017 to draw lessons from the challenges encountered and the solutions adopted. 240 Towards ensuring a smooth handover of SFP, WFP will adopt a three-step approach: a) assessing the community strength of managing SFP, b) analysis of results, and c) provisioning targeted support USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 33 package. WFP will be supported by LWU in the training of women in villages for a better implementation of school meals in the schools. 241 The MGD programme also envisages informal partnerships between schools and farmer groups for ensuring a continuous supply of vegetables. The role of WFP monitoring assistants (MA) will expand and they will become the change agents for the community. The MAs will also be responsible for community mobilisation. 2.6. Theory of Change for MGD-SFP FY14 242 The MTE had developed a Theory of Change (ToC) for FY14 programme and identified the assumptions based on the results framework. These assumptions were critical for achieving the intermediate and long-term outcomes envisaged by the programme. The MTE had reported that many of these assumptions were invalid or problematic. The end-line evaluation has examined each of these assumptions and has presented the detailed findings in Annex H. 243 The evaluation founds all the assumptions to be relevant and the status of some of the indicators have shifted from red to orange and orange to green. However, only eight out of the 15 assumptions are in the green zone. The following six assumptions are in the orange zone and need attention: i. Partners and contributors adequately providing effective complementary activities: UNICEF has to ensure access to water for ensuring functionality of toilets and school gardens. ii. There would be continued government support for a national school feeding programme: GoL is indicated commitment towards providing adequate finance for NSMP. The current NSMP is supported by The World Bank. iii. Government is willing to commit funding to school feeding: This will be visible during the current phase (FY17) of implementation. iv. The viability of the programme depends on the activities around systems strengthening at all levels: Activities around systems strengthening have commenced. However, the governance structure is unclear, but is expected to be clearer over the next three years. At the ground level, many of the trained storekeepers and also some cooks have left and they have now been replaced with new ones. v. There are adequate staff and equipment and structures in schools to support learning: Grade-wise classrooms are not available in all schools. Improved teaching and instruction materials have not been provided to all schools. vi. Training activities are designed to support learning principles: Currently, not all cooks are trained. With multiple cooks associated with SFP, it is imperative to train them all. 244 The assumption on monitoring and reporting systems is in red zone highlighting the need of immediate intervention. A data management system needs to be established at MoES. 245 Annex H details out the implicit assumptions made in the Theory of Change along with the MTE findings. It also elaborates on the reasons emerging from the end-line evaluation as to why the assumptions are valid and may or may not need additional focus for sustaining the programme and achieving the outcomes. 3. Conclusions and Recommendations 3.1. Overall Assessment/ Conclusions 246 The MGD-SFP was positive in terms of design and implementation. The targeting of the most vulnerable groups under the programme and aligning with the government’s policies and strategies reflects a strong understanding of the country context. Building community ownership through their participation in programme activities along with communication on the importance of education and nutrition are highlights of the programme design. 247 The most important area that requires WFP’s attention is the absence of a robust monitoring mechanism. Knowledge on nutrition education needs to be strengthened. Also, access to water is critical for WASH, school gardens, and cooking. Conclusions against evaluation criteria USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 34 248 Relevance: The initial MMS followed by the school lunch programme were implemented in the economically and educationally backward districts of the country. By targeting school going children, the programme has contributed to the overall objective of improving the education and nutrition status in the country. By promoting community contribution for school meals, the programme promoted ownership among the communities. The contribution of timber and labour by the community and zinc sheets and nails from WFP helped in the construction of kitchens, dining areas, and storage infrastructure. 249 The establishment of school gardens has helped in demonstrating the cultivation of different vegetables. It also helped the students to understand the agri-nutrition systems which will be useful in the medium to long term. The transfer of technical knowledge on nutrition and school gardens has helped the communities to diversify their food basket. 250 The promotion of literacy activities at the community level emphasising on the participation of girls and mothers aided in communicating the importance of education and nutrition to the community. This activity also complemented the establishment of library and book corners in the school and thus creating an enabling environment for children to study. 251 The programme was coherent with SMAP 2016-2020, ESDP (2016-2020), and NNSPA through which the government is combating malnutrition and attempting to improve the enrolment, attendance and reduce dropout and grade repetition of school children. The programme design was completely aligned with NSMP being implemented by MoES. WFP has been collaborating with MAF and MoES for formulating guidelines for school gardens and following up with training and establishment of the gardens. 252 WFP has undertaken capacity building activities under the programme to support the demand and supply side stakeholders and processes. At the operational level, training was provided to VEDC members. At the strategic level, training was provided to officials at the central, provincial and district levels for effective implementation of SFP. 253 MGD-FP FY14 was coherent with WFP Country Programme 200242. The programme was fully aligned with component 3 (providing school meals), component 4 (livelihood initiatives for nutrition) and partially aligned with component 1 (emergency preparedness and response), component 2 (mother￾and-child health and nutrition) and component 5 (food fortification and marketing). 254 MGD-SFP FY14 had emphasised on increasing enrolment, attendance, and retention for both boys and girls, including differently-abled children. WFP advocated the importance of education at the community level through the distribution of posters. The gender components included in MGD-SFP FY14 included measures such as the promotion of sharing of tasks in school gardens between boys and girls without defining tasks by gender, strengthening of nutrition education, and mainstreaming of gender equality into field-level activities by using new literacy materials to challenge traditional gender roles and improve sensitivity. 255 Effectiveness: MGD-SFP reached out to 455,024 individuals directly and another 252,894 individuals indirectly. This includes students, parents, farmers, and other community members. 256 The program has contributed towards improving dietary practices, creating awareness among parents about the importance of education, and improving the learning environment of children at the school and home. These have contributed towards achieving the objectives of improved literacy of school-age children (SO1) and increased use of health and dietary practices (SO2). 257 There has been considerable improvement with respect to children’s reading and comprehension abilities. The proportion of Grade 2 students with at least 75 percent comprehension of the Lao language has increased to about 6.9 percent as compared to 1.9 percent during the baseline. An interesting finding from the evaluation is that the comprehension ability of Lao language was considerably higher among Grade 3 students as compared to Grade 2 students. Overall, it was found that the level of comprehension is better among boys than girls. 258 The program has also worked towards improving attendance and enrolment of children. However, given that the attendance and enrolment rates reported during the baseline were itself quite high, there was no significant improvement over the project period. Nonetheless, by providing support to children for learning and improving their learning environment, the program has contributed to increasing the proportion of students who complete primary school from 60 to 80 percent. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 35 259 The proportion of students taking extra classes beyond school hours has increased by 40 percent. Around 35 percent increase in students reporting having someone at home to help them with their studies, and around 47 percent increase in the proportion of students reporting reading of any kind of extracurricular book were observed. Moreover, 96 percent of parents could name at least three benefits of primary education as compared to the baseline (45%). 260 The proportion of teachers’ regularly attending school (at least 90% of the school days) has improved by 10 percent. The proportion of trained teaching staff was also found to have doubled. 261 Significant improvement could be seen with respect to the nutrition intake among children and diversification of their food basket. There has been a great improvement in DDS of children in project schools. The mean DDS during baseline was quite low at 5, but it has now increased to 7.6. The findings from the end-line evaluation reveal that consumption of legumes and beans, nuts and seeds, dairy products, and eggs have increased significantly as compared to the baseline. 262 Improving knowledge around health and nutrition among students, parents, and other community members requires attention during the next award cycle. With respect to WASH, availability of water is a critical aspect that needs to be addressed to induce the adoption of hygienic practices and sustainable behaviour change. 263 School gardens established under the project have contributed to the provisioning of school lunch and improving knowledge about farming techniques among children. As per the monitoring reports, a high number of school gardens have been constructed/ rehabilitated (Annex R). However, these need to be used to impart nutrition education to students and other community members. The scarcity of water for irrigation are crucial factors hindering the full utilisation of the gardens. 264 Lao Women Union provided critical support by offering voluntary services for cooking school lunch. Additionally, they were also responsible for collecting firewood, vegetables, and fetching water. However, they feel that the incentives provided to them are not commensurate with the efforts. Given that cooks play a very important role in the provisioning of school meals, it is imperative that they are adequately trained and incentivised. The community contribution of vegetables to the school is affected during the winter months and also is contingent on the family’s economic condition and availability of a surplus. Lack of availability of water is affecting the regularity in cooking and increasing the workload of women in water scarce areas. 265 Capacity building of cooks, storekeepers, and teachers needs emphasis. While trainings on safe food preparation, storage practices, and warehouse management have been conducted, there are a number of cooks, storekeepers, and teachers who have not yet been trained. 266 Access to improved sanitation facility, improved water source, and hand-washing facility is critical for improving health and hygiene conditions. Hand-washing stations were observed in very few sample schools and less than one-third sample schools had access to improved water source. However, more than two-third of the sample schools had access to improved sanitation facilities. 267 The availability of water is a critical factor for sustaining the programme as it is closely linked to school gardens, cooking, health, and hygiene. 268 Capacity building of community members, along with members of VEDC, teachers, and other support staff on health and nutrition, programme management, record keeping, improved farming techniques, commodity management, and safe food preparation and storage is critical. The role of VEDC needs to be strengthened further in terms of enhancing community participation. 269 The monitoring mechanisms under MGD-SFP needs to be robust. It was found that schools were not maintaining records such as monthly food utilisation, the number of meals served, and the number of students receiving the meals, despite being trained and provided with registers. There is further scope for improving the formats for reporting against performance indicators of the programme. 270 Close coordination between WFP-CO with numerus actors such as government departments (MoES, MAF, MPI), bilateral and multilateral organisations (DFAT, BEQUAL, the World Bank, and UNICEF), non-governmental organizations (PI, BBM, and CRS), and local government recognized groups (VEDC and LWU) have greatly helped the programme in achieving the above mentioned results. 271 Recommendations from FY14 baseline study and FY14 mid-term evaluation have been successfully integrated into the programme to a great extent over the programme period (Annex T). Based on the USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 36 evaluation team’s qualitative discussions with stakeholders and findings from the primary survey, additional areas of improvement have been included as part of the recommendations. 272 Impact: The cost for each family towards supporting the programme was found to be lower in villages where a higher number of households support the SFP. This can be used as an example to incentivise communities to contribute. Several parents also felt that the programme has helped enhance their economic productivity. They also have additional time for leisure and recreation 273 The project’s focus on nutrition awareness also led to positive changes in dietary practices at the household level. 274 Gender equality requires further focus with respect to gender-balanced selection of cooks and storekeepers, and improved participation of women in VEDC. Having male cooks will gradually change social norms around cooking. Women in leadership roles will help in redefining the conventional gender roles that are prevalent in the society. As a result, it will help impact gender inequality in the society in the medium and long-term. 275 The programme has been able to achieve improvement in education and nutrition outcomes for girls and boys alike. 276Efficiency: The programme was efficient in terms of design and implementing activities that conformed to the educational and nutritional needs of the community. It targeted schools from the most backward districts of the country. Seeking collaboration with donors and NGOs, WFP has ensured availability of technical expertise in implementing the activities under the programme. 277 The combination of rice, lentils, fortified oil from WFP, and vegetables, eggs, and meat from the community helped in meeting the food and nutritional needs of the children. For safe storage, cooking, and maintaining hygiene, the programme supported the construction of the kitchen area, dining area, storage, hand-washing stations, and toilets with support from the community and other donors. WFP partnered with NGOs for establishing libraries and book corners in school and community book banks and also supplied books and reading materials. Improved teaching materials and instruction guides were provided to teachers for improving teaching standards. Engaging the community for constructing school infrastructure and supporting the school lunch helped in building a relationship between the school and community. 278The annual per capita expenditure per student is estimated to be US$ 16.8 which is almost equal to US$ 16.5 allocated under NSMP. 279 Sustainability: MGD-SFP has created an effective platform for creating knowledge on health and nutrition, and the importance of education. Its sustainability beyond external support depends upon GOL’s capacity to maintain and operate the programme, community support for the programme, as well as support from other partners towards sustaining the benefits of the programme. 280 The current activities like school gardens will need improved access to water for functioning throughout the year. Communities also need to be made aware of their expected roles and the in kind/ cash contribution. Policy makers and departmental officials need to be further trained. Concurrent monitoring and a robust M&E system would be critical for the sustainability of the program. 3.2. Good practices and Lessons Learned Good practice Adopting a holistic approach 281 The MGD-SFP FY14 worked upon improving the ability of children to understand, read, and speak Lao language. To achieve this, the program aimed at providing an enabling learning environment to children both at school and home. Parents were engaged in the process to enhance their understanding on importance of education which led to a greater community ownership and participation. The programme also supported activities to improve the quality of teaching instruction and supported construction/rehabilitation of classrooms and library. Further, to reduce short-term hunger among students and improve attentiveness, school meals were provisioned. This was accompanied by promotion of healthy nutritional practices, construction/rehabilitation of school infrastructure to support the provisioning of school meal, and WASH facilities to improve health and hygiene. Capacity building of stakeholders was also undertaken to enhance the effectiveness, USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 37 efficiency, and sustainability of the program. Adopting this holistic approach has led to an improved understanding of issues and familiarisation with addressing these issues among the community. A participatory approach to development 282 MGD-SFP had been designed with community participation at its core. Community contribution has been a critical input for infrastructure creation, provisioning of school meals, and maintenance of school gardens. This has instilled a sense of ownership among the community regarding the programme. An additional highlight is the increased interest among parents in their children’s education and school-related matters. 283 The end-line findings indicate that involvement of parents in literacy campaigns, reading camps, and other literacy awareness activities under the program have contributed towards improved literacy of school children. 284 Training of parents on nutrition and the importance of kitchen gardens has diversified the household food basket. Collaborating with multiple actors 285 While the program focussed on creating demand through community participation, it has aimed to address supply side issues by involving the government and preparing them to meet these demands. To do so, the program was designed to complement the government’s NSMP. MoES, PESS, and DESB have been made responsible at the national, provincial, and district levels, respectively, for implementation of program activities, monitoring, and record keeping. This has led to institutionalisation of the program. 286 The programme has worked in close coordination with bilateral and multilateral organisations such as DFAT, BEQUAL, the World Bank, and UNICEF), and non-governmental organizations such as PI, BBM, and CRS. Partnership with local government recognized groups such as VEDC and LWU have helped the programme in engaging with the community and promoting their participation. Lessons Learned 287 Given that enrolment rate and attendance are already quite high among primary school students, it would be beneficial to focus more on retaining children in school and providing support to address issues of repetition and dropout. Support towards enhancing reading abilities of children and promoting healthy learning environments are highly effective for this. 288 It was also observed that children’s performance on EGRA improves significantly in Grade 3 as compared to Grade 2. This is important for assessing children’s literacy and comprehension abilities. 289 In order to retain children in school after lunch and utilise the lunch breaks, it is important that teachers also stay back in the school. 290 It has been demonstrated that cooks play a pivotal role in the provisioning of school meals. There is a need to offer them adequate incentives in order to sustain their contribution. 291 The programme has the potential to impact long-term social relations and gender roles in the community. Sharing of cooking responsibilities by both men and women would set the right precedence for challenging the prevalent gender roles. Similarly, more women should be encouraged to take up the role of storekeepers and in leadership roles in the VEDC. 292 It was observed that availability of water is a critical factor for continuing such a programme. It is required for multiple purposes such as cleaning, cooking, washing, maintaining school gardens, and WASH. However, schools, particularly in the Northern provinces are faced with serious water scarcity. It is imperative to find solutions to address this issue. 293 It was learnt that the coverage of trainings and other capacity building efforts for all relevant stakeholders under the programme need to be expanded considerably. Capacity building is critical in order to enable the stakeholders to fulfil their responsibilities more effectively and boost participation in the programme. 294 A robust monitoring mechanism needs to be put in place. Proper records such as monthly food utilisation, the number of meals served, and the number of students receiving the meals are crucial for monitoring programme performance and measuring effectiveness. It also helps understand challenges and shortcomings better. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 38 3.3. Recommendations 295 The recommendations, rationale and proposed actions against each recommendation are presented in the table below. Table 6: Recommendations, rationale and proposed actions Sl. No. Recommendations Rationale Proposed actions Type Timeframe Priority 1 Children should not be leaving school during lunch break One of the objectives of MGD is to prevent accident related death of children many of which usually happens during lunch break when the children go unescorted. Similar to the policy on retaining children during lunch time in NSMP supported schools, a government order from MoES needs to be issued directing non-NSMP supported schools to also keep children in school during lunch break. Currently the programme mandates provisioning of school lunch after which children still have one more hour of lunch break left. Given that teachers go back home and classrooms are locked during this period, children too go back home. WFP in collaboration with BEQUAL, PI, and BBM may design extra-curricular activities for lunch break to engage the children. This will ensure that teachers as well as students stay back in school and the lunch break is utilised properly. This will also help reduce accident related death of children. Strategic Short term High 2 Improving access to water in schools Many school gardens are functioning for one semester coinciding with the rainy season. For making school gardens an effective contributor to the school meals programme it is essential that access to water is provisioned. Water will also be required for fishponds. Many schools did not have water in the toilets. Under SFP-FY17 award, WFP will support UNICEF to install 25 FGS and 25 boreholes and improve access to water. While this will take care of the schools with no water points, it will still fall short of the water requirements for all schools. For schools which are drawing water from surface water, pipelines can be drawn and connected to a pump-set irrigating the school garden. WFP with MGD or Non-MGD fund and in collaboration with other donors can install rainwater harvesting units to compensate for the groundwater drawn for school gardens. The funding can be arranged from WFP’s climate change initiatives and support from other stakeholders. Strategic Medium term High 3 Training of a group of cooks in each school Instead of resting the onus on 1-2 persons for cooking, it will be helpful if multiple groups are trained on cooking school meal. Adopting a tree structure approach, master trainers from each village should be trained in cooking. These in turn should train multiple groups in their villages. VEDC should be responsible for maintaining the roster for cooking and informing the groups. In addition, groups should also remind the ones responsible for cooking meals the subsequent day. Incentivisaton of cooks needs to be improved. Strategic Short term High USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 39 Table 6: Recommendations, rationale and proposed actions Sl. No. Recommendations Rationale Proposed actions Type Timeframe Priority 4 Strengthening gender dimensions of the project- cooking The design promotes gender inclusiveness in all its activities. However, socially defined gender roles prevail with cooking being women’s responsibility, thus adding to the drudgeries they face. The youths (both girls and boys) should be encouraged to volunteer for the school meals programme activities such as cooking, gardening, fetching water and collecting wood. This will have three benefits; 1) that it will help in breaking down the stereotyping of gender roles at the household level, 2) it will help in lowering the burden of the current cooks and bring in values of shared responsibility, and 3) it will ensure that women do not suffer any kind of drudgery resulting from such responsibility. Lao Youth Union can be engaged along with LWU for taking this initiative. Strategic Medium term High 5 Strengthening gender dimensions of the project- managing commodities While the programme did not differentiate between men and women, operationally more men are involved with commodity management (storekeeper). One of the visible reasons was an infrastructural deficiency. The infrastructural requirement demanded that warehouses were to be constructed above the ground on stilts to prevent an attack from rodents. Since nothing was mentioned about access, anything from log to bamboo ladders is being used to access the warehouse which is a risk to both men and women. But more importantly, it deters away women from handling this activity. Another plausible explanation for the prevalent male dominance in commodity management stems from the idea around traditional gender roles wherein it is a ‘man’s job; to ‘manage’ things and ‘provide’ women cooks with the ingredients for cooking. It is important that warehouses are designed in a manner that it becomes safe for both men and women. The technical specifications for constructing a warehouse maybe developed and circulated among the schools. Schools should encourage equal participation of both men and women for handling operational activities. The prevailing norms around traditional gender roles need to be changed. Operational Short term High USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 40 Table 6: Recommendations, rationale and proposed actions Sl. No. Recommendations Rationale Proposed actions Type Timeframe Priority 6 Convergence between the horizontal and vertical structures of the government MPI, MoES, and MAF need to converge their approaches for achieving the target of nutrition and education. Joint workshop of all the three ministries should be conducted by WFP and a roadmap should be presented by MoES for successful transition from SFP to NSMP. The expectations of each ministry should be clearly articulated. The ministries should conduct workshops on expected roles and responsibilities for their respective provincials and district departments. The third step would be a workshop at the provincial level and district levels where officials from these departments should collaborate for the future course of action. The successful handover of SFP to NSMP is highly dependent on the availability of funds for the programme. MoES through MPI has to ensure that funds are disbursed timely to the schools for effective and efficient implementation of SFP after the handover. Strategic Medium term High 7 Harmonisation of MAF’s technical support to school gardens and crop diversification for farmers MAF is providing technical support for school gardens. Separately, MAF is also working on various initiatives on promoting crop diversification. If both these activities are conducted in common villages, it will help in ensuring sufficiency of vegetables for school meals. MAF is providing technical inputs to ‘School Garden Training Guidelines’ and is expected to conduct district and school￾level training on school gardens and greenhouses. DAFO officers are expected to visit communities on a regular basis. Additionally, WFP and MoES should collaborate with MAF and identify schools and areas which are progressive in nature and appreciate crop and food diversity. In these schools and areas, pilot agriculture projects should be initiated. This will enable peer and cross learning. Strategic Medium term Medium 8 Capacity building of all stakeholders to deepen their understanding around community led school feeding programme. While training and capacity building activities have been undertaken for VEDC members, government officials, cooks, storekeepers, and teachers, it needs to be followed up with refresher training. Additional training is required on the handover process, roles and responsibilities, and on thematic areas of education, health, hygiene, and nutrition. For a successful handover of the SFP, it will be essential that all stakeholders are aligned with the programme objectives and have clarity on their roles and responsibilities. Officials from MoES, PESS, and DESB who will be responsible for school lunch should be trained together for at least two sessions for clarity of roles and responsibilities. DESB members should then train VEDC members, school teachers, cooks and storekeepers on implementing the programme effectively. DESB members with support from VEDC should train the community on their support for successful implementation of the programme. In addition teachers, USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 41 Table 6: Recommendations, rationale and proposed actions Sl. No. Recommendations Rationale Proposed actions Type Timeframe Priority cooks and communities should be trained repeatedly on issues of hygiene, education, health, and nutrition. 9 Strengthening of M&E systems A strong M&E system will measure the programme performance and will enable to undertake course corrections. Maintaining school records on enrolment, attendance, dropout, and repetitions should be made mandatory. This information should feed into government data systems. This will help in standardisation of the targets achieved. However, limited government records have been found to be a common feature across many of the South East Asian countries. In order to improve maintaining of school records, further advocacy and collaboration with the education sector is needed. A template to capture teacher’s attendance and details of school lunch should be submitted by schools to DESB each month. PESS should compile all DESB data and submit it to MoES. The data should be validated either by LWU or Lao Front for National Development and counter signed before it submission to DESB. Operational Short term High USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 42 Annexes Annex A: Scope of Work for End-Line Evaluation Table 7: Scope of Work for End-Line Evaluation Scope End-line evaluation FY14-16 Evaluation coverage The evaluation will cover the WFP Lao School Feeding USDA McGovern-Dole FY14, including all activities and processes related to its formulation, implementation, resourcing, monitoring, evaluation, and reporting relevant to answer the evaluation questions. Expectation Expected to provide an evidence-based, independent assessment of performance of the operation so that WFP and program partners can adjust course as necessary for the remainder of the program term and to inform any future program design. Areas to be covered during Evaluation Phongsaly, Oudomxay, LuangNamtha, LuangPrabang, Saravane, Sekong and Attapeu Focus of end-line evaluation The end-line evaluation and baseline will focus primarily on the following three activities, throughout which Gender Equality and the Empowerment of Women (GEEW) will be fully mainstreamed: Activity 1: Review of relevant documents including project documents, internal/external administrative records, collected data, monitoring reports and Project-Level Results Framework; Activity 2: Field visits to WFP school feeding sites to conduct surveys and interviews with focus groups at the village level; Activity 3: Interviews with representatives and staff members of governmental implementing partners, as well as interviews with community participants impacted by the project. Theory of Change Using Theory Of Change from Mid Term Evaluation, based on the Results Framework of MGD FY14-16 Gender Analysis Analysis not available, although WFP Lao had successfully completed the WFP Gender Transformation Programme Results Framework McGovern-Dole’s SO 1: Improved literacy of school age children Results level: 1.1 (Improved Quality of Literacy Instruction), 1.2 (Improved Attentiveness), 1.3 (Improved Student Attendance). The activities are designed to achieve results: 1.1.1 (Consistent Teacher Attendance), 1.1.2 (Better Access to School Supplies & Materials), 1.1.4 (Increased Skills & Knowledge of Teachers), 1.1.5 (Increased Skills and Knowledge of School Administrators), 1.2.1 (Reduced Short Term Hunger), 1.2.1.1 (Increased Access to Food). While result 1.1.3 (Improved Literacy Instructional Materials) will be achieved through partners. In addition, the project will contribute towards achieving results 1.3.1 (Increased Economic & Cultural Incentives), 1.3.2 (Reduced Health Related Absences), 1.3.3 (Improved School Infrastructure), USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 43 Scope End-line evaluation FY14-16 1.3.4 (Increased Student Enrolment), and 1.3.5 (Increased Community Understanding of Benefits of Education) As well as the foundational results: 1.4.1 (Increased Capacity of Government Institutions), 1.4.2 (Improved Policy and Regulatory Framework), 1.4.3 (Increased Government Support), and 1.4.4 (Increased Engagement of Local and Community Groups). McGovern-Dole’s SO 2: Increased Use of Health and Dietary Practices Results level: 2.1 (Improved Knowledge of Health and Hygiene Practices), 2.2 (Increased Knowledge of Safe Food Prep and Storage Practices), 2.3 (Increased Knowledge of Nutrition), and 2.6 (Increased Access to Requisite Food Prep and Storage Tools and Equipment). The program also contributes to 2.4 (Increased access to clean water and sanitation services), while intermediate result 2.5 (access to preventative health interventions) is addressed by partners. Partnership to achieve MGD results Ministry of Education and Sports, World Bank, Australian DFAT, UNFPA, UNICEF, FAO USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 44 Annex B: Stakeholders and Report Users Table 8: Stakeholder mapping Stakeholder Interest in the [Intervention/Project/Operatio n] Involvement in Evaluation and likely use Who (specifically for the Evaluation) Internal (WFP) stakeholders WFP CO Lao PDR Responsible for decision making on the CSP implementation It has a direct stake in the evaluation and an interest in learning from experience to inform decision-making. It is also called upon to account internally as well as to its beneficiaries and partners for performance and results of its operation. Liaison with stakeholders outside and inside country Responsible for design of next cycle of USDA Mc Govern Dole The key informant, primary stakeholders, and users of this evaluation. Involved in using evaluation findings in deciding on next cycle of USDA Mc Govern Dole Initial briefing and overview of WFP work in Lao PDR, programme documents, help evaluation team better understand the context of implementation and strategy for future; Support the evaluation team through an introduction to key stakeholders; Review the Inception Report and Draft Evaluation Report  CD- Sarah Gordon Gibson  DCD- Hakan Tongul  Head of Programme- Nanna Skau  Utomo Tjipto (Evaluation Manager, M&E officer)  Strategic Outcome and Activity Managers of SO1- Serena Mithbaokar  Programme Policy Officer (School Lunch)- Air Sensomphone  National Officer (Government Partnership)- Yangxia Lee  Nutrition Officer- Khizar Ashraf  Agriculture for Nutrition Officer￾OuthaiSihalath  M&E officer- Phetsamone Southalack  Head of Supply Chain- Kevin Howley A workshop with the stakeholders has been conducted during the scoping visit. Further interviews with the stakeholders will be carried out during the field mission. WFP Field Offices LuangNamtha, Oudomxay, Pakse Responsible for day-to-day CSP implementation Liaise with stakeholders at subnational levels Direct contact with beneficiaries The key informant, primary stakeholders. As users of evaluation will be affected by outcomes. Head of Field Offices, Programme Assistants, Field teams Stakeholders in LuangNamtha and Oudomxay will be interviewed RBB Responsible for both oversight of COs and technical guidance and support, the RB management has an interest in an independent/impartial account of the operational performance. The Regional Evaluation Officer supports CO/RB Key informant and primary stakeholder – involved in planning for next USDA Mc Govern Dole Interested in the independent account of USDA MGD performance and  Jennifer Shin  Yumiko Kanemitsu  Laura de Franchis  DomagojVrbos Based on availability, discussions will be held with the stakeholders USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 45 Stakeholder Interest in the [Intervention/Project/Operatio n] Involvement in Evaluation and likely use Who (specifically for the Evaluation) Applying to learn from evaluation to other country offices. WFP HQ Policy and Programme WFP HQ technical units are responsible for issuing and overseeing the rollout of normative guidance on corporate programme themes, activities and modalities, as well as of overarching corporate policies and strategies. They also have an interest in the lessons that emerge from evaluations, as many may have relevance beyond the geographical area of focus. No interviews will be conducted WFP Executive Board (EB) The WFP governing body has an interest in being informed about the effectiveness of WFP operations. The WFP governing body has an interest in being informed about the effectiveness of WFP operations. This evaluation will not be presented to the EB but its findings may feed into annual syntheses and into corporate learning processes. Office of Evaluation (OEV) Provides indirect, independent oversight DE Help Desk No interviews will be conducted External stakeholders School Meals beneficiaries They are the ultimate recipients of WFP support beneficiaries and therefore, have a stake in WFP determining whether its assistance is appropriate and effective. As such, the level of participation in the review of women, men, boys, and girls from different groups will be determined and their respective perspectives will be sought. Key informants and primary stakeholder – providing perspective on results, outcomes and emerging impact of WFP’s intervention Will be affected by the decision to continue or to stop the intervention A sample of schools to be selected and within each school, a sample of the following persons will be interviewed:  School-going children in primary schools (equal number of boys and girls)  Teachers, Cooks, Storekeeper, Administrator  Parents (mothers and fathers), Village Education Development Committee member (VEDC), Farmer Groups  Lao Women’s Union Wherever possible equal numbers of women and men will be consulted Individual and group interviews (structured and semi-structured) Government of Lao Ministry of Education and Sports Responsible for providing guidance on School Feeding priorities and approaches, and Key informant and primary stakeholder on government policy,  Director General of the Ministry of Education and Sports- Dr. Mithong Souvanvixay, USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 46 Stakeholder Interest in the [Intervention/Project/Operatio n] Involvement in Evaluation and likely use Who (specifically for the Evaluation) Ministry of Agriculture and Forest Ministry of health ensuring alignment with Government policy. priorities, views on support by WFP and on expanding school feeding, on GoL’s commitment to providing/mobilizing resources, issues, and opportunities in handing over of the programme, capacities, and convergence to be explored. Will have perspective on next USDA Mc Govern Dole cycle. Lessons learnt from this end-line DE.  Deputy Director General, Early Childhood Education- Mdm Siphaphone Manivanh  Director and Director Additional of Ministry of Agriculture and Forest Individual interviews Education authorities and province and district level (Govt of Lao PDR’s Provincial Education & Sports Services and District Education & Sports Bureau) Responsible for overseeing education sector performance, including implementation of National School Meals Policy, and liaising with other government departments at a decentralized level Primary stakeholder and key informant – on the implementation of the school feeding components. Will have perspective on challenges and achievements. May assist in facilitating field visits.  Provincial Education and Sports Services  District Education and Sports Bureau Group interviews UN country team UNICEF Involved in WASH policy and implementation Secondary stakeholder – on overall context and delivering as one. The user of DE for lessons learnt purposes Bishnu Timilsina, Chief of WASH UNICEF Will be interviewed as per availability UNFPA Adolescent girls in education to reduce early marriages Will be interviewed as per availability DONOR agency USDA Funder of MGD-SFP Primary stakeholder and informant - on the relationship with WFP and the priorities moving forward  Ada Ihanachor, Programme Analyst FAS/USDA  Traci Johnson, M&E staff FAS￾USDA Telephonic discussion as per availability NGOs USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 47 Stakeholder Interest in the [Intervention/Project/Operatio n] Involvement in Evaluation and likely use Who (specifically for the Evaluation) PLAN International, Big Brothers Mouse, SVN NGOs are WFP’s partners for the implementation of some activities while at the same time having their own interventions. The results of the evaluation might affect future implementation modalities, strategic orientations, and partnerships. Individual interviews as per availability Others WFP-Lao PDR also has established partnerships with the World Bank, Australian DFAT, UNFPA, UNICEF, and FAO to achieve project objectives. Their respective perspectives will be sought as the engagement of these actors influences the effectiveness of the programme as well as its sustainability. Relevant stakeholders from World Bank and Australian DFAT will be interviewed as per availability USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 48 Annex C: Map of the Intervention Area USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 49 Annex D: List of Districts under MGD FY14 Table 9: SFP’s coverage by provinces and districts S.No Province District 1 Phongsaly Phongsaly 2 Mai 3 Samphanh 4 Bounneau 5 Ngotou 6 Bountai 7 LuangNamtha Namtha 8 Sing 9 Long 10 Viengpoukha 11 Nalae 12 Oudomxay Xay 13 Namor 14 Nga 15 Beng 16 Houn 17 Parkbeng 18 LuangPrabang Ngoi 19 Phonethong 20 Saravane Ta Oy 21 Toumlam 22 Laongarm 23 Samouay 24 Sekong Kaleum 25 Dakcheung 26 Thateng 27 Attapeu Samakhixai 28 Sanamxai 29 Sanxai 30 Phouvong 31 Khammouane Nakkai USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 50 Annex E: Planned Outcomes 1. The MGD-SFP’s activities are aligned to support two strategic objectives (SO): improved literacy of school-age children (MGD-SO1), and increased use of health and dietary practices (MGD-SO2). 10 presents the outcomes and activities under each of the SOs of the programme. The details of activities and indicators are captured in the results framework in Annex G50 comprising the logical flow of activities leading to intermediate outcomes and in turn the SOs. Table 10: Strategic objectives, expected outcomes, and activities under the MGD supported SFP Strategic Objective Expected outcomes Activities MGD SO 1: Improved Literacy of School-Age Children MGD 1.1 Improving Quality of Literacy Instruction MGD 1.2 Improving Attentiveness by reducing short-term hunger (MGD 1.2.1) and increased access to nutritious food (MGD 1.2.1.1, 1.3.1.1) MGD 1.3 Improving Student Attendance  Training for teachers and school administrators  Providing school supplies and literacy instruction materials  Providing school meals (mid-morning snack, lunch)  Establish school gardens  Provide training on food preparation and hygiene  Community awareness ofthe benefits of education  Improve school infrastructure (store, kitchen, school garden, access to water) MGD SO 2:Increased Use of Health and Dietary Practices MGD 2.1 – 2.3 Improved Knowledge of Health and Hygiene Practices, Safe Food Prep and Storage Practices, Nutrition MGD 2.4-2.6 Increased Access to Clean Water and Sanitation Services, Preventative Health Services, and Requisite Food Prep and Storage Tools and Equipment  Deliver nutrition, health and hygiene training  Provide training on safe food prep and storage practices  Provide input on nutrition into the national curriculum  Improve access to water and sanitation facilities  Training on safe food prep and storage practices to school cooks and storekeepers Foundational Results MGD 1.4.1 and 2.7.1: Increased Capacity of Government Institutions MGD 1.4.2: Improved Policy and Regulatory Framework; TA to support inclusive education center MGD 1.4.3: Increased Government Support MGD 1.4.4and 2.7.4: Increased Engagement of Local Organisations and Community Groups 2. To improve literacy, the programme aims to increase student enrolment and attendance, enhance learning outcomes and improve the nutritional status of school children and their families. The distribution of school meals has several foreseen outcomes like the alleviation of short term hunger which is likely to improve the overall attentiveness of children during classes and school days; improved nutrition status of children through provision of fortified food; and incentivise parents to send their children to school through a reduction in cost of schooling which is likely to translate into improved demand for education. 3. The activities include training on commodity management, food preparation and storage practices and better access to water and sanitation services in school, promoting safe health and 50 As per the ToR for the end-line evaluation USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 51 dietary practices. Cooks and storekeepers receive food incentives to encourage their participation in the programme. 4. The MGD-SFP also supports interventions on the supply side that aim at building school teachers’ and school administrators’ skills and capacities and improve their attendance and performance in schools. Combined with better access to supplies and materials, enhanced teaching capabilities create an enabling environment that is likely to augment regular attendance in school and improve learning outcomes for students. 5. Aligned with the National Nutrition Strategy (2016-2020), the MGD-SFP has supported the establishment of school gardens to supplement nutrition in school lunch and promote nutrition education for children. Establishment of school gardens inculcates a practice to improve a diet with home-grown foods51. Beginning the academic year 2016-17, school gardens were introduced to 504 schools. This was expanded to 940 schools in 2017-18. 6. With the capacity development of the Village Education Development Committees (VEDC), the programme also aims to engage community groups and thereby build the community’s ownership of the programme. 7. Central to the strategic objectives, outcomes and the activities are four foundational results that contribute to the higher impact results. These pertain to the strengthening of the external ecosystem vis-à-vis policy framework and capacities of the government. MGD-SFP engages in building the capacity of the government to implement and expand its National School Meals Programme (NSMP), which currently operates in five provinces and is funded through a World Bank-managed trust fund, and to eventually take over the MGD-SFP by 2021. 8. WFP implements the MGD-SFP under its own strategic objective 4 (Reduce undernutrition and break the intergeneration cycle of hunger). To align with the Government Policy on Promoting School Lunch, WFP has on a smaller scale implemented a Cash-Based Transfers (CBT) in addition to the USDA-provided food aid in 100 schools in Beng and Nalae districts. 9. Planned and updated performance indicators are presented in Table 11. 51FAO. 2010. A new deal for school gardens: FAO Rome, 2010 http://www.fao.org/docrep/013/i1689e/i1689e00.pdf USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 52 Table 11: Planned and updated performance indicators (Source: Performance Monitoring Plan, WFP 2017) Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks Develop Partnerships with Farmer Groups to Supply Food to Schools Number of public-private partnerships formed as a result of USDA assistance (Multi-focus) 50 50 50 NA for end-line evaluation Number of farmers benefiting from developed partnership to supply food to school as a result of USDA assistance 500 500 500 NA for end-line evaluation Number of students benefiting from developed partnerships to supply food to schools as a result of USDA assistance 10,000 10,000 10,000 NA for end-line evaluation Number of schools receiving food from farmer groups 40 50 60 NA for end-line evaluation Amount of food provided to schools (dollar value) from farmer groups per semester 4,000 6,000 10,000 NA for end-line evaluation Building/Rehabilitation: Kitchens Number of educational facilities (i.e. school buildings, classrooms, and latrines) rehabilitated/constructed as a result of USDA assistance (kitchens, cook areas) 379 396 564 Number of educational facilities (i.e. school buildings, classrooms, and latrines) rehabilitated/constructed as a result of USDA assistance (dining areas) NEW n/a n/a n/a 800 Added FY 17 target Building/Rehabilitation: Warehouses and Storerooms Number of educational facilities (i.e. school buildings, classrooms, and latrines) rehabilitated/constructed as a result of USDA assistance (Other 379 396 725 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 53 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks school grounds or school buildings) Establish School Gardens Number of gardens established as a result of USDA assistance 200 268 300 Number of schools with improved access to water facilities as a result of USDA assistance New n/a n/a n/a 10 Added FY 17 target Establish Fish Ponds Number of fish ponds established as a result of USDA assistance New n/a n/a n/a 10 Added FY 17 target Capacity Building: Local, regional, national level Number of national school meal sustainability workshops held as a result of USDA assistance 1 0 0 Number of study tours outside Laos as a result of USDA assistance 1 0 0 Number of exchange visits within Lao PDR as a result of USDA assistance 0 1 0 Number of national school meal review workshops held as a result of USDA assistance 0 0 1 Raising Awareness on the Importance of Education Number of TV and Radio programs created nationwide as a result of USDA assistance 1 1 1 0 Removed and changed all years to 0 Number of awareness raising posters distributed as a result of USDA assistance 5,000 0 0 Training: Good Health and Nutrition Practices Number of training and workshops provided on good health and nutrition as a result of USDA assistance 59 83 83 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 54 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks Number of individuals trained in child health and nutrition as a result of USDA assistance (male) 4,867 6,846 6,846 Number of individuals trained in child health and nutrition as a result of USDA assistance (female) 3,524 4,958 4,958 Number of schools implementing hand-washing activities/setting up hand washing stations as a result of USDA assistance New n/a n/a n/a 10 Added FY 17 target Training: Commodity Management Number of cooks and storekeepers trained in commodity management as a result of USDA assistance 4,500 0 4,500 Training: Food Preparation and Storage Practices Number of cooks and storekeepers trained in food preparation and storage practices as a result of USDA assistance 4,500 0 4,500 Establish Activities to Promote Literacy Number of students benefiting from campaign to promote literacy as a result of USDA assistance 10,000 10,000 10,000 1,068 The updated figure for FY 17 target Provide School Meals Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (male) 76,355 72,097 57,267 53,645 For FY 17- Removed 3,622 male students in 63 poor performing schools, revised FY 15-16 Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (female) 74,247 70,107 55,685 52,527 For FY 17- Removed 3,158 female students in 63 poor performing schools, updated FY 15 figure Number of daily school meals (breakfast, snack, lunch) 24,999,932 23,605,864 18,799,832 18,580,100 Using 166 school days for FY15. From FY16, using USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 55 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks provided to school-age children as a result of USDA assistance 175 school days (based on MoES's regulation) and removed students of 63 schools Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (new) 150,602 18,799 14,969 13,802 For FY 17- Removed students of 63 schools Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (continuing) 0 123,405 98,283 92,370 For FY 17- Removed students of 63 schools Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (male) 96,777 88,822 70,131 65,546 The updated figure for FY 17 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (female) 90,950 84,132 67,214 55,898 The updated figure for FY 17 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (new) 187,727 22,772 18,889 24,884 Updated figure for FY 17 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (continuing) 0 1,50,182 1,72,954 96,560 Updated figure for FY 17 Number of individuals benefiting indirectly from USDA-funded interventions 98,215 15,139 13,218 55,036 Updated figure for FY 17 Number of students enrolled in schools receiving USDA assistance (female) 87,950 81,132 64,214 52,527 Updated figure for FY 17 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 56 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks Number of students enrolled in schools receiving USDA assistance (male) 95,277 87,322 68,631 53,645 Updated figure for FY 17 Number of individuals benefiting directly from USDA-funded interventions (male) 97,357 88,972 70,197 65,671 Updated figure for FY 17 Number of individuals benefiting directly from USDA-funded interventions (female) 90,980 84,182 67,218 55,908 Updated figure for FY 17 Number of individuals benefiting directly from USDA-funded interventions (new) 188,337 22,972 18,959 24,901 Updated figure for FY 17 Number of individuals benefiting directly from USDA-funded interventions (continuing) 0 150,182 118,456 96,678 Number of students regularly (80%) attending USDA supported classrooms/schools (male) 47,639 48,027 41,179 26,823 Updated figure for FY 17 Number of students regularly (80%) attending USDA supported classrooms/schools (female) 39,578 40,566 35,318 23,637 Updated figure for FY 17 Take Home Rations Number of individuals receiving take-home rations as a result of USDA assistance (male) 18,922 15,225 11,364 1,446 WFP stopped providing take-home-rations to secondary and primary students in September 2015. The amount distributed beforehand would cover the 1st semester started in Sep USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 57 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks 2015. Only cooks (1 cook covers 50 students) and storekeeper (one per school) receive THR afterwards. The updated figure for FY 17 Number of individuals receiving take-home rations as a result of USDA assistance (female) 13,703 11,025 8,229 3,371 Updated figure for FY 17 Number of individuals receiving take-home rations as a result of USDA assistance (new) 32,625 5,940 4,435 0 Number of individuals receiving take-home rations as a result of USDA assistance (continuing) 0 20,310 15,158 4,817 Number of take-home rations provided as a result of USDA assistance 74,250 61,500 48,186 9,634 Provide: Energy Saving Stoves Number of schools receiving energy saving stoves as a result of USDA assistance 200 268 300 100 The currently available stoves are not for institutional cooking, so each school will receive 3 stoves on average. Develop Partnerships with Farmer Groups to Supply Food to Schools Number of public-private partnerships formed as a result of USDA assistance (Multi-focus) 50 50 50 Number of farmers benefiting from developed partnership to supply food to school as a result of USDA assistance 500 500 500 Number of students benefiting from developed partnerships to 10,000 10,000 10,000 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 58 Updated Performance Indicators (Activities): WFP Lao PDR, MGD FY 2014 Award Original Requested change Activity Indicator Target for FY 2015 Target for FY 2016 Target for FY 2017 Target for FY 2017 Remarks supply food to schools as a result of USDA assistance Number of schools receiving food from farmer groups 40 50 60 Amount of food provided to schools (dollar value) from farmer groups per semester 4,000 6,000 10,000 Production of Books and Supplementary Reading Materials Number of textbooks and other teaching and learning materials provided as a result of USDA assistance 0 10,000 20,000 Enrolment Campaigns Number of enrolment campaign events held as a result of USDA assistance 1 1 1 Enrolment rate is already high in Laos (97%) and the government's interest is more on reducing drop out and repetition rates. Removed USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 59 Annex F: Break-up of Contribution from Non-USDA Funding for WFP School Feeding Programme 1 USDA has allocated $27 million to MGD-SFP FY14. Additional support is provided by Australia, Cuba, Japan Association for the World Food Programme, and Yum! Brands Inc. As per the figures in the MTE report, contributions received for the overall MGD-SFP as of May 2016 was US$ 45,958,344, which is 100 percent against the total requirements. This evaluation will only evaluate USDA’s funding for the programme. A break-up of the contributions is provided in the box below. 2 As noted in the MTE, the MGD-SFP has paid explicit attention, both in design and in implementation, to ensure complementarity and avoiding duplication with related programmes. The MGD operation is deliberately targeted to areas not served by other SF programmes, and potential duplication has been addressed during implementation. While the World Bank funded NSMP and the Catholic Relief Services (CRS) managed programme funded by a separate MGD grant have the same design, their instruments of implementation are widely different from that of MGD-SFP. In addition to funding from USDA’s McGovern Dole program, WFP expects to receive the following from non-MGD donors:  Multi-year cash contributions from private sector partners (est. US$ 980,000 annually): Several private sector partners provide contributions to school meals and nutrition activities promoting access to nutrition dense foods for children below 2 years old: Yum! Brands, from 2016 to 2018 for three years, provides an annual US$ 500,00052 contribution to the school meals programme; Yum! Australia has given a two-year US$ 400,000 contribution to nutrition education campaigns under WFP’s nutrition programme and JAWFP has contributed an estimated US$ 280,000 contribution to school meals which will support in the implementation of the cash-based transfers in schools under WFP school meals programme.  Technical assistance from WFP Brazil Centre of Excellence Against Hunger: The Government of Brazil has committed to support the Government of Laos in implementing its ‘Plan of Action on Promoting School Lunch 2016-2020’, including technical support on social safety nets.  The Global Agriculture and Food Security Program (GAFSP): This trust fund will carry on over 2016 to 2022 for a sum totalling US$ 30,000,000. WFP as a recipient entity is managing US$ 6,000,000 for technical assistance to improve food security and nutrition in Laos. There is some overlap here with WFP school meals activities, particularly through measures to encourage local agriculture and introduce dietary diversity for local populations. In addition to these contributions, the governments of France, Germany, Japan, and Luxembourg are committed to providing funds for emergency response, in the event of a natural disaster. This would include emergency provisions of school meals to the school age children affected by natural disasters. Source: WFP FY17 MGD Proposal, Lao PDR 52US$ 500,000 is the minimum commitment. Depending on the result of fund raising, Yum! Brands would contribute up to US$ 1,000,000 annually. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 60 Annex G: Project-Level Results Framework USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 61 Amendment Update USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 62 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 63 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 64 Annex H: Theory of Change The ToC presented here was developed during the MTE of the programme. Figure 6: MGD Laos Implicit Theory of Change USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 65 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 66 Assumptions and risks MTE finding indications Red font indicates assumption is not valid Orange font indicates assumption is problematic Green font indicates assumption is valid End-line evaluation finding indications Red font indicates assumption is at risk Orange font indicates assumption needs more focus Green font indicates the assumption is valid and okay Table 12: Assumptions and risks Sl. No. Assumption MTE findings End-line evaluation findings 1 Partners and contributors adequately providing effective complementary activities / inputs (e.g. WASH – MoH, Plan; Literacy/education activities – MoES, Big Brother Mouse, STC, World Bank, PLAN (new with WFP); Deworming – MoH, WHO; MNCH – WFP, agriculture support to schools – MAF, FAO; institutional strengthening - CRS, DFAT, FAO) Some activities are more effective than others and coverage is an issue. For example, literacy and education activities benefit from being linked to the BEQUAL consortium. WASH activities are also effective where they occur. School gardens face significant challenges due to water and management issues. All these activities face problems of achieving sufficient scale. Stakeholders involved in systems strengthening collaborate well but, in addition to their own internal funding and management constraints, face the challenge of a lack of government budget and ineffective monitoring data when using government staff. There has been a progress since MTE. Programme partners and other stakeholders are adequately providing support to the programme. GoL is gearing up towards accepting all the WFP supported schools post 2021. The only constraint is that of provisioning of access to water in schools. This is required for supporting the school gardens and fishponds and also in toilets and hand washing stations. 2 The programme links at many points with the inputs and activities of other donors. It was assumed that these other donors would maintain a strong, cooperative, co-ordinated presence. Within the sphere of school feeding this assumption is valid since the Technical Working Group is an effective forum for WFP, World Bank and to some extent DFAT etc.to coordinate. WFP’s linkages with other donor agencies through various platforms contributed towards meeting the SFP’s target 3 With important roles assigned to NGOs in programme design, another notable assumption is that the NGOs in question, and by extension all relevant elements of the NGO sector, are trusted partners, adequately capacitated and institutionally stable. This assumption is valid in the sense that WFP’s NGOs partners do seem institutionally stable and have appropriate technical and managerial skills, however, they face coverage constraints rooted in numbers of staff and budget resources. In addition, the NGOs’ MOU(s) and ongoing Government conservatism around civil society severely constrain changes - like scaling up or geographic enlargement and makes the process very time consuming. NGO’s are expected to support with training and providing teaching and instructional materials, books and supporting community-based literacy activities. The NGOs are well entrenched to continue with these supports and government conservatism should not hamper the progress. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 67 Sl. No. Assumption MTE findings End-line evaluation findings 4 There would be continued government support for a national school feeding programme. The Government policy framework and the presence of a school feeding unit in the MoES reflect government support, but overall, the budget is lacking and is constrained by competing priorities. The remaining lead time is 3 years until the 2020 Policy target date for budgets and processes, may prove to be insufficient. GoL is committed towards Schools meals Programme. The current support from World Bank and MGD did not necessitate the creation of a budget line for School Meals. GoL is also cognisant of the donor support required in the education sector because of the limited financial resources. While it is desirable that there is a budget line but if GoL can seek funds from other donors till it can support NSMP from its coffers will also ensure the continuity of the programme. 5 Since the government has a School Meals Policy (also a part of the convergence approach101 of the NNS), the Government is willing to commit funding to school feeding Related to #4 above. To date, this assumption is not valid since Government has not committed any budget for school feeding so far and relies on World Bank funds for the WFP handed over” schools. Discussions with policy department officials suggest that they are getting trained on continuing SFP in MGD supported schools post 2021. However, as of now the line of sight in the vertical and horizontal structure is unclear and is expected to be clearer over the next 3 years. 6 The viability of the programme depends on the activities around systems strengthening at all levels (national, Provincial, district, school), as reflected in the Foundational Results. These receive sufficient attention and are implemented as thoroughly as the rest of the programme. It is valid that the viability of the programme depends on systems strengthening and this is recognised by WFP as evidenced by an increased emphasis in the last two years. However, the current human resource model used to support systems strengthening by WFP at the grassroots level is not adequate. WFP did provide training to stakeholders at the grassroots level. However, it has proved to be inadequate as few of the trained people did not continue their association with the programme and there was a limited scope of refresher training. Also, the shifting from MMS to School Lunch forced WFP to change its approach. WFP should use the ToT approach to train grassroots level stakeholders that will allow repeated training and new training at the village level without depending on any institution. 7 Corresponding to the programme’s design emphasis on upgrading monitoring and reporting systems associated with SF, the ToC notes the assumption that the improved systems are adopted and used efficiently Activities to improve monitoring and reporting and collaboration with MoES are still under way and WFP and MoES are discussing different models of support to district staff to generate quality data. But this MTE's has found the current quality of data unsatisfactory. Cohesive monitoring and reporting are required at all levels of the governance structure. Uniform data management from schools to DESB to PESS to MoES and Donor agencies needs to be established. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 68 Sl. No. Assumption MTE findings End-line evaluation findings 8 It had to be assumed that there would be an adequate response to natural disasters. This assumption remains valid. WFP is well versed in handling such emergencies. WFP’s CSP has building resilience against climate change and disasters as one of the objectives. Aligned to that is the establishment of greenhouses that will protect the school's gardens from excessive rains. 9 The much global debate about SF has concerned the causal links between school meals and enhanced academic performance, as well as actual attendance at school. For this programme, an obvious basic assumption was that its causal assumptions about the influence of SF and related measures on student attentiveness and attendance (and literacy) are correct in the local context. An impact assessment found that schools receiving the WFP school feeding programme performed slightly better than control schools across all educational indicators except for enrolment. Pupils in schools receiving lunch were less likely to be absent at both at morning and afternoon rollcalls than pupils in schools that were receiving MMS. Girls in lunch schools were also more likely to be reported more attentive than girls in MMS schools, although no difference was seen among boys. Literacy was not an indicator of this impact assessment but is included in the baseline and end-line surveys for the MGD project. There is an onus on the project to produce evidence that will speak to this assumption. MGD-SFP is being implemented in the districts ranked lowest on BEQUAL assessment. Hence, the performance assessed vis-à-vis the control schools (Baseline FY17) suggests that there has been an improvement in the education indicators. The community literacy activities have also positively influenced the importance of education and nutrition. Importantly, children now have dietary diversity and that is pushing changes in food habits at home. Currently, the children are going back home after lunch and this needs to be stopped by promoting extra￾curricular activities for children after lunch and participation from teachers in school lunch programme. 10 For assumption 9 to be correct, it is assumed that there are adequate staff and equipment and structures in schools to support learning The Education for All National Review 2015 found that Lao PDR is able to improve the basic structure of education sector such as access to education, gender equality, school rehabilitation and construction, support for school facilities (classrooms, dormitories, water, and sanitation), expansion of the school grants and stipend scheme, increments teachers’ salaries and incentives and so on. However, there are still major issues with the quality of teaching. In many of the schools, students from different grades are sharing a room. That needs to be changed for increasing quality of education. Improved teaching and instruction materials have been provided and teachers trained. However, not all the teachers have received these improved materials and need to be provided along with training. 11 The programme design noted the problems of staff turnover and (re)deployment. The ToC, therefore, notes the assumption that these personnel changes will not beat a level that diminishes the effectiveness of staff and institutional capacity development. Staff turnover and deployment was not cited in project documents as an issue in monitoring reports or in ET interviews and FGDs Discussions with stakeholders at different levels did not highlight this as a challenge. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 69 Sl. No. Assumption MTE findings End-line evaluation findings 12 Teachers want to be trained, engage and benefit Key informant interviews confirm this assumption is valid. Teachers have been trained but refresher courses and training for new teachers should be included as part of the training process. 13 Training activities are designed to support learning principles (theoretical and practical methodologies with sufficient mentoring and supportive supervision) This assumption is partially valid. Training on cooking is a mix of practical and theoretical but is less geared towards sub￾optimal conditions e.g. how to maintain flavour and nutrient value whenever the food contributions from the community are sparse. Training on school gardens currently offers limited application to nutrition education. Mentoring and supportive supervision is the responsibility of MAF technical staff and has been weak due to lack of travel funds and often the approach/skills of extension staff. The approach to the training of cooks needs to be changed. With multiple cooks associated with the school meals programme, it is imperative to train them all. As highlighted in assumption 6, ToT approach should be adopted for training. Mentoring and supportive supervision from MAF in school gardens are still weak. Close engagement with MAF on school gardens along with kitchen gardens will give the desired impetus to the success of the programme. 14 Parents and other local community members, particularly members of VEDCs, are willing to perform the roles that the programme envisages for them. VEDCs badly need better support to grow into the roles designed for them – i.e. collaborating with teachers and “owning” local education and school feeding. VEDC institutional culture currently provides members with a certain local status, but engagement focuses on ‘easier’ tasks such as identifying school-age children in the village. On the seemingly straightforward issue of availability and motivation of cooks (usually mothers) consistently providing meals, the variability points to the absence of a common approach that would exist with better support. While VEDCs have a role to play, the governance structure of the country puts the Naiban as the decision maker at the village level. It is necessary that Naiban is definitely trained along with other VEDC members. Again, ToT training approach should be adopted so that all members get trained directly by MoES or through master trainers (trained by MoES). Training of all VEDC members will also ensure the training of women (teachers, LWU representatives and parents). 15 The WFP/ MoES relationship reflects/supports the effectiveness of MoES role in school feeding WFP and MoES need a regular and collegial contact at all levels of the MoES system to maintain their relationship in favour of the effectiveness of the school feeding programme. Engagement constraints at Provincial and district level are exemplified by the monitoring function temporarily being removed from government staff to try and provide better quality data for WFP’s donors WFP is working in close coordination with MoES and augurs well for the sustainability of the programme. That WFP will continue to provide technical support to MoES post 2021, also highlights the nature of engagement between the institutions. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 70 Annex I: Role of WFP's partners in MGD SFP Activity Implemented by Partners Role of Partners Provide School Meals WFP and EDF MoES  Training on school lunch  Mediation for resolving issues Support School Agriculture WFP and EDF (Khammouane Province) MAF, FAO, IFAD  Developing ‘School Garden Training Guidelines’  Training of VEDC members  Support development of national curriculum on school agriculture Increased Access to Clean Water and Sanitation Services WFP, UNICEF MoES, Nam Saat53,  Improve access to water in 240 schools  Construction of 25 FGS54 and 20 boreholes in identified schools  Establish 100 handwashing stations  Hand washing campaigns in 100 schools  Training to cooks on hygiene Provide Incentives and Training to School Cooks and Storekeepers WFP Communiti es  Distribution of labour for sustaining SFP beyond 2021  Ownership of cooking activities  Ownership of managing store and supply Improve School Infrastructure WFP MoES, World Bank  The mobilisation of workforce and resources for the construction of storeroom, kitchen, dining rooms etc.  Promotion of smoke reducing stoves  Assessment of the readiness of the communities to manage SFP Curriculum Development on Nutrition and School Agriculture WFP, BBM MoES, BEQUAL Consortium , FAO  Integrate nutrition and school agriculture in the school curriculum  Integration of math concepts with school gardening  Writing a book on nutrition and school garden Increased Access to Books for Schoolchildren WFP, PI, BBM MoES  Provision of quality reading materials to 40 schools (PI)  Formation of child clubs (PI)  Initiate and support “reading buddies approach” (PI)  Provision of quality reading materials to 310 schools (BBM)  Promotion of “book parties” concept  Writing a book on nutrition and school garden (BBM) Improved Literacy Instruction Materials BBM MoES  Supply of teaching instruction materials for improving Lao teaching skills to non-Lao speaking children  Supply of reading materials to improve teaching  Train teachers on implementing ‘reading time’ Provide Training on Literacy Instruction to Teachers PI, BBM MoES  Implement accelerated oral Lao language course  Training of Pedagogical Advisors from DESB and grade 1 teachers  Train teachers on implementing ‘reading time’ Increase the Engagement of School Management and Community on Literacy and Importance of Education PI, BBM MoES, USAID  Promotion of reading habit (PI)  Workshop for teachers and DESB staff for promoting reading festivals and camps  Organising parenting awareness sessions for promoting reading at home  Encouraging the additional purchase of books by parents Capacity Development WFP MoES, MAF, VEDC, Lao Women Union, CRS  Training at village level for ownership of SFP  Community training on greenhouse food production  Exposure visits for National Assembly members to enhance their understanding of school feeding, school agriculture, and nutrition  Training for MoES, MoH, MAF provincial and district level staff for enabling them to provide technical assistance to communities 53 Nam Saat under the Department of Hygiene in the Ministry of Health (MoH) is responsible for rural water supply services and the promotion of sanitation and hygiene in both urban and rural areas of Laos. 54 Gravity Fed Water Supply Schemes USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 71 Annex J: Status of Poverty, Nutrition and Education in Lao PDR, 2012 Poverty Gap Index Source: Lao PDR 2015 Census-Based Poverty Map USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 72 Food and Nutrition Security Source: Food and Nutrition Security Atlas of LAO PDR USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 73 Net Enrolment in Primary Grades (%) Source: Food and Nutrition Security Atlas of LAO PDR USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 74 Annex K: ODA Assistance to Lao PDR LAO PDR has been the beneficiary of economic assistance from development partners and over the years, this assistance has increased to support GoL in achieving the goals targeted under National Socio￾Economic Development Plan (NSEDP 2016 -2020). The following table presents the assistance received by Laos over 2011-2015 period55. Table 13: Comparison of Planned and Actual ODA disbursements from 2011 to 2015 Year Planned Disbursements Actual Disbursements 2011 139 205 2012 412 452 2013 413 455 2014 341 451 2015 161 407 Total 1,466 1,970 While Table 14 presents the planned and actual disbursements by development partners, Table 15 presents the planned and actual disbursement for 2015-2016 and planned disbursement for 2017 disaggregated by SDGs. Table 14: Disbursements by Development Partners No: Donor Planned Disbursements 2015-2016 Actual Disbursements 2015-2016 1 UNDP 11,864,687.00 14,302,069.79 2 UNICEF 1,295,971.38 1,538,281.40 3 UNFPA 2,930,312.50 3,414,315.82 4 USAID 40,114,342.00 45,735,087.23 5 WHO 9,530,633.00 9,847,020.00 6 NEDA 17,207,310.11 17,515,419.76 7 IOM 545,000.00 545,000.00 8 UNHABITAT 504,000.00 504,000.00 9 UNOCD 468,275.00 466,000.00 10 Luxdev 10,778,718.84 10,621,629.00 11 EU 20,813,243.42 19,635,724.62 12 UNIDO 959,000.00 881,324.60 13 UK 146,723.00 134,360.20 14 KFW 13,983,269.00 12,481,454.32 15 UNESCO 569,910.52 492,558.09 16 ILO 1,750,000.00 1,490,000.00 17 Finland 3,165,360.00 2,420,044.00 18 AFD 1,500,000.00 1,071,401.51 19 WFP 19,271,377.35 10,486,068.52 20 Germany 2,854,694.73 1,332,155.79 21 ADB 47,247,660.00 16,571,934.84 55Source of information for this annex: Foreign Aid Implementation Report (FAIR) Fiscal Year 2015/16, Lao PDR USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 75 No: Donor Planned Disbursements 2015-2016 Actual Disbursements 2015-2016 22 Japan 131,646,616.82 14,547,446.66 23 Australia - 34,672,682.69 24 FAO 3,221,878.77 - 26 France 1,703,110.20 - 27 IFAD - 7,347,556.41 28 SDC 22,971,614.23 - Total 367,043,707.86 228,053,535.24 Table 15: ODA projects by SDGs SDGs Number of Project 2015-2016 2017 Plan Actual Plan SDG 1 13 7,406,849.41 4,191,609.09 4,132,737.00 SDG 2 16 5,200,353.98 9,964,325.41 7,263,324.17 SDG 3 78 36,160,858.14 39,012,027.07 29,435,841.83 SDG 4 57 33,514,110.92 24,310,447.66 6,045,333.83 SDG 5 3 185,296.55 44,745.00 35,700.00 SDG 6 17 57,169,906.61 3,073,432.28 53,671,213.32 SDG 7 18 27,115,579.16 4,396,691.49 14,180,034.24 SDG 8 21 7,176,983.52 4,908,478.33 1,874,144.30 SDG 9 29 24,723,729.66 16,859,613.96 8,209,644.28 SDG 10 3 345,000.00 555,100.00 18,000.00 SDG 11 7 5,999,648.26 1,221,203.67 4,636,467.39 SDG 12 10 2,015,830.00 2,673,930.00 - SDG 13 9 3,069,462.57 707,089.79 1,882,153.57 SDG 14 6 5,780,001.00 3,621,032.58 436,000.00 SDG 15 19 7,124,918.30 4,501,938.14 3,740,661.00 SDG 16 22 3,856,678.16 4,404,677.42 2,031,921.84 SDG 17 8 6,588,160.33 469,294.09 3,556,071.52 SDG 18 11 13,582,404.20 12,345,008.15 577,467.92 Multi SDGs 84 112,915,276.70 89,850,800.11 151,704,556.73 N/A 16 7,112,660.40 942,091.03 6,565,519.20 Total 447 367,043,707.86 228,053,535.24 299,996,792.14 Infrastructure, Health, Education, Natural Resource Management and Environment, and Agriculture and Rural Development are the top five sectors that had received ODA support. As presented in Figure 7, Infrastructure sector received the highest ODA percentage of 20% of total ODA between years 2011- USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 76 2015. Other sectors received almost a quarter of ODA, including Governance, UXO and Illicit Drug Control. Figure 7: ODA disbursements by Sector Working Groups 20% 18% 16% 12% 11% 23% Infrastructure Education Agriculture and RD NRM and environment Health Others USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 77 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 78 Annex L: Evaluation Mission Schedule 1 The timelines for the end-line evaluation were discussed with WFP-CO after the scoping visit during the debriefing meeting on 16 February 2018. The revised timelines were shared in the scoping visit report which was agreed upon by WFP-CO. 2 Data collection was divided into two phases owing to the celebrations around the Lao New Year (11-17 April 2018). Phase one took place between March 26 and April 6, 2018, and the second phase was carried out between April 30 and May 11, 2018. The data collection schedule is presented in the table below: Table 16: Evaluation Mission Schedule Days Dates Activity Team member Locations/sites Stakeholders Debriefing by WFP team Day 1 29 January 2018 Debriefing of NRMC team by WFP CO WFP CO and NRMC core team Vientiane and New Delhi WFP CO and NRMC core team Literature Review and Planning The NRMC team spent 2 weeks to review some of the documents shared by WFP CO to understand the programme. In this phase, the team also planned for the scoping mission. Scoping Mission Day 1 12 February 2018 First meeting with WFP CO to discuss the programme details and understand perspectives of the WFP CO in implementation and performance of the programme. Team Leader, Evaluation Manager, and Gender Specialist Vientiane WFP CO including the Country Head, Programme Head and Evaluation Manager Day 2 and 3 13 and 14 February 2018 Visit field sites in LuangNamtha Evaluation Manager and Gender Specialist 2 programme schools in LuangNamtha School going children of grade 1 to 5, VEDC, school head, school teachers, cooks and storekeeper, observation of school lunch, WFP field office representative Meeting with the MoES Team Leader Vientiane The Deputy Director, Ministry of Education and Sports (MoES) Day 4 15 February 2018 Meeting with the MAF Team Leader and Evaluation Manager Vientiane Director and Director Assistant, Ministry of Agriculture and Forest (MAF) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 79 Days Dates Activity Team member Locations/sites Stakeholders Internal discussion within NRMC to collate the observations from the scoping mission to propose changes in the evaluation design and methodology Team Leader, Evaluation Manager, and Gender Specialist Vientiane NRMC core team Day 5 16 February Debriefing meeting with WFP CO presenting the observations from the scoping visit and discussion of changes in the proposed evaluation methodology Team Leader, Evaluation Manager, and Gender Specialist Vientiane WFP CO Meeting with Social and Behavioural Change Communication (SBCC) expert and Project Manager, Agriculture for Nutrition Project, WFP Evaluation Manager and Gender Specialist Vientiane Social and Behavioural Change Communication (SBCC) expert and Project Manager, Agriculture for Nutrition Project, WFP Preparation Phase 19 to 28 February 2018 Development of scoping mission report including proposed changes in evaluation methodology, mapping of indicators to the evaluation questions, development of quantitative and qualitative data collection tools, a literature review of programme documents Evaluation Manager and Gender Specialist, Technical inputs provided by Team Leader New Delhi NA 1 March 2018 Presentation of a revised evaluation methodology for approval to USDA. The methodology was approved on the same day. Team Leader, Evaluation Manager, and Gender Specialist New Delhi USDA, WFP CO and NRMC 26 February to 12 March 2018 Development of draft inception report Evaluation Manager and Gender Specialist, Technical inputs provided by Team Leader New Delhi NA 12 March to 19 March 2018 Preparation of field work, translation of evaluation tools, software development, the hiring of data collectors, development of field movement plan indicating movement of each team member including the data collection teams and the core teams Lao-partners (Geo-Sys), Evaluation manager Vientiane and New Delhi NA USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 80 Days Dates Activity Team member Locations/sites Stakeholders Training Day 1 to 4 19 to 22 March 2018 Simultaneous classroom training of quantitative and qualitative teams on data collection tools and ethical guidelines. Evaluation Manager and Gender Specialist and Geo-Sys Vientiane Data collection team Day 5 23 March 2018 The pilot of tools in two schools in Vientiane and debriefing with the data collection teams Evaluation Manager and Gender Specialist and Geo-Sys Vientiane School children, teachers, school head, cook and storekeeper Day 6 and 7 24 and 25 March 2018 Incorporation of changes in the tools based on the pilot by NRMC and Geo￾Sys. Data collection teams move to the field. Evaluation Manager and Gender Specialist and Geo-Sys NA NA Phase 1 of Field Mission Day 1 26 March 2018 Quantitative data collection in two schools by three teams Qualitative data collection in one school by one team Debriefing of the team at the end of field day. Data collection team. Monitoring of quantitative data by Evaluation Manager, Gender Specialist (one school each) and Field Manager from Geo-Sys. The gender specialist and researcher were present in the qualitative interviews as well. Laongam in Salavan province Children, parents, school head, school teachers, cooks, storekeeper, VEDC, farmer groups, the village head Day 2 27 March 2018 Quantitative data collection in three schools by three teams Qualitative data collection in one school by one team Debriefing of the team at the end of field day. Data collection team. Monitoring of quantitative data by Evaluation Manager, Gender Specialist and Field Manager from Geo-Sys. The Evaluation Manager and researcher were present in the qualitative interviews as well. Laongam in Salavan province and Thateng in Sekong province Day 3-5 28 to 30 March 2018 Quantitative data collection in 15 schools (both project and control Data collection team. Monitoring of data by Field Manager from Geo-Sys supported by Evaluation Manager, Lanam, and Thateng in Sekong province, Sanamxai Children, parents, school head, school teachers, cooks, USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 81 Days Dates Activity Team member Locations/sites Stakeholders schools). The qualitative team collects data across 6 schools. Gender Specialist, and researcher from NRMC and Xaixettha in Attapeu province storekeeper, VEDC, farmer groups, the village head Day 4 29 March 2018 Discussion with Government official from agriculture department at the provincial level Evaluation Manager, Gender Specialist Attapeu Head of PAFO, Attapeu Day 4 29 March 2018 Discussion with Government official from the education department at district level Evaluation Manager, Gender Specialist Xaixettha Head of DESB, Xaixettha Day 5 30 March 2018 Discussion with Government official from the education department at the provincial level Evaluation Manager, Gender Specialist, and Researcher Sekong Head of PESS and WFP Liasoning Coordinator, Sekong Day 5 30 March 2018 Discussion with Government official from agriculture department at the provincial level Evaluation Manager, Gender Specialist, and Researcher Sekong Head of PAFO, Sekong Day 6-7 31 march and 1 April 2018 Schools closed. Documentation of qualitative data. Data management and upload of quantitative data, preliminary checks on data quality and debriefing of field teams. Data collection team, Geo-sys and NRMC team NA NA Day 8-12 2 April to 6 April 2018 Quantitative data collection in 15 schools (both project and control schools). The qualitative team collects data across 5 schools. Data collection team. Monitoring of data by Field Manager from Geo-Sys, Evaluation Manager and Researcher from NRMC Vapi in Salavan province, Salavan in Salavan province and Laongam in Salavan province Day 8-12 3 April 2018 Discussion with Government official from the education department at the provincial level Evaluation Manager and Researcher Salavan Deputy Head, Academics, PESS and WFP Liasoning Coordinator, Salavan Discussion with Government official from agriculture department at provincial level Evaluation Manager and Researcher Salavan Deputy General Manager, PAFO, Salavan Discussion with Government official from agriculture department at district level Evaluation Manager and Researcher Salavan Head of DAFO, Laongam, Salavan USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 82 Days Dates Activity Team member Locations/sites Stakeholders Discussion with Government official from the education department at district level Evaluation Manager and Researcher Salavan WFP Liasoning Coordinator, DESB, Laongam, Salavan 4 April 2018 Discussion with WFP Field Office, Pakse Evaluation Manager and Researcher Pakse Head, and Project Officer, WFP Field Office, Pakse 5 April 2018 Discussion with WFP-CO, Vientiane Evaluation Manager and Researcher Vientiane WFP-CO Evaluation Manager and Program Team for MGD-SFP Break in field mission due to Lao New Year Phase 2 of Field Mission Day 0 27 April 2018 Refresher training on tools Geo-Sys Vientiane Data collection team Day 1 30 April 2018 A quantitative survey in 3 schools and qualitative survey in 1 school in Ngoy Data collection team. Monitoring of data by Field Manager from Geo-Sys, Ngoy Children, parents, school head, school teachers, cooks, storekeeper, and VEDC. Day 2-5 1 May to 4 May 2018 Travel to Oudomxay Data collection team. Monitoring of data by Evaluation Manager and Researcher from NRMC. Oudomxay Children, parents, school head, school teachers, cooks, storekeeper, VEDC and Farmer Groups. A quantitative survey in 17 schools and qualitative survey in 6 schools in Oudomxay Discussion with Government official from the education department at district level Evaluation Manager and Researcher from NRMC. Oudomxay Deputy Head, DESB, Nga, Oudomxay Discussion with Government official from agriculture department at district level Evaluation Manager and Researcher from NRMC. Oudomxay Deputy Head, DAFO, Nga, Oudomxay Discussion with Plan International Evaluation Manager and Researcher from NRMC. Oudomxay Discussion with Plan International Discussion with Government official from the education department at the provincial level Evaluation Manager and Researcher from NRMC. Oudomxay Head of Pre-Primary Education, PESS, Oudomxay Discussion with Government official from agriculture department at the provincial level Evaluation Manager and Researcher from NRMC. Oudomxay Project Coordinator, PAFO, Oudomxay USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 83 Days Dates Activity Team member Locations/sites Stakeholders Discussion with Government official from the education department at the provincial level Evaluation Manager and Researcher from NRMC. Oudomxay Head, DESB, Beng, Oudomxay Discussion with Plan International Evaluation Manager and Researcher from NRMC. Oudomxay Plan International Coordinator for Nga, Oudomxay Day 6-7 5 May to 6 May 2018 Schools closed. Documentation of qualitative data. Data management and upload of quantitative data, preliminary checks on data quality and debriefing of field teams. Travel to Phongsaly (Data collection team) Travel to LuangNamtha (NRMC team) Data Collection team, Geo-Sys and NRMC NA NA Day 8-9 7 May to 8 May 2018 A quantitative survey in 11 schools and a qualitative survey in 4 schools in Phongsaly Data collection team. Phongsaly Children, parents, school head, school teachers, cooks, storekeeper, VEDC and Farmer Groups. Day 8 7 May 2018 Discussion with Government official from the education department at district level Evaluation Manager and Researcher LuangNamtha Deputy Head, DESB, Viengphoukha, LuangNamtha Visited model school in Ban Thiao Evaluation Manager and Researcher LuangNamtha Deputy School Head Visited primary school in Nongkham village Evaluation Manager and Researcher LuangNamtha Discussion with Government official from the education department at the provincial level Evaluation Manager and Researcher LuangNamtha Coordinator for Primary School Education, PESS, LuangNamtha Day 10-12 9 May to 11 May Travel to LuangNamtha Data collection team. LuangNamtha Children, parents, school head, school teachers, cooks, storekeeper, VEDC and Farmer Groups. A quantitative survey in 11 schools and qualitative survey in 3 schools in LuangNamtha USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 84 Days Dates Activity Team member Locations/sites Stakeholders Day 10 9 May 2018 Visited project school in Nong-Gien, Ngoy district Evaluation Manager, Gender Specialist, and Researcher LuangPrabang School Head Visited project school in Huay Phouang, Ngoy district Evaluation Manager, Gender Specialist and Researcher LuangPrabang Day 11 10 May 2018 Discussion with Big Brother Mouse Evaluation Manager, Gender Specialist and Researcher LuangPrabang Director, Bog Brother Mouse, LuangPrabang Day 8-12 7 May to May 11 Meetings were held with the Government officials of the MoES, MoAF and MoH, representatives from World Bank, UNICEF and FAO. These meetings were scheduled in advance by NRMC in consultation with WFP-CO. Team Leader, Evaluation Manager from NRMC Vientiane Government officials of the MoES, MoAF and MoH, representatives from World Bank and Australian DFAT, Director Assistant MoAF, Director of Agro Processing and Agri-Business Promotion Division Day 13 May 11 Debriefing presentation to WFP-CO Team Leader, Evaluation Manager, Gender Specialist and researcher from NRMC Vientiane WFP-CO and NRMC USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 85 Annex M: Evaluation Matrix Table 17: Evaluation Matrix Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation EVALUATION CRITERIA: RELEVANCE 1. To what extent was the design of the MGD-SFP contributing to realizing the Government of Lao's National School Meal Policy and WFP's Country Strategic Plan 2017-21?  Policy and programme context of the Government of Laos and the respective provinces  Alignment of the programme design or implementation with respect to the NSMP and the WFP’s Country Strategic Plan (CSP);  The scale of activities;  Alignment with Government’s preferred mode of local procurement;  The relevance of the design in addressing varied sectors￾education, WASH, health, gender issues  Public private partnerships formed; Ownership of the communities and village level organisations  Review of the WFPs country strategic plan (2017-21)  Review of NSMP (2014)  Review of programme results framework, programme ToC  Mapping of programme’s inputs activities and outputs with goal and objectives and with National priorities and WFP’s country mandate  Workshop/ Key informant interviews (KII) with WFP programme staff, MoES, MoH, MoAF  Programme documents (Theory of change, results inthe framework, programme design)  NSMP and CSP 2017-21  WFP Programme staff, MoES, MoH, MoAF  Documents related to NSMP and WFPs country strategic plan (2017-21) are available and reliable  Public private partnerships formed: As per review of programme documents, this data may not be available.  Data from interviews with MoES, MoH, MoAF will be triangulated with that provided by WFP staff and secondary documents to assess the reliability 2. To what extent is the MGD￾SFP in line with the needs of the most vulnerable  The demographic, social and economic context of the project provinces and districts  Desk review of the context of school feeding in the project areas  Programme documentation  Project documents  Review of national school meal workshops held as a result of USDA assistance  Data on project design and gender is available USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 86 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation groups (men and women, boys and girls)? 3. To what extent were the MGD-SFP implementation tailored to local needs? 4. To what extent was MGD￾SFP programme based on a sound gender analysis? 5. To what extent was the design and implementation of the MGD-SFP programme gender￾sensitive? 6. Has the MGD-SFP programme made any difference to gender relations in the medium or longer term?  What are the key issues tackled by the programme that affect the targeted population with respect to education, nutrition, and WASH  Implementation design in light of the NSMP and WFP’s CSP and WFP’s gender policy  Infrastructure availability and access for poor and vulnerable groups  Reach of the initiative to poor and vulnerable groups, highlighting inclusion and equity  How does the design address key concerns and needs of the population including primary and secondary stakeholders (in light of capacity building, interpersonal communication and continued dialogue on nutrition, literacy, and education)  The inclusion of gender strategy in design and implementation  How is gender integrated into the process (participation of women in school meals) and as an outcome (literacy, attendance, and attentiveness of girls)  Use of secondary sources such as project baseline, midline, and other evaluations to identify the needs of target groups  KII and focus group discussions (FGD) with secondary stakeholders including parents (mothers and fathers), school staff, programme staff and Government officials  Workshop/KII with WFP programme staff  Comparison of primary data of the end-line evaluation with the mid￾term and baseline to identify the progress from the design stage.  Secondary data available in public domain including country surveys conducted by UN  Data made available by the government on the District Health Services or District Educational Services  WFP programme staff, school authorities, Government authorities and community (especially girls and women) EVALUATION CRITERIA: EFFECTIVENESS USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 87 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation 7. Do recommendations made during FY14-16 baseline and mid-term evaluation was integrated into FY14-16 programme implementation, and if so, whether these recommendations were successful in strengthening the programme?  Recommendations made during the baseline and mid-term (and evolving context of education in the country and the NSMP)  Map the recommendations and the review; examine alignment with the WFP country strategy and NSMP; assess the potential of the recommendation to strengthen the programmes vis-à-vis contextual factors and policy considerations.  Examine the changes in the indicators or addition of new or omission of indicators to the programme design specific to the recommendations and identify the trend of the indicators to assess the strength of the implementation  Review of the baseline and mid-term reports  Examine the ToC assumptions and risks  Review of the indicators and means of verification of indicators to assess the implementation changes  Primary data from the end￾line survey (quantitative and qualitative) to analyse the level of success in strengthening the programme  Project documents (baseline and mid-term reports)  A quantitative survey in schools;  FGD and KII with primary and secondary stakeholders  Based on a review of programme documents, the data is available and reliable 8. To what extent has the MGD-SFP contributed to the improved attentiveness of school age children (boys and girls) in WFP covered schools. (MGD 1.2)  Number/percent of students in classrooms identified as inattentive by their teachers; total number of children in the classroom; disaggregation by gender  Reason for inattentiveness; missed schools meals if any, and reasons to miss school meals; disaggregation by gender  Primary data from school  Comparison with monitoring data of the project  The baseline of FY 14-17  A qualitative and quantitative survey of the school staff, parents, and WFP staff  School teachers, school going children, school heads, cooks, storekeepers and parents  Attendance registers  School enrolment data  Monitoring data of the project  School teachers, school going children, school  Attentiveness: Method of data collection lacks reliability. While method entails asking two teachers about the attentiveness level of a student (indicating triangulation), the results may not truly reflect the reason for such the level of attentiveness. Thus, USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 88 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation 9. To what extent has the MGD-SFP contributed to the increased enrolment of school age children (boys and girls) in WFP covered schools. 10. To what extent has the MGD-SFP contributed to increased attendance of school age children (boys and girls) in WFP covered schools. (MGD 1.3)  Number of children receiving and consuming daily school meals; the number of daily meals provided; reasons for not receiving or consuming;  Change in attentiveness compared to a baseline of the FY14-17 programme  Number/percent of girls and boys enrolled in the school in last one year; last five years data  Gross and net enrolment ratio of districts  Reasons for change in enrolment, if any  Number of children receiving daily school meals; daily meals provided  Number of enrolment campaigns  Disaggregated data by gender  Number/percent of girls and boys who regularly attend school (at least 80 percent of school time) in last one year; Validation by parents  Primary data from school  Comparison with monitoring data of the project  Enrolment data from school  The baseline of FY 14-17  A qualitative and quantitative survey of the school staff, parents, and WFP staff  Primary data from school, parents  School attendance register of selected students heads, cooks, storekeepers and parents  Attendance registers  School enrolment data  Monitoring data of the project results cannot be attributed to the programme.  Receipt and consumption of school meals: The data availability may be limited to WFP’s monitoring data. Preliminary visits to schools suggest that schools, while having a format for filling the data, often do not complete the formats.  Enrolment: Partial data availability due to lack of maintenance of such data in all schools.  Attendance: Lack of availability and reliability of data as most schools do not record attendance on a daily basis. Use of self￾reported data from students may not be USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 89 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation  Reasons for change in attendance (school meals, teacher’s interest and improved capacity, school’s improved infrastructure, parent’s increase in interest in education, children’s improved attentiveness)  Number of children receiving daily school meals; the number of daily meals provided  The difference in enrolment between project and control areas  Disaggregated data by gender Overall: Frequency of school lunch in a school week; reasons for non￾receipt of food supplies, if any; textbooks and teaching materials provided; schools having clean space for cooking and storage of supplies; classrooms and educational facilities provided; capacity building of school teachers, cooks and storekeepers; training on health and nutrition and hygiene  The roll call for students  Self-reported data on attendance  Comparison with the baseline of FY 14-17  A qualitative and quantitative survey of the school staff, parents, and children reliable, thus triangulation with data from parents. This data will only be available for the last one week.  Given the sources of information, where possible reported data will be triangulated using information from different stakeholders (such as frequency of school meals cooked in the last week). Where possible, reported data can also be triangulated using available records. In absence of such records such as attendance, the data cannot be triangulated. 11. How do literacy and health related absence outcomes compare across the MGD-  Number of health-related absence (self-reported and validated by parents) of school going children  Primary data from school, parents  School attendance register of selected students  School teachers, school going children, school heads, cooks, storekeepers and parents  Self-reported data by students on health-related absence will be triangulated with that of USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 90 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation SFP covered schools? (MGD 2.1) 12. Is there evidence of a positive impact of the project on literacy and literacy instruction, in MGD-SFP, covered schools? (MGD 1.1) during baseline and end-line; disaggregated by sex  Literacy levels of children  The proportion of students passing grade 5  Reasons for change in literacy levels (school meals, teacher’s interest and improved capacity, school’s improved infrastructure, parent’s increase in interest in education, children’s attendance) Overall: Amount of food provided to schools; reasons for non-receipt of food supplies; quality and quantity of school supplies; textbooks and teaching materials provided; schools having clean space for cooking and storage of supplies; classrooms and educational facilities provided; capacity building of school teachers, cooks and storekeepers; training on health and nutrition and hygiene  EGRA and school tests and results for children  Comparison with monitoring data of the project  The baseline of FY 14-17  A qualitative and quantitative survey of the school staff, parents, and children  Attendance registers  Monitoring data of the project  School enrolment data  School test results of school going children parents to increase reliability;  Attendance data, if available, will be used to triangulate the data  Teachers of the students interviewed will be asked to corroborate the information provided by the student  EGRA is a tested method of assessing literacy levels. Data from EGRA will be reliable.  Data on the impact on literacy instruction is not available and cannot be observed or documented by the evaluation during the primary survey. Data on training of teachers are available. 13. What are the intended and unintended outcomes of school gardens in MGD-SFP covered schools in these 7  Number of schools with school gardens  Number of schools with school gardens that supplement school  A qualitative and quantitative survey of the school staff, parents, and children  School teachers, school going children, school heads, cooks, storekeepers and parents  Both secondary and primary data is available and reliable  Data can be triangulated based on information USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 91 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation provinces?(MGD 1.2.1.1, 1.3.1.1) 14. Do stakeholders view school gardens as a learning tool on agriculture and nutrition in MGD-SFP covered schools in these 7 provinces? 15. How often are schools utilizing produce from their school gardens, to supplement USDA donated food, in MGD-SFP covered schools in these 7 provinces?(MGD 1.2.1.1, 1.3.1.1) meals (2 harvests in a month for school lunch)  What all is grown in the school gardens; what are the harvest patterns;  Access to water for the school gardens  Knowledge of maintenance of school gardens  Do school administrators, children and parents know (awareness) and understand the role of school gardens?  What are the changes in the diet of the children in school and at home  Volunteers for school gardens;  Perceived benefits of school gardens by stakeholders  Does it add extra effort for the school staff  Do the current resources allow for the school gardens to continue without pressure on school/education budgets?  Institutional, social and policy triggers and barriers to sustaining school gardens  Qualitative interviews with farmer groups and village level education development committees  Observation of schools  Comparison with monitoring data of the project  The baseline of FY 14-17  Success stories and learning from qualitative case studies  village level education development committees  Monitoring data of the project  School budgets and resources registers received from different stakeholders and observation of schools. EVALUATION CRITERIA: EFFICIENCY USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 92 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation 16. Was the MGD-SFP implemented in the most efficient way compared to alternatives? 17. Did the targeting of the MGD-SFP mean that resources were allocated efficiently?  The technical and allocative efficiency of the programme  Leverage of Government funds and resources  Choice of resources (human, financial, infrastructure, time); Sufficiency of resources  If available, direct project costs (training, capacity building, food supplies, school supplies)  Number of schools directly or indirectly benefitted from programme activities  Whether resource utilisation was optimal or alternate choice could have been made  Qualitative data; discussions with WFP  Discussions with partners  Programme data  from intervention areas  WFP programme staff and documents  Project functionaries  Implementation Partners  Available budgets  Monitoring data  Data availability cannot be commented upon at this stage EVALUATION CRITERIA: IMPACT 18. Is there evidence that current school gardens are effective at increasing knowledge of nutrition in MGD-SFP covered schools in these 7 provinces? (MGD 2.3)  Change in dietary practices from baseline to end-line  Comparison of schools with and without school gardens  Triggers and barriers to increasing effectiveness of school gardens (capacities, institutional bearings, policy environment, support from external aid, the sufficiency of support, engagement of communities)  Qualitative and quantitative interviews  Programme data  A literature review of HGSF  School teachers, school heads, cooks, storekeepers and parents  village level education development committees  Programme staff  Monitoring data of the project  Secondary data is available and reliable  Primary data will be available in a qualitative form  Primary data and secondary data will be triangulated to arrive at the findings USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 93 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation 19. What are the gender￾specific impacts of MGD￾SFP and the related activities on girls and boys students? Did the MGD-SFP influence the gender context? Are these results likely to be sustained after completion of the project?  Above indicators/information areas disaggregated by sex  Gender participation in process and outcomes (VEDC, school cooks etc.)  Role of female teachers, cooks, storekeepers, and school heads in improving outcomes  Number of girls and boys benefitted from receipt and consumption of school meals  Increase in enrolment of girls and boys in school  Increase in attentiveness, literacy, and attendance of girls and boys going to school  Change in perception of parents with respect to girls’ education  Reasons for change in the above indicators  Dietary changes at home; treatment of boys and girls in school, at home and in community  Qualitative and quantitative interviews  Programme data  Analysis using the gender specific approach  School going children, school heads, cooks, storekeepers, and parents  village level education development committees  Programme staff  Monitoring data of the project  Both secondary and primary gender aspects is available and reliable  Quantitative and qualitative data will be triangulated to arrive at the findings EVALUATION CRITERIA: SUSTAINABILITY 20. To what extent did the MGD-SFP implementation arrangements include considerations for  Handing over plan and processes for adoption of the programme by government agencies  Review of programme documents  Qualitative interviews with WFP Programme  WFP Programme staff and schools  Data is available and reliable  Programme data, past evaluations, primary data USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 94 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation sustainability, such as capacity building of national and local government institutions, communities and other partners? 21. To what extent is it likely that the benefits of the MGD-SFP will continue after WFP’s work ceases?  Alignment of MGD-SFP with the NSMP  Which activities or outcomes continue to happen without the programme’s support? For example, improved capacities of government institutions, school teachers, cooks and store-keepers ensure better information for children and parents.  Evidence of best practices that can be replicated or adopted  The capacity building required for further scaling up/replication  Examine aspects of the programme that align with GOL’s priorities  Advocacy initiatives during the programme period  The commitment of government and school officials to take forward or adopt best practices or innovations from the initiative  Critical factors for replication in other areas; external social and cultural environment for implementation; clarity of objectives and processes; documented success stories staff and schools; government officials  Education, nutrition, health department officials  Parents, community influencers and stakeholders’ perspectives will be triangulated to arrive at the findings. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 95 Evaluation Questions Key information areas/indicators Data Collection Method/Analysis Target Groups/Sources of Information Availability and Reliability of Data and Methods of Triangulation  Issues and concerns in the sustainability of the programme without external aid  Resources required for sustenance USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 96 Table 18: Mapping of evaluation findings with end-line evaluation criteria (as outlined in the ToR) Evaluation Criteria MGD FY14-16 End-line Evaluation Questions Evaluation findings (Paragraph number) Relevance To what extent was the design of the School Meals programme contributing to realizing the Government of Lao's National School Meal Policy and WFP's Country Strategic Plan 2017-21? Para 65-70 and para 71-75 To what extent is the School Meals programme in line with the needs of the most vulnerable groups (men and women, boys and girls)? Para 61-64; and para 191-193 (under effectiveness) To what extent was the School Meals (Mid-Morning Snack and School Lunch) Programme based on a sound gender analysis? Para 77 To what extent was the design and implementation of the School Meals (Mid-Morning Snack and School Lunch) programme gender￾sensitive? Para 76, 78 and 79; and para 194-197 (under effectiveness) Has the School Meals (Mid-Morning Snack and School Lunch) programme made any difference to gender relations in the medium or longer term? This has been addressed under effectiveness (para 198-99) as the question seeks to answer how ‘effective’ various activities and achievements of the programme have been in influencing gender relation in the medium or longer term. Effectiveness Do recommendations made during FY14-16 baseline and mid-term evaluation were integrated into FY14-16 programme implementation, and if so, whether these recommendations were successful in strengthening the programme? Para 180-186 To what extend has the project contributed to improved attentiveness of school age children (boys and girls) in WFP covered schools? Para 84-90 To what extend has the project contributed to increased enrolment of school age children (boys and girls) in WFP covered schools? Para 91-93 To what extend has the project contributed to increased attendance of school age children (boys and girls) in WFP covered schools? Para 94-102; and Box 1 How do literacy and health related absence outcomes compare across the WFP covered schools? Para 103-147 Is there evidence of a positive impact of the project on literacy and literacy instruction, in WFP covered schools? Para 148-153 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 97 Evaluation Criteria MGD FY14-16 End-line Evaluation Questions Evaluation findings (Paragraph number) What are the intended and unintended outcomes of school gardens in WFP covered schools in these 7 provinces? Para 154 to 160; and Box 2 Do stakeholders view school gardens as a learning tool on agriculture and nutrition in WFP covered schools in these 7 provinces? Para 168. 189, 190 How often are schools utilizing produce from their school gardens, to supplement USDA donated food, in WFP covered schools in these 7 provinces? Para 156-160 Impact Is there evidence that current school gardens are effective at increasing knowledge of nutrition in WFP covered schools in these 7 provinces? Para 189-190 What are the gender-specific impacts of School Meals programme and the related activities on girls and boys students? Did the School Meals programme influence the gender context? Para 191-201 Efficiency Was the School Meals programme implemented in the most efficient way compared to alternatives? Para 210-215 Did the targeting of the School Meals programme mean that resources were allocated efficiently? Para 203-209 Sustainability To what extent did the School Meals Programme implementation arrangements include considerations for sustainability, such as capacity building of national and local government institutions, communities and other partners? Para 227-233 To what extent is it likely that the benefits of the School Lunch programme will continue after WFP’s work ceases? Para 217-225, and para 234-238 To what extent did the School Meals Programme implementation were tailored to local needs? Para 18-29 9(context) and para 61-64 (under relevance) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 98 Annex N: Evaluation Approach and Methodology 1. The end-line evaluation was conducted between February 2018 and June 2018. The scope of the evaluation was from September 2015 till June 2018. The data collection is scheduled between March and May 2018 due to which information on certain data points such as attendance, attentiveness, and food utilisation will cater to the school year starting in September 2017 until February 2018. Evaluation Criteria and Questions 2. The evaluation complies with the ToR in the application of the international evaluation criteria of relevance, effectiveness, efficiency, impact, and sustainability with gender equity and inclusion mainstreamed throughout. For each evaluation question, Annex M presents the key information areas, the approach to collect data, and the data source. Wherever appropriate, gender dimensions were factored into the sub-questions, judgement criteria, and indicators for each evaluation question. 3. Under the criteria of relevance, the evaluation sought to understand whether the design of the MGD-SFP was aligned with the national priorities and the WFP's Country Strategic Plan (CSP) 2012-15. In addition, the evaluation examined whether the design and implementation of the programme were relevant to the local needs of the most vulnerable groups and based on a sound gender analysis. To address the questions, the evaluation reviewed the programme’s result framework and ToC in light of the NSMP and the WFP’s CSP assessing the alignment with national priorities and WFP’s mandate. The evaluation also examined the contextual factors (social, economic, cultural and political) that influenced the programme design and its implementation and the extent to which such factors have been considered to tailor the programme design to suit the local needs (which vary by regions). The evaluation assessed the alignment of the programme design with the gender policy of the Government and that of WFP. Further, through primary and secondary data, the evaluation identified whether local gender issues have been integrated into the implementation strategy and the outcomes of these strategies. 4. To assess the effectiveness of the programme, the evaluation sought to establish the end-line values of literacy (MGD 1.1), attendance (MGD1.3), attentiveness (MGD 1.2), and health-related absence (MGD 1.2 and 2.4) and compare them with the baseline values to identify the change over the programme period. Further, through qualitative evidence, the evaluation tried to understand whether and how has the project impacted literacy in WFP covered schools through analysis of project outcomes in light of certain variables such as regularity of provision of school lunch, school infrastructure, and teacher’s attendance in school. Also, the role of school gardens as perceived by the beneficiaries and school authorities were examined to understand whether the gardens are viewed as a learning tool on agriculture and nutrition and whether the current resources allowed sustenance of the gardens. 5. The Government’s willingness and capacity to take over the MGD-SFP was critical to sustaining the programme beyond the MGD grant period. Discussions with the officials during the scoping mission underscored a strong political will to move away from food aid to self-sufficiency during the implementation of the scaled-up programme. However, limited human and financial resources and inadequate technical and operational capacities of the staff were critical constraints as acknowledged by the Government. Convergence of action among the Departments of Health, Education, and Sports, and Agriculture were lacking. WFP-CO views community participation as the key to sustaining the outcomes of the programme and is geared towards building an ecosystem for community ownership and therefore, the evaluation assessed the processes set-up by WFP to strengthen capacities at the national and the field levels and assess the sustainability plan USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 99 delineating clear responsibility of the stakeholders and the design of the exit plan. Discussions with government officials, WFP and the community (parents and VEDC) highlighted the different challenges in the handing-over plan. 6. The MTE and impact evaluation of the programme had made recommendations which specifically cater to rationalisation and strengthening of the programme's reporting system and capacity; strengthening of the local capacities especially that of VEDC; development of sustainable school feeding models that are cash-based and involve local procurement, and on strengthening of complementary activities such as development of school gardens, nutrition education, and mainstreaming of gender into the field-level activities. The end-line evaluation assessed the rationale of these recommendations and applying the primary and secondary data assessed the extent to which these recommendations were integrated into the programme. The evaluation also specified the reasons for the programme to not incorporate the recommendations which might be due to contextual, structural or implementation related issues. Approach and Methodology 7. In light of the evaluation questions, review of the programme documents and drawing from the observations from the scoping mission, the evaluation presented an approach that included the key thematic and functional areas that the programme interventions work upon. These interventions were categorised under four domains: 1) Individual (school going children), 2) Institutional (school teachers, school administrators, school cook and supply managers), 3) Social (parents and community), and 4) External environment (policy, government support, civil society). 8. Central to the programme are the primary and pre-primary school going children whose well-being were affected by their parents and community members and the institutional players. These actors in combination with the external policy environment influenced the outcomes of enrolment, attendance, literacy, and the quality of literacy instructions. The external policy influence acted as the larger enabling environment which requires supportive Government policies in the domain of education, nutrition, WASH, and gender. 9. The evaluation methodology was guided by the ToR, the proposed approach and the results framework of the FY 14 programme. The end-line evaluation was a non-experimental, cross￾section study which followed a pre-and-post study design given the absence of counterfactuals in the baseline conducted in 2014. The evaluation, therefore, will not be able to attribute any changes to the programme. Appropriate descriptive and normative questions were designed to allow comparison with the baseline. Plausible causal links between the programme’s work and the observed changes were identified through the methods used for the evaluation. 10. Drawing upon the risks and assumptions of the ToC developed during the MTE of the programme, the methodology examined the causality implied in programme design and, identified the validity of the underlying assumptions in the context of the programme, determined key factors or issues that explained the extent to which the programme has achieved (or is likely to achieve) its objectives. 11. A mixed method approach was employed to answer the questions under the specified evaluation criteria using structured interviews for quantitative data and semi-structured interviews and discussions for qualitative data. The qualitative component used the participatory method of H￾form tool to gather data from individuals as well as institutions. A summary of the methodology presenting the methods of data collection, the target groups/ sources of information, the nature of the interviews, and the indicative outputs are presented in Table 19. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 100 Table 19: Summary of the evaluation methodology Methods of data collectio n Primary Survey Secondary Review Quantitative (Structured interviews) Qualitative (Focus group discussions, key informant interviews) Target Group/ Source of Information  Children  Parents  School head  School teachers  School cook  School Storekeeper  School Observation  Children  Parents  School head  School teachers  School cook  School Storekeeper  School Observation  Network Partners  Government officials of MoAF, MoES and MoH  VEDC and Farmer Groups  A literature review of available public sources  Project documents and results framework  Standard project reports and semi-annual reports  Community strength assessments  Baseline and mid-term reports  Impact review on school meals  MIS and monitoring data Indicative outputs of data collection  Establish end-line values of key indicators  Assess change from baseline  Examine the effectiveness of the programme  Positive and negative trends indicating a change  Change in enrolment and attentiveness (from direct and indirect influence)  Change in availability and access to food supplies and other school materials  Identify key change agents Wherever appropriate gender disaggregated analysis will be provided  Children and parents: Quantity and quality of receipt and consumption of school lunch, dietary diversity, regularity to school, change in attentiveness, reasons for change (disaggregated by sex), economic incentives  School: observed changes, infrastructural support, school gardens, gender considerations in implementation, perceived risks Others: Implementation modalities, changes and reasons, policy perspective, province and district level initiatives, resources and partnerships, innovation and learning  Build context and relevance of the programme  Establish alignment of the programme with government initiatives and WFP’s country strategy  Resources utilised  Chart performance indicators to assess the extent of change  Key design elements that contribute to direct and indirect change  Pathways created for sustainability; scope of replicability; key considerations for recommendations 12. The data from the secondary review and the primary survey (quantitative and qualitative) were triangulated to assess the reliability and validity of the data. The evaluation matrix presents the sources from where the data for the evaluation questions were collected. Triangulation was mainly undertaken within methods, where appropriate, by comparison of perspectives of different stakeholders on the same issue. In some cases, results were corroborated by use of different methods. Integration of Gender into the Methodology 13. The evaluation has integrated gender dimensions into its design. 14. In addition, the evaluation also examined the role and nature of participation of men and women in the programme specifically through the VEDC. Qualitative interviews focused on assessing the factors that have affected literacy, attendance, and attentiveness in school, for both boys and girls. Through quantitative assessment, the analysis integrated demographic information such as ethnicity and gender that may affect the responsiveness and utilisation of the school feeding. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 101 15. The evaluation adopted a mixed method approach to capturing all voices from the field (men, women, boys, and girls) were taken into consideration during the evaluation. The data collection team was adequately trained to ensure that the views of all key groups are considered, reflected and triangulated, with due attention to issues focussing on gender. Gender balance in the team was ensured. In case of the quantitative interviews, parents (men and women), boys and girls were interviewed separately. FGD participants included both men and women almost in equal numbers, and questions to assess their views on gender issues were included as part of the FGD checklist. Further, the qualitative team included one male and one female moderator to enhance participation of women in the discussion. 16. The evaluation required that GEEW is mainstreamed throughout the analysis. As can be seen through the evaluation questions, presented in Annex M, question 2, 3, 4 and 5 under the criteria of relevance aimed to understand- a) whether the programme aligned with the priority of women, men, girls and boys b) whether the programme was based on a sound gender analysis, c) whether the design and implementation were gender sensitive and d), whether the programme had made any impact on gender relations. Evaluation questions 7, 8 and 9 under the criteria of effectiveness will enable the evaluation to provide sex-disaggregated data on the attainment of programme outcomes related to literacy, attendance, and attentiveness of the students. Question 19 under the criteria of impact, specifically responds to influence of the programme on the gender context and the gender-specific impacts of the programme. 17. The evaluation laid equal emphasis on analyses of sex disaggregated data in school feeding, literacy, and attendance in Lao PDR. In addition, the evaluation also examined the gender roles in the implementation of the MGD-SFP with respect to participation in local community groups and involvement in school feeding management committees. Qualitative data was analysed to identify reasons that led to drop-out or affected attendance of boys and girls in schools. Further, variables such as ethnicity and distance from the school were used to contextualise the variations in evaluation findings for both boys and girls. Data Collection Methods and Tools 18. Secondary Data/ Literature Review: A systematic review of the literature was conducted to build the context of the MGD-SFP. The programme monitoring data on outputs and results and the evaluation reports (in conjunction with primary data) were used to analyse and present, output, and outcome related performance indicators. The project reports helped to build the narrative around the evaluation questions. Review of Lao National Nutrition Strategy, GoL’s Policy on Promoting School Lunch, WFP’s Strategic Plan 2017-21, WFP’s Gender Policy and WFP’s School Feeding Policy allowed the evaluation to identify linkages and alignment of the MGD-SFP with WFP’s mandate and the Government’s priorities. 19. Primary Data Collection Tools: The quantitative tools (shared in a separate file) was developed for the following set of respondents: a. School-going children: Children from grades one to five were selected through random sampling and interviewed. The questions mainly catered to their receipt of school lunch, attendance, health￾related absenteeism, and their home and class learning environment. To address the low recall of grade 1 and grade 2 students, support from class teachers was sought to corroborate the data. Further, the evaluation methodology aimed to select parents for interviews of the same sampled children to validate the information collected from the children. For children in class 2 and 3, the Early Grade Reading Assessment (EGRA) tool was also administered to assess their learning/literacy levels56. 56There are foreseen limitations in analysis of data provided by EGRA which have been detailed in the section on Limitations and Risks. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 102 b. Parents: Parents of the sampled school going children were interviewed to collect data on receipt of school lunch, their contribution to the programme, the dietary diversity, the educational environment at home, and water and sanitation status at home. c. School-based respondents: A school questionnaire for the school head was developed to include a wide range of information that will be reported and observed. The information included enrolment data, attendance data, student attentiveness, school capacities, infrastructure, maintenance, food utilisation and specific questions on the provision of school lunch. In addition, questions related to school gardens, water and sanitation, and hygiene were also included. The other respondents from school included school teachers, cook, and storekeepers. The tools have been provided in a separate document along with this report. 20. In addition to the above respondents, focussed group discussions (FGDs) were held with the VEDC and farmer groups in the villages. Interviews were held with Government officials at the national, provincial and district level especially from MoES, MAF and MoH focusing on the current engagement of the Government with the implementation of the MGD-SFP, current levels of capacity and staffing requirements and plans to strengthen the same and support required to take-over and sustain the programme. Plans for convergence among the three departments and related issues were also explored. 21. The quantitative questionnaires aptly covered the key programme outcomes including literacy, attendance, and attentiveness. The participatory qualitative tools consisted of discussion guides for FGDs and key informant interviews. Data Analysis 22. The methodology and the questionnaires have been developed in line with the baseline data collection to ensure comparability of datasets. Existing baseline data (FY14) has been used for reference in the analysis phase, and information on implementation indicators was taken from existing WFP monitoring of the MGD-SFP. 23. The indicators specified in the ToR were mapped with the type of tools, respondent group and the question numbers which facilitated the identification of methods of data triangulation and assessed the reliability and validity of the data collected. 24. The analysis of quantitative data included descriptive analysis of sex-disaggregated data showing the comparison of baseline and end-line. This highlighted the change in intermediate and immediate outcomes over time. Since the baseline did not have a control group, a pre-and-post analysis provided the difference in the two scenarios without attribution to the programme. The evaluation also used the findings from the MTE and the Impact Assessment Review to supplement the analysis. 25. Quantitative data was collected through Computer Aided Personal Interview (CAPI) using tablets. The software was programmed to minimize data entry errors using built-in constraints and skip￾pattern logic. Data collection teams were trained to cross-check the data before it is uploaded to the server. Data was sent to a central server daily, and the evaluation team conducted range and consistency checks on the data daily during the first week and on a weekly basis for the duration of the data collection period to identify and address any errors in the data collection process. 26. Qualitative data were analysed using content analysis. The analysis focused on providing the reasons for the change observed. The best practices, challenges, and learnings were captured through qualitative data. Qualitative data was translated into English on a weekly basis and checked by the evaluation team for consistency based on the field visits by the core evaluation team and the quantitative data. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 103 27. Data on attainment of outputs and outcomes have been disaggregated by sex and age. An assessment of qualitative data on reasons that led to drop-out or affect attendance of boys and girls in schools was conducted. Variables such as ethnicity and distance from the school have been used to contextualise the variations in evaluation findings for both boys and girls. Site Mapping 28. The end-line evaluation covered the diversity of the geographical locations of the schools (i.e., both schools that are easily accessible by roads and those that are located in difficult terrains were surveyed). The schools across all the programme provinces covered in the baseline were included to capture the variances across provinces. 29. The advantage of this approach was that the intensity of the examination vis-à-vis community interactions increased and added value to the data. Moreover, the information gathered was more nuanced and retained the robustness of the quantitative design. This ensured upholding both WFP’s mandate and the government’s concerns. This approach and methodology were approved by WFP-CO and USDA during a meeting held on 1 March 2018. 30. As per the ToR, the baseline study of 2015 covered 85 schools in 10 districts across 6 provinces57. To enable comparison, the end-line evaluation covered all the programme provinces covered during the FY14 baseline study ensuring the geographical representation of the sample. Following an intensive approach58, the schools were selected from 8 districts spread across these provinces. The sample of the schools wasspread across the districts in the same proportion as the number of programme schools present in the districts, thus ensuring the statistical representation of the sample. 31. While it was proposed that the schools be selected from the list of schools covered during the baseline FY14 using simple random sampling, two concerns limited the use of this method. First, it needed confirmation that schools located in hard to reach areas are duly covered to include beneficiaries from remote areas where access is limited, highlighting their specific needs. Second, as advised by WFP-CO, the evaluation was expected to include schools covered by its implementing partners. To address these two concerns, the evaluation of selected schools using simple random sampling method and then mapped the schools on the district map using geo-referenced data to identify their remoteness levels. In the case of a district with none of the chosen schools located in hard to reach areas, a purposive selection was made to add such a school to the list. Similar replacements were made to ensure inclusion of schools also covered by the implementing partners. 32. As per the proposal, the end-line evaluation was to cover studentsfrom schools but actually sampled 450 students and 450 parents across 58 schools from among the list of schools covered in the baseline. A summary of the sampling stages for the selection of respondents, approach for selection of respondents and the sample size and distribution by provinces and by respondents has been provided below. Sample Size Covered by Provinces and Respondent Category 33. Quantitative: The following tables present the sample covered for the quantitative survey: Table 20: Sample covered during quantitative survey Phase 1 DATA COLLECTION PHASE 1 (26 March to 6 April) (Southern Provinces) S. No. Province District No. of Schools Visited No. of students interviewed No. of parents interviewed No. of cooks No. of storekeepers No. of school heads No. of teachers 1 Attapeu Xaisettha 5 73 75 5 5 2 Attapeu Sanamxai 4 60 42 4 4 4 4 57LuangPrabang was not included in the baseline survey of FY 14. 58As agreed in discussion with WFP-CO and USDA on 1 March 2018 to revise the sampling units USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 104 DATA COLLECTION PHASE 1 (26 March to 6 April) (Southern Provinces) S. No. Province District No. of Schools Visited No. of students interviewed No. of parents interviewed No. of cooks No. of storekeepers No. of school heads No. of teachers 3 Salavan Salavan 5 71 62 5 5 4 Salavan Vapi 5 69 65 5 5 5 Salavan Laongam 7 102 91 7 7 7 8 6 Sekong Lanam 5 75 73 5 5 7 Sekong Thateng 5 75 64 5 5 5 6 Total 3 7 36 525 472 16 16 36 38 Table 21: Sample covered during quantitative survey Phase 2 DATA COLLECTION PHASE 2 (30 April to 11 May) (Northern Provinces) S. No. Province District No. of Schools Visited No. of students interviewed No. of parents interviewed No. of cooks No. of storekeepers No. of school heads No. of teachers 1 LuangPrabang Ngoy 3 45 20 3 3 3 3 2 Oudomxay Xai 9 135 93 8 8 8 8 3 Oudomxay Beng 6 90 76 6 6 6 6 4 Oudomxay Nga 2 30 21 2 2 2 2 5 Phongsaly Boun-Tai 5 75 61 5 5 5 5 6 Phongsaly Phongsaly 6 90 96 6 6 6 6 7 LuangNamtha Vientphoukha 11 165 145 11 11 11 11 Total 3 6 42 630 512 41 41 41 41 34. Qualitative: The following table presents the sample covered for the qualitative survey: Table 22: Sample covered during qualitative survey Phase 1 DATA COLLECTION PHASE 1 (26 March to 6 April) (Southern Provinces) S. No. Province District Project/ Control No. of Schools Visited FGD Parents Cook IDI Storekeeper IDI School Head IDI Teachers IDI FGD VEDC 1 Attapeu Xaisettha Control 2 1 2 1 2 Attapeu Sanamxai Project 2 2 1 1 1 1 3 Salavan Salavan Control 2 2 1 1 4 Salavan Vapi Control 1 1 1 5 Salavan Laongam Project 2 1 2 1 1 1 1 6 Sekong Lanam Control 1 1 1 7 Sekong Thateng Project 3 3 2 2 2 1 Total 3 7 6/7 13 11 4 4 6 6 4 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 105 Table 23: Sample covered during qualitative survey Phase 2 DATA COLLECTION PHASE 2 (30 April to 11 May) (Northern Provinces) S. No. Province District Project/ Control No. of Schools Visited FGD Parents Cook IDI Storekeeper IDI School Head IDI Teachers IDI FGD VEDC 1 LuangPrabang Ngoy Project 1 1 1 1 1 2 Oudomxay Xai Project 2 2 1 1 1 3 Oudomxay Beng Project 2 2 1 1 1 4 Oudomxay Nga Project 2 1 1 1 1 1 5 Phongsaly Boun-Tai Project 2 2 1 1 1 6 Phongsaly Phongsaly Project 2 2 1 1 1 1 7 LuangNamtha Vientphoukha Project 3 3 1 2 1 1 1 Total 3 6 11 14 13 6 6 6 6 2 Ensuring Quality 35. WFP’s DEQAS defines the quality standards expected from this evaluation and sets out processes with in-built steps for Quality Assurance, Templates for evaluation products and Checklists for their review. DEQAS is closely aligned to the WFP’s evaluation quality assurance system (EQAS) and is based on the UNEG norms and standards and good practice of the international evaluation community and aims to ensure that the evaluation process and products conform to best practice. DEQAS was systematically applied to this evaluation guiding the evaluation team at NRMC. 36. Having a rich experience of working with UN agencies, NRMC has a deep understanding of the UNEG norms, standards, and ethical guidelines and aspires to strictly adhere to the same. In addition, NRMC’s internal quality protocols have been integrated with the process to obtain reliable data and ensure the quality of deliverables. Some of the key steps taken to ensure quality at various stages of the survey are: a. The data collection team comprised well experienced personnel for collecting and collating both quantitative and qualitative information. Separate teams for each were formed to ensure that the tasks are clearly assigned and the teams can get well versed with their respective tools. Qualitative discussions with the Government officials, WFP field offices and partners were conducted by the NRMC core team (with the help of a translator where required) to ensure the quality of discussions with these stakeholders. b. As part of quality control and to ensure timeliness of collection of data, NRMC, with support from Geo-Sys, a Laos-based firm, developed a detailed field movement plan in advance of the survey. These plans indicated the daily team movement outlining the number of quantitative and qualitative interviews conducted. It also included the plan for the core team reflecting their division of time on monitoring of field data in the initial phase of the survey and discussion with secondary stakeholders of the programme. Towards this, NRMC was supported by WFP in contacting the secondary stakeholders (government officials, other partners etc.) and scheduling the meetings with them in advance, as per the field plan. c. The data collection team, which is based in Laos, was trained by NRMC on the tools. Specific sessions were conducted on ethical issues faced during data collection, data integrity and interacting with children. They were also provided a detailed explanation of the tools including imparting a field training applying the tools. 37. A robust monitoring process was followed for quality assurance during data collection. A field plan was created and shared with WFP CO highlighting the estimated time to complete the primary USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 106 survey. At least two members of the core evaluation team of NRMC was present in the field during the entire period of data collection accompanying the qualitative and quantitative interviews. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 107 Annex O: Documents Gathered 1 WFP CO has shared a host of documents for the evaluation. The table below described the nature of documents received. Table 24: Documents Gathered Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents [if applicable] Appraisal mission report N/A Project document (including Logical Framework in Annex) Country Strategic Plan, Lao (2017-21), Project Proposal including the results framework Y Relevance Standard Project Reports Standard project Reports (2014, 2015, 2016), Narrative reports. Y Relevance Budget Revisions Budget increases to development activities—Lao People's Democratic Republic Country Programme 200242 Y Relevance, Efficiency, Effectiveness Note for the record (NFR) from Programme Review Committee meeting (for original operation and budget revisions if any) N/A Approved Excel budget (for original intervention and budget revisions if any)  USDA Commitment letter for agreement #FFE-439- 2014/049-00 Laos, 2014  USDA Third Year Operational Plan for Agreement #FFE-439- 2014/049-00-C Laos, 2016 Y Relevance, Efficiency Intervention/Project Plan (breakdown of beneficiary figures and food requirements by region/activity/month and partners) COMP 2017, 2018,Project Proposal including the results framework, Amendments, commitment letters Y Relevance Other Country Office Strategic Documents (if applicable) Country Strategy Document (if any) Lao People’s Democratic Republic Country Strategic Plan (2017– 2021), WFP Y Relevance, Sustainability Other Assessment Reports [if applicable] Comprehensive Food Security and Vulnerability Assessments CLEAR, Levels and Trends in Child Malnutrition 2016, 2017 Y Relevance, Effectiveness USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 108 Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Nutrition factsheets, Situation assessments, impact evaluations, project reports etc. Crop and Food Security Assessments (FAO/WFP) Stakeholder mapping of the nutrition and agriculture sectors in the target areas of ENUFF project Y Relevance, Effectiveness Emergency Food Security Assessments N/A Food Security Monitoring System Bulletins N/A Market Assessments and Bulletins N/A Joint Assessment Missions (UNHCR/WFP) N/A Inter-Agency Assessments N/A Rapid needs assessments N/A Cash and voucher feasibility studies Cash Transfer WFP Lao monitoring findings -Nalae and Beng districts (2018) Y Relevance, Sustainability Other Monitoring & Reporting (if applicable) M&E Plan Performance Monitoring Plan Y Relevance, Effectiveness, Impact M&E Policy USDA Monitoring and Evaluation Policy Y Relevance M&E Indicators and Definitions USDA Food for Progress and McGovern-Dole Indicators and Definitions Y Effectiveness, Impact Country Situation Report (SITREP) WFP Lao Country Brief, Lao country fact sheet (FAO), Nutrition Country profile Lao (2015), Saber Country Report, IFPRI, various relevant reports Y Relevance Country Executive Brief WFP Lao Country Brief Y Relevance Food Distribution and Post-distribution Monitoring Reports  Mid-Term Evaluation of McGovern-Dole School Feeding in Laos 2015–2016 Evaluation Report  Impact Assessment Review on School Meals and WASH Activities (2002-2016)  Project reports  USDA Mc Govern Dole-Semi-Annual Reports Y Relevance, Effectiveness, Impact USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 109 Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix  Others Monthly Monitoring Reports Semi-annual and annual project reports Y Effectiveness, Impact, Efficiency Beneficiary Verification Reports N/A Donor specific reports USDA Semi-annual Reports Y Effectiveness, Impact, Efficiency Output monitoring reports (if applicable) Actual and Planned beneficiaries by activity and district/ location by year Actual and Planned beneficiaries by activity by year; location wise monitoring data is not available, USDA semi-annual reports Y Effectiveness, Impact, Efficiency Male vs. Female beneficiaries by activity and district/ location by year Male vs. Female beneficiaries by activity by year, location wise monitoring data is not available, USDA semi-annual reports, USDA semi-annual reports Y Effectiveness, Impact, Efficiency Beneficiaries by age group N/A Actual and Planned tonnage distributed by activity by year N/A Effectiveness, Impact, Efficiency Commodity type by activity N/A Actual and Planned cash/voucher requirements (US$) by activity by year N/A Operational documents (if applicable) N/A Organogram for main office and sub-offices Activity Guidelines Mission Reports Pipeline overview for the period covered by the evaluation Logistics capacity assessment Partners (if applicable) Annual reports from cooperating partners N/A Effectiveness, Impact, Sustainability List of partners (Government, NGOs, UN agencies) by location/ activity/ role/ tonnage handled List of partners has been provided in ToR. Relevant documents have been shared for their roles. Y Effectiveness, Sustainability USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 110 Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Field level agreements (FLAs), Memorandum of Understanding (MOUs) FLAs with BBM and Plan International have been shared Y Effectiveness, Efficiency, Sustainability Cluster/ Coordination meetings (if applicable) N/A Logistics/Food Security/nutrition cluster documents NFRs of coordination meetings Other Evaluations/ Reviews Evaluations/ reviews of past or on-going operation  Mid-Term Evaluation of McGovern-Dole School Feeding in Laos 2015–2016 Evaluation Report  Impact Assessment Review on School Meals and WASH Activities (2002-2016) Y Effectiveness, Impact, Sustainability Resource mobilisation (if applicable) Resource Situation COMP 2017, COMP 2018 N/A Contribution statistics by month Resource mobilization strategy NFRs Donor meetings Maps (if applicable) N/A Operational Map Logistics Map Food/Cash/voucher Distribution Location Map Food Security Map Other documents collected by the team (including external ones) (if applicable) Specify Specify USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 111 Annex P: List of Sample Schools Project Schools Sl. No. Province Code Province Name District Code District name Village Code School Code School Name Latitude Longitude 1 02 Phongsaly 201 Phongsaly 201094 10201079 Phonkeo 21.672220 102.088924 2 02 Phongsaly 201 Phongsaly 201061 10201048 Yaofang 21.887864 102.058615 3 02 Phongsaly 201 Phongsaly 201087 10201004 Mongchao 21.667378 102.088519 4 02 Phongsaly 201 Phongsaly 201098 10201070 Phonhom 21.674447 102.260181 5 02 Phongsaly 201 Phongsaly 201083 10201031 Hathin 22.030147 102.298885 6 02 Phongsaly 201 Phongsaly 201084 10201054 Sobkang 22.099431 102.244458 7 02 Phongsaly 207 Boun-tai 207006 10207018 Phiasouk 21.430750 102.043111 8 02 Phongsaly 207 Boun-tai 207034 10207023 Turka 21.425806 102.164528 9 02 Phongsaly 207 Boun-tai 207052 10207051 Aneng 21.291398 101.998961 10 02 Phongsaly 207 Boun-tai 207003 10207004 Navay 21.361838 101.979465 11 02 Phongsaly 207 Boun-tai 207002 10207048 Bountai 21.391376 101.970639 12 03 LuangNamtha 304 Viengphoukha 304039 10304001 Namfa 20.621422 101.077371 13 03 LuangNamtha 304 Viengphoukha 304020 10304007 Nongkham 20.689781 101.084274 14 03 LuangNamtha 304 Viengphoukha 304037 10304016 Thalouang 20.743290 101.017372 15 03 LuangNamtha 304 Viengphoukha 304030 10304019 Thonglat 20.835595 101.009461 16 03 LuangNamtha 304 Viengphoukha 304005 10304028 Nammang 20.663524 101.066995 17 03 LuangNamtha 304 Viengphoukha 304040 10304036 Lamon 20.634765 101.083668 18 03 LuangNamtha 304 Viengphoukha 304059 10304012 Konthou 20.522722 101.146167 19 03 LuangNamtha 304 Viengphoukha 304008 10304020 Vieng-mai(Lavair) 20.644450 101.043291 20 03 LuangNamtha 304 Viengphoukha 304022 10304013 Nam-eng 20.721343 101.142983 21 03 LuangNamtha 304 Viengphoukha 304025 10304008 Namloung 20.771354 101.211992 22 03 LuangNamtha 304 Viengphoukha 304032 10304009 Namkap-Mai 20.705467 101.010155 23 04 Oudomxay 401 Xai 401065 10401216 Ban Lang 20.633580 101.851460 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 112 Sl. No. Province Code Province Name District Code District name Village Code School Code School Name Latitude Longitude 24 04 Oudomxay 401 Xai 401024 10401232 Donxai 20.685490 101.987550 25 04 Oudomxay 401 Xai 401104 10401223 Thouting 20.644560 101.689800 26 04 Oudomxay 401 Xai 401056 10401214 Na-om 20.657870 101.885770 27 04 Oudomxay 401 Xai 401138 10401029 Mokkha 20.606270 101.810590 28 04 Oudomxay 401 Xai 401174 10401069 Phonhom 20.807550 101.945180 29 04 Oudomxay 401 Xai 401013 10401140 Homsouk 20.698750 101.977570 30 04 Oudomxay 401 Xai 401034 10401070 Houaykhoum 20.648756 101.979253 31 04 Oudomxay 401 Xai 401035 10401021 Thiao 20.635880 101.971420 32 04 Oudomxay 405 Beng 405098 10405016 Nahouay 20.305031 101.645473 33 04 Oudomxay 405 Beng 405064 10405040 Namdo 20.298969 101.618384 34 04 Oudomxay 405 Beng 405057 10405057 Namkhong 20.345715 101.582743 35 04 Oudomxay 405 Beng 405100 10405109 Phonsi 20.236921 101.568798 36 04 Oudomxay 405 Beng 405077 10405092 Phoukham 20.266029 101.594688 37 04 Oudomxay 405 Beng 405021 10405015 Nangoua 20.398824 101.679746 38 04 Oudomxay 404 Nga 404108 10404080 Phonkham 20.459320 101.978730 39 04 Oudomxay 404 Nga 404010 10404058 Mai 20.451130 102.028950 40 06 LuangPrabang 606 Ngoy 606145 10606039 Moklak 20.396664 102.508340 41 06 LuangPrabang 606 Ngoy 606033 10606013 Hatsa 20.927776 102.764273 42 06 LuangPrabang 606 Ngoy 606106 10606162 Houaykheung 20.443426 102.731082 43 14 Salavan 1407 Laongam 1407006 11407066 Dongnoi 15.361295 106.200900 44 14 Salavan 1407 Laongam 1407016 11407083 Nathan 15.545489 106.065713 45 14 Salavan 1407 Laongam 1407017 11407075 Tabaeng 15.571584 106.040793 46 14 Salavan 1407 Laongam 1407042 11407049 Dongbang 15.494342 106.151548 47 14 Salavan 1407 Laongam 1407027 11407005 Nongkae 15.488273 106.077098 48 14 Salavan 1407 Laongam 1407037 11407128 Sanumna 15.513548 106.250270 49 14 Salavan 1407 Laongam 1407061 11407007 Vangpuay 15.481115 106.227109 50 15 Xekong 1504 Thateng 1504007 11504004 Kapu 15.445419 106.458102 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 113 Sl. No. Province Code Province Name District Code District name Village Code School Code School Name Latitude Longitude 51 15 Xekong 1504 Thateng 1504032 11504030 Thon-gnai-mai 15.405369 106.571038 52 15 Xekong 1504 Thateng 1504042 11504029 Pong-nua 15.468168 106.363842 53 15 Xekong 1504 Thateng 1604056 11504048 Kongtayoun 15.393699 106.392550 54 15 Xekong 1504 Thateng 1504004 11504002 Houaxe 15.434265 106.411519 55 17 Attapeu 1703 Sanamxai 1703001 11703006 Khang 14.760171 106.710247 56 17 Attapeu 1703 Sanamxai 1703015 11703007 Hatyao 14.675350 106.577593 57 17 Attapeu 1703 Sanamxai 1703032 11703022 Thahin-tai 14.719012 106.519237 58 17 Attapeu 1703 Sanamxai 1703006 11703044 Donphai 14.713238 106.658347 Control Schools Sl. No. Province code Province name District Code District name Village Code Village name EMIS Code latitude longitude 1 17 Attapeu 1701 Xaysetha 1701006 Toiy 11701028 14.8322029666 106.9586506390 2 17 Attapeu 1701 Xaysetha 1701003 Fungdang 11701017 14.8106334771 106.9334112020 3 17 Attapeu 1701 Xaysetha 1701014 Vatloung 11701021 14.8017556827 106.9478468760 4 17 Attapeu 1701 Xaysetha 1701032 Donekgiew 11701051 14.7505048840 106.9534707400 5 17 Attapeu 1701 Xaysetha 1701015 Somkoth 11701018 14.8049836107 106.9274213350 6 14 Saravane 1401 Saravane 1401082 Tongnoy 11401010 15.7035058775 106.3013524280 7 14 Saravane 1401 Saravane 1401087 Kaengpor 11401143 15.7404400133 106.2958839610 8 14 Saravane 1401 Saravane 1401162 Beungxaitai 11401162 15.7063190302 106.2291720560 9 14 Saravane 1401 Saravane 1401042 Naxainoy 11401172 15.6145110608 106.3110100860 10 14 Saravane 1401 Saravane 1401049 Chonggnai 11401181 15.6148543580 106.2903285520 11 14 Saravane 1405 Vapy 1405006 Sinouanxai 11405006 15.6624780561 105.8766629870 12 14 Saravane 1405 Vapy 1405023 Saphath 11405014 15.6926761226 105.9653300990 13 14 Saravane 1405 Vapy 1405018 Lao 11405018 15.7000226328 106.0270982090 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 114 Sl. No. Province code Province name District Code District name Village Code Village name EMIS Code latitude longitude 14 14 Saravane 1405 Vapy 1405033 Sim 11405041 15.6561419719 105.9888574570 15 14 Saravane 1405 Vapy 1405012 Nongpho 11405012 15.6810164582 106.0986466910 16 15 Sekong 1501 Lamam 1501010 Lavifangdaeng 11501007 15.29837208 106.7069964 17 15 Sekong 1501 Lamam 1501004 Maihouameuang 11501015 15.35526074 106.7331358 18 15 Sekong 1501 Lamam 1501014 Xeinoy 11501009 15.22772373 106.7432756 19 15 Sekong 1501 Lamam 1501043 Vatxang 11501043 15.29224221 106.7771248 20 15 Sekong 1501 Lamam 1501033 Honglai 11501040 15.38633002 106.7065464 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 115 Annex Q: End-line Values of Key Indicators Table 25: End-line values of MGD-SFP (FY14) performance indicators, disaggregated by sex and geographical location Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h MDG SO1 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of the grade￾level text Pupils reading and understanding ability were assessed using the literacy boost questionnaire developed by Save the Children Laos. The questionnaire was administered to 830 students from Grade 3. The assessment had five sub-tests and students were categorised into emergent, beginner and reader with comprehension ability based on comprehension sub-test score, as recommended in Literacy Boost guideline. Students demonstrating >75% comprehension are given here. Baselin e FY14 830 1.9 1 1.7 2.2 2.5 1.1 End￾line FY14 175 6.9 1 8.7 4.8 4.7 13 MGD 1.1.1 Average teacher attendance rate Teacher attendance was assessed using monthly school records of every teacher’s attendance and comparing this data against the number of school days per month in the academic semester September 2017-February 2018. Baselin e FY14 321 94 3 94 94 95 92 End￾line FY14 294 96.9 94.9 94.6 97.8 94.2 Percent of teachers attending at least 90 percent of the school days Regular teacher attendance was defined as attending more than or equal to 90 percent of school days; teacher attendance was Baselin e FY14 321 841,3 83 85 92 74 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 116 Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h assessed using monthly school records of all the teacher from sample schools and the number of working days per month for the academic semester September 2017-February 2018. End￾line FY14 294 93.8 1,3 97.2 97.9 97.2 83.3 MGD 1.1.4 Percent of teachers/ educators/ teaching assistants trained or certified in teaching techniques during the last one year Baselin e FY14 575 23 1 20 26 26 19 End￾line FY14 294 48.1 1 42 55 48 47 Percent of schools with at least one teacher trained End￾line FY14 57 (schools) 100 - - 100 100 Percent of teachers/ educators/ teaching assistants in target schools who demonstrate the use of new and quality teaching techniques or tools as identified by their supervisor/mentor/co ach Teachers, educators, teaching assistants who have successfully completed a pre- or in-services training programme to teach in schools or equivalent non-school based settings over the past one year were then assess whether they are using the learned techniques. Use of techniques was assess by asking their supervisors whether the teacher is using the techniques he/she learned in the classroom. Baselin e FY14 134 100 1 - - - - End￾line FY14 57 (Schools) 99 1 - - - - MGD 1.2 Percent of students in classrooms identified as inattentive by their teachers Attentiveness of students was collected based on the majority view of two teachers on the Baselin e FY14 850 19 2 722 16 11 11 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 117 Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h attentiveness of fifteen sampled students from each school End￾line FY14 870 20.7 2 17.4 23.9 21.3 19.2 MGD 1.3 Average student attendance Student attendance was assessed using monthly school records of fifteen sample students from every school and comparing this data against the number of school days per month in the academic semester September 2017- February 2018 Baselin e FY14 367 97 3 97 97 99 94 End￾line FY14 690 98.1 98.3 98.3 98.9 95.2 Percent of students regularly (at least 80 percent of the school days) attending school Student attendance was measured using the attendance record of fifteen randomly selected students from every sample school for the last academic semester (September 2017- February 2018). Baselin e FY14 367 99.5 99.5 99.4 100 99 End￾line FY14 690 98.3 98.3 98.3 98.3 98.3 Student attendance on the day of the survey The proportion of enrolled students present at school during the survey day was calculated using direct observation to count the number of students present and comparing it with the number of enrolled students. The mean proportion of students in attendance is presented here. Baselin e FY14 85 (schools) 89 3 88 89 94 81 End￾line FY14 57 (schools) 91 90 92.8 95 80 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 118 Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h MGD 1.3.4 Annual percent change in students enrolled in WFP supported school Enrolment figures from the past five academic years were collected from school records – where available; student numbers of each academic year were then compared with the previous academic year. Percent change of enrolment in the current academic year (2017-18) compared to the last academic year (2016-17) is presented here. Baselin e FY14 79 (schools) -0.7 -0.4 -1 -1.5 0.9 End￾line FY14 20 (schools) 1 3 -1 1 1 Average enrolment ratio of girls to boys at target schools A total number of male and female students per school was recorded for the past five academic years from the school records during the baseline survey. The ratio was calculated based on enrolment figures for the past academic years. Girl-boy enrolment ratio for last academic year (2016-17) is presented here. Baselin e FY14 79 (schools) 0.95 - - 0.95 0.95 End￾line FY14 20 (schools) 0.95 0.96 0.91 Average dropout rate Dropout figures from the past academic years were collected from school records –where available. The dropout rate was calculated per school from a number of students discontinuing their studies and enrolment figures for the same academic year. The dropout rate for last academic year (2016-17) is presented here. Baselin e FY14 66 (schools) 0.8 3 1.2 0.5 0.18 1.86 End￾line FY14 51 (schools) 2 2 2 1 3 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 119 Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h Repetition rate A total number of male and female students repeating in the same grade per school was recorded during survey data collection. Repetition rate was calculated per school from a number of repeating students and enrolment figures for the same academic year. Repetition rate for last academic year (2016-17) is presented here. Baselin e FY14 68 (schools) 7.8 2 9.5 6 7.3 8.4 End￾line FY14 20 (schools) 8 2 11 5 1.5 13 MGD 1.3.5 Percent of parents in programme schools who can name at least three benefits of primary education Data were collected through interviews with parents and asking them about the benefits of primary education. Baselin e FY14 810 45 1,3 48 43 42 50 End￾line FY14 708 96 1 96.2 95.5 96.7 93.9 MGD SO2 Average dietary diversity score (DDS) of school-aged children Dietary history of pupils was measured by interviewing parents using a 24-hour recall method. Mean Dietary Diversity Score is presented here. Baselin e FY14 810 5 3 5 5 5.2 4.8 End￾line FY14 706 7.6 7.7 7.5 7.9 7 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 120 Result level Indicator Source and measurement Effective sample size (N) Average value Value by sex Value by geographical location Male Female North Sout h MGD 2.1 Percent of students in target school who achieve a passing score on a test of good health and hygiene practices as a result of USDA assistance The good health and hygiene practices were identified in collaboration with WFP during the questionnaire development phase and included in the questionnaire to test the baseline knowledge of students enrolled in WFP supported schools. Students in target schools receiving at least a score of 80 percent on the test of good health and hygiene practices were considered as passed. Baselin e FY14 850 0 - - End￾line FY14 The commitment letter and the monitoring data have not referred to this target at all MGD 2.2 Percentage of food preparers at target schools who achieve a passing score (80 percent correct answer) in a test of safe food preparation and storage practices The percentage was calculated by testing cooks on safe food preparation and storage. The test contained ten questions, each worth one point. Respondents were asked to choose the best response for each question. The questions were developed in collaboration with the country office, to assess the baseline knowledge level and practices of food preparefor safe food preparation and storage practices. Baselin e FY14 85 (schools) 8.2 1 - - 8 8.6 End￾line FY14 58 (schools) 87.9 1 - - 92.8 75 ns- No significant difference between baseline and end-line, sex or geographical location. 1- Only significant difference between baseline and end-line. 2- Only a significant difference between male and female. 3- Only significant difference between north and south USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 121 Annex R: End-line Values of Other Key Performance Indicators S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 1 Number of public￾private partnerships formed as a result of USDA assistance (Multi-focus) Secondary; monitoring data provided by WFP 150 0 406 2 Number of individuals benefiting indirectly from USDA-funded interventions (male; female) Secondary; monitoring data provided by WFP; Reliability of this data is dependent on the method of data collection and definition of ‘indirect’. 126572 252894 3 Number of individuals benefiting directly from USDA-funded interventions (female; male) Secondary; monitoring data provided by WFP; Reliability of this data is dependent on the method of 498906 25652 6 24238 0 455024 23446 5 22055 9 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 122 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e data collection. 4 Number of students enrolled in schools Receiving USDA assistance (male; female) (Students enrolled" are students enrolled at the beginning of the school year at targeted schools Data will be disaggregated by school Unit of Measurement: Number: Students) Secondary; monitoring data; Primary Survey 502460 25123 0 23329 6 2914 150 9 1405 465426 23271 3 21717 8 5 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (male; female) Secondary; monitoring data; 511460 25573 0 24229 6 20517 4 10258 7 99754 471650 23582 5 22401 4 6 Number of textbooks and other teaching and learning materials provided as a result of USDA assistance Secondary; monitoring data; Primary Survey 30000 26 schools (58) 71563 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 123 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 7 Number of educational facilities (i.e. school buildings, classrooms, and latrines) rehabilitated/construc ted as a result of USDA assistance (kitchens, cook areas; Other school grounds or school buildings) Secondary; monitoring data; Primary Survey Kitchen (1339) Warehou se (1500) Kitchen-39 (58) Dining￾23(58) Warehouse￾36 (58) Kitchen (711) Warehou se (725) 8 Number of individuals trained in child health and nutrition as a result of USDA assistance (male; female) Secondary; monitoring data; 31999 18559 13440 0 0 0 19197 11508 7689 9 Number of school￾aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (male; female) Secondary; monitoring data; Primary Survey 405758 20571 9 20003 9 402403 20664 0 19576 3 10 Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance Secondary; monitoring data; Primary Survey 6740562 8 3442702 2 11 Number of take-home rations provided as a THR has been 183936 232668 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 124 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e result of USDA assistance discontinue d, thus the indicator may not be covered 12 Number of individuals receiving take-home rations as a result of USDA assistance (male; female) THR has been discontinue d, thus the indicator may not be covered 78468 45511 32957 209581 10720 9 10237 2 13 Number of students regularly (80%) attending USDA supported classrooms/schools (male; female) ("Students regularly attending" are students that attend a minimum of 80% of the school operating hours during the school year.) (USDA supported classrooms/schools is defined as those receiving direct services from the USDA supported programme.) Secondary; monitoring data; Primary Survey 252307 13684 5 11546 2 98.3 (690) 98.3 98.3 269894 13812 3 13177 1 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 125 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 14 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (female; male) Primary Survey 25 25 25 6.9 (175) 8.7 4.8 15 Number of national school meal sustainability workshops held as a result of USDA assistance Secondary; monitoring data; 1 8 16 Number of study tours outside Laos as a result of USDA assistance Secondary; monitoring data; 1 2 17 Number of exchange visits within Lao PDR as a result of USDA assistance Secondary; monitoring data; 1 6 18 Number of national school meal review workshops held as a result of USDA assistance Secondary; monitoring data; 1 2 19 Number of enrolment campaign events held as a result of USDA assistance Secondary; monitoring data; 3 0 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 126 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 20 Number of TV and Radio programs created nationwide as a result of USDA assistance Secondary; monitoring data; 3 0 21 Number of awareness raising posters distributed as a result of USDA assistance Secondary; monitoring data; 5000 26016 22 Number of students benefiting from campaign to promote literacy as a result of USDA assistance (This indicator measures the number of students who directly benefit from the literacy promotion campaign which would have been conducted with partners in the pilot province of Phongsaly) Secondary; monitoring data; 30000 15454 23 Number of schools receiving energy saving stoves as a result of USDA assistance Secondary; monitoring data; Primary Survey 768 1 (58) 48 24 Number of gardens established as a result of USDA assistance Secondary; monitoring data; 768 38 (58) 1406 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 127 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e Primary Survey 25 Number of cooks and storekeepers trained in commodity management as a result of USDA assistance Secondary; monitoring data; Primary Survey 9000 7601 26 Number of training and workshops provided on good health and nutrition as a result of USDA assistance Secondary; monitoring data; 225 8334 27 Number of target schools with improved food preparation and storage equipment as a result of USDA assistance Secondary; monitoring data; Primary Survey 1500 0 Kitchen-39 (58) Warehouse￾36 (58) 3490 28 Percent of students in target schools who start grade one and complete the last grade of primary school (cohort survival rate) as a result of USDA assistance Primary Survey but this cannot attribute the results to USDA assistance; Based on data availability 70 60 81.1 82.6 79.7 29 Percent of schools with library facilities Secondary; monitoring data; 40% (58) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 128 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e Primary Survey 30 Percent of schools with dedicated storage rooms Secondary; monitoring data; Primary Survey 98.2% (58) 31 Percent of store rooms rehabilitated/construc ted as a result of USDA assistance Secondary; monitoring data; Primary Survey 62% (58) 32 Percent of schools where food is stored off the ground Secondary; monitoring data; Primary Survey 74% (47) 33 Percent of school with a kitchen constructed as a result of USDA assistance Secondary; monitoring data; Primary Survey 67% (58) 34 Percent of schools using smoke reducing/Energy saving stoves Secondary; monitoring data; Primary Survey 2% (58) 35 Percent of schools with a dining area for the school meals constructed as a result of USDA assistance Secondary; monitoring data; Primary Survey 39.6% (58) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 129 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 36 Percent of storekeepers trained in safe food preparation and storage practices as a result of USDA assistance Secondary; monitoring data; Primary Survey Storekeeper s-65% (58) Cook- 55% (58) Teachers￾29% (298) 37 Percent of storekeepers received a book about Warehouse management in the Lao language within the last 12 months Secondary; monitoring data; Primary Survey 50% (56) 38 Percent of cook/food preparers trained in safe food preparation and storage practices as a result of USDA Secondary; monitoring data; Primary Survey SK-38 (58) Cook-32 (58) Teachers￾86 (298) 39 Percent of schools with a source of drinking area near or at school Secondary; monitoring data; Primary Survey 27.6% (58) 40 Percent of schools with toilet facility for students Secondary; monitoring data; Primary Survey 72.4% (58) 41 Percent of schools with separate toilet for female students Secondary; monitoring data; Primary Survey 42% (52) USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 130 S.N o Indicator Source Target Baseline values End-line values (Primary) End-line values (Secondary) Overall Male Femal e Overa ll Male Femal e Overall Mal e Femal e Overall Male Femal e 42 Number of farmers benefitting from developed partnership to supply food to school as a result of USDA assistance Secondary; monitoring data; Primary Survey 1500 948 43 Number of schools receiving food from farmer groups Secondary; monitoring data; Primary Survey 150 205 44 Amount of food provided to schools (dollar values) from farmer groups per semester Secondary; monitoring data; Primary Survey 20000 24644 USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 131 Annex S: Results from EGRA sub-tests Children’s background 1 In terms of ethnicity, children were classified into four main ethnic groups in Lao59 namely, Hmong￾Eiw Mien, Mon-Khmer, Lao-Tai, and Sino-Tibetan. While Mon-Khmer emerges as the predominant ethnic group in Northern regions (45%), its presence is comparatively lower in the South (34%). Lao-Tai is the most predominant ethnic group in the South with around 60 percent of students belonging to it. Children belonging to Lao-Tai ethnic group speak Lao at home and therefore, are less likely to face problems in school in terms of the medium of instruction. However, those belonging to other ethnic groups are learning to read in a second language. Further, a student belonging to ethnic groups Hmong-Eiw Mien and Sino-Tibetan were only found in the Northern provinces (7% and 16%, respectively), its presence in the South is negligible. Thus, we see that higher proportion of students in the southern provinces are comfortable with Lao as compared to those in the Northern provinces (non-Lao-Tai ethnic groups). 2 Comparing the end-line sample of students with that of the baseline, it appears that it was largely similar in terms of ethnicity with Mon-Khmer being the most predominant (59%), followed by Lao￾Tai (18%), Sino-Tibetan (13%), and Hmong-Eiw Mien (10%). 3 The sample includes an equal number of boys and girls, 52 percent boys and 48 percent girls. With respect to gender-based differences, boys and girls in the sample were found to be very similar in terms of socio-economic characteristics that were captured by the study. No statistically significant differences were found between boys and girls across both geographies. 4 The mean age of children was around 8.8, with no statistically significant variations across sex and geographical location. The mean age of students during the baseline was around 9. Children’s home literacy background 5 With respect to the proportion of students taking extra classes beyond school hours, there has been a sharp increase (statistically significant) over the project period. During the FY14 baseline study, only around 12 percent of students were taking extra classes while that has now increased to around 52 percent. A significantly (statistically) higher proportion of students in southern provinces reported that they are taking extra classes (64.1 %) as compared to 47.8 percent in northern provinces. 6 During the FY14 baseline study, only a little over half of the students (58%) in the sample reported having someone at home (not private teachers) to help them with reading and writing. With the efforts of the program and government support, this has now increased considerably and the majority (93%) students reported having someone at home to help them with their studies. This includes 92.4 percent male students and 89.1 percent female students. No statistically significant variation was observed when disaggregating the data by sex and geographical location. 7 MGD-SFP program also worked a great deal towards promoting a culture of reading among students through its various activities. As a result of this, the proportion of students reporting the reading of any kind of extracurricular book has gone up from 21 percent during baseline to around 68 percent during the end-line evaluation. The proportion of students who read such books was higher in the Northern provinces (73%) than the south (53%). 8 EGRA was designed as a method-independent approach to measuring a child’s initial reading ability. Through a series of tests, EGRA measures various skills that are necessary building blocks for children to read fluently and comprehend text. A summary of the various EGRA sub-tests administered during the primary survey for the end-line evaluation and their measurement approaches are given in the table below. 59As defined by the Lao Census 1995 and the classification of ethnic minorities based on Ethno-linguistic Family by LFNC USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 132 Table 26: EGRA sub-tests and their measurement approaches EGRA sub-test Measurement approach Letter recognition Number of Lao letters correctly identified. (A chart of 33 Lao letters were shown to every pupil, who was asked to read them.) Familiar word recognition Number of frequently used words correctly read. (50 commonly used words were shown to every pupil, who was asked to read them.) Invented-word reading Number of invented words (nonsense words) correctly read. (50 invented words were shown to students, who were asked to read them to the best of their ability). Oral reading fluency Number of words read correctly per minute on a grade-level passage to determine fluency. The number of words students could read correctly from the passage, irrespective of the time it took, was served as a measure of accuracy. (A passage was given to each student, who was asked to read it to the best of his/her ability.) Reading comprehension and listening Comprehension Number of comprehension questions answered correctly based on grade￾level passages (two passages) that the student read. If the student could not read the passages, it was read by enumerators and the same questions were asked to measure the listening comprehension ability. 9 The following table indicates statistically significant improvement among students over the project period on all the other five sub-tests (in addition to reading comprehension) as well. Table 27: Performance of students on other EGRA sub-tests, baseline-end-line comparison Sub-test N Overall Male Female North South Mean number of correct letters in 60 seconds (letter recognition) * BL FY14 830 11.7 11.6 11.9 - - EL FY 14 175 13.49 14.73 12.12 12.91 15.11 Mean number of correct familiar words in 60 seconds (Familiar word recognition) * BL FY14 830 2.9 2.5 3.3 - - EL FY 14 175 6.69 7.59 5.70 6.87 6.2 Mean number of correct invented words in 60 seconds (Invented word reading) * BL FY14 830 1.3 1.2 1.3 EL FY 14 175 2.83 2.98 2.66 2.87 2.72 Percentage of students who could read at least 35 words per minute* BL FY14 830 3% - - - - EL FY 14 175 10.9% 14.1% 7.2% 10.9% 10.9% The mean number of questions answered correctly for listening comprehension* BL FY14 830 1.6 1.7 1.6 - - EL FY 14 175 2.62 2.76 2.46 2.45 3.09 *- indicates a significant difference between baseline and end-line overall values The following inferences can be drawn from the table above: USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 133  An average student could correctly identify 13 out of the 33 letters within 60 seconds  An average student could correctly identify close to 7 familiar words within 60 seconds  An average student could correctly identify close to 3 invented words within 60 seconds  Basis the assumption that an average student should be able to correctly read at least 35 words per minute, it was found that only 11% of students in the sample could do so  An average student could correctly answer around 3 questions for listening comprehension USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 134 Annex T: Baseline and MTE recommendations and follow up Table 28: Baseline and MTE recommendations and follow up S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 R.1 WFP maintains a strong partnership with MoES to create a better teaching and learning environment and to develop strategies and take actions to improve primary school pupils’ reading and comprehension abilities MTE emphasised the need to further improve WFP collaboration with MoES in order to maximise the benefits of MGD-SFP and prepare GOL for handover. Therefore, it was recommended that WFP continues to build and maintain a strong partnership with MoES and other stakeholders, especially those involved in the TWG. WFP has been closely associated with MoES towards upscaling of school meals programme. It is also supporting the department for achieving the education targets for the country. In May 2018, WFP and MoES signed a joint School Feeding Handover Plan. Moreover, for the sustenance of the school gardens which is a critical complementary activity and towards ensuring a regular flow of funds for school meals programme, WFP has to work towards convergence between MoES, MAF and MPI. The convergence platform (could be setting up of school meals unit as a nodal implementing agency) would lead for successful implementation of national school meals programme in the country. R.2 The program should work closely with local communities and schools to mobilise contributions of food items to the school meal programme. Stakeholders, during the MTE, felt that there was scope for improving community engagement in the program. Strong community engagement was the key to sustainable school feeding and VEDCs were identified as pivotal towards achieving this. MTE recommended creating a model to strengthen community capacity development activities, principally targeting schools and VEDCs, to facilitate the growth of local ownership of school feeding and related activities and improved functionality to play a stronger role with school staff. MTE also recommended that the program should experiment with cash-based, local procurement models of school feeding, given Nearly all the schools in the sample (90%) reported that there was community contribution for school lunch through the supply of vegetables, eggs, meat, and contributing time for cooking, or labour for infrastructure creation. Around 80 percent of the VEDCs reported that they have been trained with the help of USDA assistance. It is important that the village head being the decision maker at the village level and also a part of the VEDC is definitely trained. A capacitated village head will assist in the successful implementation of the school meals programme by guiding the VEDC and the community towards achieving the goals. The program is setting up model schools that have well-functioning school gardens, greenhouses, fish ponds, etc. It is envisaged that this will help generate knowledge on dietary/ crop-diversity, climate change, low cost agricultural solution among students and communities. Additionally, this will also lead to economic benefits that will provide an incentive to the VEDC and community members. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 135 S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 that sustainability of in-kind approaches have been found to be questionable. A pilot on cash-based transfers has been initiated in Beng district of Oudomxay province and Nalae district in LuangNamtha. R.3 Improve water and sanitation facilities at school- separate toilets for boys and girls, safe source of drinking water near or at school. It was recommended that WFP should work in collaboration with partners identified in the results framework towards this. MTE findings suggest that there were noticeable gaps in WASH infrastructure that needed to be addressed. Therefore, MTE recommends WFP, along with UNICEF, to advocate for WASH activities which are an important complementary activity. The evaluation team observed good coverage of toilet facilities and separate toilets for boys and girls in nearly all schools. WFP collaborated with UNICEF for constructing toilets and GIZ for constructing handwashing stations. However, lack of availability of water is affecting usage of WASH facilities. A very small proportion of schools (28%) had access to improved source of drinking water. Thus, water availability- for drinking, cooking and cleaning, remains a challenge in most schools. R.4 Increased training coverage for cooks and storekeepers in order to improve the hygienic condition of food served to pupils, to ensure proper stock management, and to improve the record keeping of food utilisation in schools. MTE observed that the need for capacity development remains. There emerged a strong need to increase understanding of education and nutrition activities among government officials and community members. It was recommended that WFP and its partners strengthen capacity building. For building capacity at the national level, WFP has facilitated an exchange programme for MPI, MoES and MAF officials to Brazil. The visit has encouraged the departments in detailing out a clearer oversight of roles and responsibilities at the national, provincial and district level. WFP, in collaboration with MoES, is also attempting to set up School Meals Centre of Excellence. Other exchange visits for government officials have been initiated since October 2017. WFP is also planning to conduct joint training to build capacities of MoES, MoH and MAF officials for providing technical assistance to communities on school agriculture and nutrition, education, hand-washing and hygiene, using a ToT model. Capacity development activities for VEDCs are also being conducted to deepen the understanding of the community led school meals programme. End-line evaluation findings suggest that there still remain significant gaps in terms of imparting training to cooks and storekeepers. In the initial stages of the programme, there were 1-2 cooks in every village who weretrained on food preparation and storage techniques. However, this was soon increased to multiple cooking groups per village. On an average, now there are about 15-20 cooks in every village. However, most of the USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 136 S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 new cook volunteers have not been trained- nearly 50 percent of the cooks in the end-line sample were untrained. Similarly, around one-third storekeepers have not received any training so far. It was shared that many of the appointed storekeepers, who had been trained in commodity management and record keeping, had left their jobs now. As a result, new storekeepers were appointed but they have not been trained yet. This, in turn, is also leading to poor record keeping since they are not very familiar with the formats and processes. Thus, training coverage needs to be expanded to the new cooks and storekeepers, along with refresher trainings for those trained earlier to aid better recall. R.5 Student knowledge of nutrition, good health, and hygiene practices were found to be very poor. It was recommended that WFP should work with the government and partners for proper implementation of activities (e.g. raising awareness on nutrition and hygiene, training on good health and nutrition practices, production of supplementary reading materials etc.) identified in the resulting framework. MTE concluded that collaboration with GOL and other development partners is essential for successful implementation of all aspects of the operation. It was recommended that WFP should work with other partners to advocate for and assist the strengthening of GOL monitoring systems. Model schools were to be developed to strengthen community capacity development activities. This would further facilitate increased community ownership towards the school feeding program. It was also recommended that MGD-SFP should continue to engage with TWG and the Donor Coordination Group to advocate for and assist the strengthening of MoES monitoring systems and support their attention to Gender. For promoting life-skills and improved nutrition through awareness, WFP is working towards integrating nutrition and school agriculture with curriculum for primary education. Based on the results from a pilot implementation of the integrated curriculum, the revised curriculum will be made for nationwide deployment. WFP will continue its engagement with Laos National Assembly members. As part of this engagement, WFP will facilitate field trips for National Assembly members to both the MGD and NSMP supported schools. These visits along with workshops would help in understanding the nuances of school feeding, school agriculture and nutrition which in turn would support the handover process and upscaling of the school meals to all schools. School gardens have been recognised to meet the dual objectives of nutrition education as well as providing ingredients for school meals. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 137 S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 R.6 Strengthen school level record keeping and maintenance of quality data for enrolment and food receipts. MTE observed a limited evidence base for program results. It was identified that this was primarily due to weaknesses in reporting against large (and often changing) set of performance indicators. MTE recommended further improvements in monitoring systems, esp. in the table used for reporting against plans and targets. School records on enrolment, attendance, dropout, repetition, and food utilisation was found to be very limited across schools as well as at the district and provincial levels. This emerges as an issue across most South East Asian countries. Semi-annual reports submitted by WFP to USDA from October 2016 onwards are following newly improved formats. However, as discussed in earlier sections, there remains scope for further improvement. As part of the monitoring system, WFP will sign a MoU with Lao Front for National Development for supporting monitoring activities of the school meals programme. This will provide an opportunity to validate the numbers being provided by schools. This will also help in ironing out lacunae that currently exists in providing information on school meals. R.7 Adapt WFP’s deployment of local-level personnel so as to provide more effective support to meaningful capacity development at community level. To ensure sustainability of SFP, WFP has drawn strategy that focusses on building capacities of VEDCs. VEDCs are expected to play the pivotal role of providing guidance and ensuring support to SFP after the completion of the MGD programme. To empower the VEDC members, capacity development activities like training, workshops and exchange programmes will be conducted. Under exchange programmes, VEDC members will visit NSMP supported schools that will help them understand the implementation of NSMP and draw lessons from the challenges encountered and the solutions adopted. It is important that the village head being the decision maker at the village level and also a part of the VEDC is definitely trained. A capacitated village head will assist in the successful implementation of the school meals programme by guiding the VEDC and the community towards achieving the goals. To complement this approach, WFP will be supported by LWU who will train women at village level so that they have equal participation and contribution towards the implementation of school meals in the schools. USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 138 S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 R.8 While school feeding caters to a wide range of educational, social protection and nutrition￾based objectives, MTE recommended that WFP’s role in directly supporting complementary activities (such as literacy programmes) should be reconsidered. This is because these are not linked to WFP’s core competencies. Moreover, given the target to handover schools in 2020, WFP should focus its efforts on activities directly related to the delivery of MGD-SFP. The end-line evaluation finds merit in the activities planned under MGD￾SFP and using the school as an entry level for implementing these activities. Discussions with parents reflect that they are happy that the food needs of the children are being met and they are spending more time in school. This directly relates to attendance and reduced absenteeism. Supporting teachers with improved teaching and instruction materials has helped in improving the quality of education. The availability of colourful and age-appropriate books in schools have created the eagerness among children to learn and know more. The literacy activities under the programmes have helped in diffusing the importance of education and nutrition among the communities. This has a direct bearing on the dietary diversity and educational support to children at home. R.9 MTE found that nutrition education was a weaker element of the program and needs to be strengthened further. This would promote understanding by the community of the minimum nutritional standards for the food basket. It was recommended that ‘model schools’ with effective nutrition education should be highlighted and used for peer learning. MTE further recommended that WFP should collaborate with MoES, MoH and MAF to review experience and seek a better balance between using school gardens (a) for educational purposes and (b) to provide ingredients for school meals. It has been recognised that while school gardens may not be able to produce adequate quantities of food to support the school meals, they play an important role in educating children about nutrition and agriculture. Exposure visits to model schools will be helpful in creating the understanding of maintaining the school gardens. This also has to be supported by ensuring water sufficiency for the gardens. In parallel, WFP can engage with MAF in promoting kitchen garden. The similar nature of activities at school and at home will build capacities around nutrition education. The added advantage is that families will be self-reliant for providing vegetables for school meals and will also help in maintain dietary diversity at home also. R.10 In order to continue to mainstream focus on gender into field level activities and teaching materials, MTE recommended collaboration with partners for focussing on issues of Gender equality is yet to be achieved in certain activities like cooking and commodity management. The end-line evaluation finds that women are loaded with an unfair share of support activities such as collecting firewood, collecting vegetables and seasoning, fetching water, washing USDA Mc-Govern Dole FY 14End-line Evaluation in Lao PDR Final Report 139 S.No Baseline FY14 Mid-term evaluation FY14 Findings from the end-line evaluation FY14 gender. There was a felt need to sharpen gender focus around activities such as using new literacy materials and reading times to challenge unsuitable gender roles and improve sensitivity. It was also felt that the use of lunch-break for gender-specific activities needed to be investigated. and maintaining hygiene of the kitchen and dining room, by virtue of them providing their services as cooks. Operationally, more men were found to be involved with commodity management (storekeepers) while women were in charge of cooking. This reiterates the prevalent gender roles in society and is likely to have a counter-effect in terms of impacting conventional gender roles in the long term. Moreover, lunch-breaks in most schools were not being utilised for any activity. After having lunch, teachers go back home to have food (since school meals are not provided to teachers). As a result, children too were going back home. The end-line evaluation finds that the literacy activities under the programme targets girls, boys and women for promoting the importance of education and nutrition. With women being exposed to such information, it is expected to improve dietary practices at home. As a result of this, children, as well as pregnant and lactating mothers, will have access to more nutritious food. [WFP Lao PDR Country Office] take place [New Delhi, October 2108]