DECENTRALIZED EVALUATION FOR EVIDENCE-BASED DECISION MAKING WFP EVALUATION Agreement #FFE-696-2015/007-00 WFP Rwanda November 23, 2021 WFP's USDA McGovern-Dole International Food for Education and Child Nutrition Program's Support in Rwanda 2016-2021 Endline Decentralized Evaluation Key personnel for the evaluation WFP Rwanda Evaluation Manager Sarah Cruz PREPARED BY Bruce Ravesloot, Team Leader Jeanne Downen, Senior Evaluator Justin Tuyiringire, National Evaluator Daria Muteteri, National Evaluator Juventine Mujawase, National Evaluator Christine Kayiraba, National Evaluator Monica Mueller, Research Analyst/ Quality Assurance Manager Lloyd Banwart, Data Analyst Jeremie Kaelin, Research Associate Acknowledgements The Evaluation Team would like to acknowledge the valuable assistance received from the colleagues at the World Food Programme in Rwanda and thank them for their time, inputs and flexibility to complete the evaluation in complex and challenging circumstances. At WFP Rwanda we thank Sarah Cruz, Head of M&E, for her support and dedication as evaluation manager, and Amy Blauman, Programme Policy Officer for the Home Grown School Feeding Programme, who served as focal point from inception through the fieldwork and reporting phases. We further thank Edith Heines, Representative and Country Director, and all the management staff who gave valuable time and feedback. All extended their support and hospitality to the team and assisted us with our many requests. In addition, we extend our appreciation to the Government of Rwanda officials, United Nations partners, WFP cooperating partners and the donors, who all provided valuable input. The team at World Vision Rwanda were generous with their time and expertise, contributing to the development of the literacy assessment tool, training the field team on its application, and adding insight to assessment results. Finally, we thank the communities and households who shared their views and experiences with us. It is our sincere hope that the people with whom WFP and the Government work will benefit from this evaluation and experience positive gains in their lives as a result of implementing its recommendations. Disclaimer The opinions expressed in this report are those of the evaluation team, and do not necessarily reflect those of the World Food Programme or USDA. Responsibility for the opinions expressed in this report rests solely with the authors. Publication of this document does not imply endorsement by WFP or USDA of the opinions expressed. The designation employed and the presentation of material in maps do not imply the expression of any opinion whatsoever on the part of WFP or USDA concerning the legal or constitutional status of any country, territory, or sea area, or concerning the delimitation of frontiers. This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this document can be made available by contacting fas.monitoring.evaluation@usda.gov Contents Executive Summary ..........................................................................................................................................i 1. Introduction..................................................................................................................................................1 1.1 Evaluation features ............................................................................................................................................1 1.2 Context................................................................................................................................................................2 1.3 Subject being evaluated.............................................................................................................................7 1.4 Evaluation methodology, limitations, and ethical considerations .............................................................13 2. Evaluation Findings.................................................................................................................................... 17 2.1 Relevance ..........................................................................................................................................................17 2.2 Effectiveness .....................................................................................................................................................22 2.3 Efficiency ...........................................................................................................................................................36 2.4 Impact and sustainability ................................................................................................................................37 2.5 Factors affecting the results ...........................................................................................................................39 3. Conclusions and recommendations......................................................................................................... 45 3.1 Conclusions.......................................................................................................................................................45 3.2 Recommendations...........................................................................................................................................50 Annex 1: Stakeholder Analysis ..................................................................................................................... 57 Annex 2: Evaluation Timeline ....................................................................................................................... 59 Annex 3: Government Policies and Priorities Related to Food Security, Social Protection and Education ........................................................................................................................................................62 Annex 4: Map ..................................................................................................................................................64 Annex 5: Results Framework ........................................................................................................................ 66 Annex 6: Outcome and Outputs Tables....................................................................................................... 70 Annex 7: Terms of Reference........................................................................................................................ 97 Annex 8: Data Collection Tools ................................................................................................................... 119 Annex 9: Evaluation Matrix......................................................................................................................... 124 Annex 10: Methodology............................................................................................................................... 134 Annex 11: List of People Interviewed......................................................................................................... 145 Annex 12: Field Schedule............................................................................................................................. 147 Annex 13: Supplementary Tables: EGRA Survey Results ......................................................................... 148 Bibliography..................................................................................................................................................153 Acronyms ......................................................................................................................................................156 List of Figures Figure 1: Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text ..........................................................................23 Figure 2: Number of students regularly (80%) attending USDA- supported classroom/ school.......................28 Figure 3: McGovern-Dole Programme coverage in Rwanda...................................................................................65 List of Tables Table 1: Percentage of food insecure households, by district, 2015 and 2018..................................................... 2 Table 2: Attendance pre- and post- COVID-19 school closures............................................................................... 7 Table 3: McGovern-Dole Programme overview ........................................................................................................ 7 Table 4: Commodity distributions, expected v. actual ...........................................................................................12 Table 5: OECD-DAC criteria and evaluation questions...........................................................................................13 Table 6: Key evaluation questions ............................................................................................................................14 Table 7: Reading performance for third-grade students at endline.....................................................................23 Table 8: Reading comprehension, third grade (all students to read and/or listen)............................................24 Table 9: Student health and hygiene knowledge, baseline, midterm and endline.............................................31 Table 10: Midterm evaluation recommendations1 and actions............................................................................39 Table 11: Stakeholder analysis and mapping..........................................................................................................57 Table 12: Timeline of tasks and deliverables, by phase .........................................................................................59 Table 13: Government policies and priorities related to food security, social protection and education.......62 Table 14: Number of individuals benefiting directly from USDA-funded interventions (cumulative figures).70 Table 15: Number of individuals benefiting indirectly from USDA-funded interventions (cumulative figures) .......................................................................................................................................................................................70 Table 16: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (cumulative figures)................................................70 Table 17: Number of teachers in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance (cumulative figures) ............................................................71 Table 18: Number of teachers in target schools who attend and teach school at least 90 percent of scheduled school days per school year (cumulative figures) ................................................................................71 Table 19: Number of teachers receiving awards (annual figures) ........................................................................72 Table 20: Number of textbooks and other teaching and learning materials provided as a result of USDA assistance (cumulative figures) .................................................................................................................................72 Table 21: Number of textbooks and other teaching and learning materials (literacy starter kits) provided as a result of USDA assistance (annual figures)...........................................................................................................72 Table 22: Number of students benefiting from literacy starter kits provided as a result of USDA assistance (annual figures) ...........................................................................................................................................................73 Table 23: Number of target schools with supplemental reading materials available to students as result of USDA assistance (cumulative figures) ......................................................................................................................73 Table 24: Number of teaching and reading materials procured as a result of USDA assistance to be used in schools (annual figures) .............................................................................................................................................73 Table 25: Number of books distributed in communities to establish school and community libraries (annual figures) .........................................................................................................................................................................73 Table 26: Number of community-generated reading materials (annual figures) ...............................................74 Table 27: Number of students benefiting from libraries and new reading materials provided as a result of USDA assistance (annual figures) .............................................................................................................................74 Table 28: Increased skills and knowledge of teachers (cumulative figures)........................................................74 Table 29: Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance (annual figures) ...............................75 Table 30: Increased skills and knowledge of school administrators (cumulative figures).................................75 Table 31: Number of school administrators and officials in target schools who demonstrate use of new techniques and tools as a result of USDA assistance (annual figures).................................................................76 Table 32: Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance (cumulative figures)..................................................................................................................76 Table 33: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (cumulative figures) ......................................................................................................................77 Table 34: Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (cumulative figures)..................................................................................................................77 Table 35: Number of daily school meals provided to school age children as a result USDA assistance (annual figures) .........................................................................................................................................................................77 Table 36: Number of school-aged children receiving daily school meals as a result of USDA assistance (annual figures) ...........................................................................................................................................................78 Table 37: Number of students enrolled in schools receiving USDA assistance (annual figures) ......................78 Table 38: Number of individuals benefiting directly from USDA funded interventions (annual figures) ........78 Table 39: Number of individuals benefiting indirectly from USDA funded interventions (annual figures) .....79 Table 40: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (annual figures) .............................................................................................................................79 Table 41: Number of students regularly (80%) attending USDA supported classrooms/schools (annual figures) .........................................................................................................................................................................80 Table 42: Percent of school-age children receiving a minimum acceptable diet (male) (annual figures)........80 Table 43: Number of public-partnerships formed as a result of USDA assistance (other) (annual figures)....80 Table 44: Number of smallholder farmers benefiting from partnerships with schools (annual figures) ........80 Table 45: Value of public and private sector investments leveraged as a result of USDA assistance (private) (annual figures) ...........................................................................................................................................................81 Table 46: Number of school gardens established and maintained (annual figures)..........................................81 Table 47: Number of students benefiting from the establishment and maintenance of school gardens (annual figures) ...........................................................................................................................................................81 Table 48: Number of students regularly (80%) attending USDA supported classroom/school (annual figures) .......................................................................................................................................................................................82 Table 49: Percent of students who miss more than 10 school days per year due to illness (cumulative figures) .........................................................................................................................................................................82 Table 50: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) (cumulative figures) ........................................82 Table 51: Number of educational facilities (i.e. school buildings, classroom, and latrines) rehabilitated/constructed as result of USDA assistance (latrines) (cumulative figures) ....................................82 Table 52: Number of fuel-efficient stoves provided and rehabilitated (annual figures)....................................83 Table 53: Number of individuals directly benefiting from the provision and rehabilitation of fuel-efficient stoves (annual figures) ...............................................................................................................................................83 Table 54: Number of students enrolled in school receiving USDA assistance (cumulative figures).................84 Table 55: Percent of parents in target communities who can name at least three benefits of primary education (cumulative figures)..................................................................................................................................84 Table 56: Percent of parents in target communities who can name at least three benefits of primary education (annual figures).........................................................................................................................................84 Table 57: Number of parents trained on the importance of literacy (annual figures) .......................................85 Table 58: Number of reading clubs established as a result of USDA assistance (annual figures)....................85 Table 59: Number of student writing competitions facilitated as a result of USDA assistance (annual figures) .......................................................................................................................................................................................85 Table 60: Number of students benefiting from the development of reading clubs (annual figures) ..............86 Table 61: Percent of students who, by the end of two grades primary schooling, demonstrate that they can read and understand the meaning of grade level text (annual figures) ..............................................................86 Table 62: Number of SGACs or similar school governance structures supported as a result of USDA assistance (annual figures) ........................................................................................................................................86 Table 63: Number of parents trained as part of SGACs (annual figures).............................................................86 Table 64: Percent of students who can identify at least three key health and hygiene practices (cumulative figures) .........................................................................................................................................................................87 Table 65: Number of students reached with health and hygiene messages as a result of USDA assistance (annual figures) ...........................................................................................................................................................87 Table 66: Number of individuals trained in child health and nutrition as a result of USDA assistance (annual figures) .........................................................................................................................................................................87 Table 67: Percent of cooks and storekeepers who can identify at least three safe food preparation and storage practices (cumulative figures) .....................................................................................................................88 Table 68: Number of cooks and storekeepers trained on food preparation and storage practices (annual figures) .........................................................................................................................................................................88 Table 69: Number of individuals trained in child health and nutrition as a result of USDA assistance (cumulative figures) ....................................................................................................................................................89 Table 70: Number of schools using an improved water source and sanitation facilities (cumulative figures)89 Table 71: Number of students benefiting from newly constructed/rehabilitated latrines (annual figures)....89 Table 72: Number of schools with improved sanitary facilities (annual figures) ................................................89 Table 73: Number of rainwater catchment systems constructed and/or enhanced (annual figures) .............90 Table 74: Number of students benefiting from newly constructed and/or enhanced rainwater catchment systems (annual figures) ............................................................................................................................................90 Table 75: Number of schools using an improved water sources (annual figures) .............................................90 Table 76: Number of students receiving deworming medication(s) (cumulative figures) .................................91 Table 77: Number of students receiving deworming medications (annual figures) ..........................................91 Table 78: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) (cumulative figures) ........................................91 Table 79: Number of educational facilities (i.e school buildings, classrooms and latrines) rehabilitated/constructed as a result of USDA assistance (kitchens, cook areas) (annual figures) ..................92 Table 80: Number of educational facilities (i.e school buildings, classrooms and latrines) rehabilitated/constructed as a result of USDA assistance (other school grounds or school buildings, i.e. storeroom) (annual figures).......................................................................................................................................92 Table 81: Number of students benefiting from kitchens, cook areas and storerooms built or rehabilitated (annual figures) ...........................................................................................................................................................92 Table 82: Number of government staff trained or certified as a result of USDA assistance (cumulative figures) .........................................................................................................................................................................93 Table 83: Number of government staff trained at national level (annual figures) .............................................93 Table 84: Number of government staff trained at district level (annual figures)................................................93 Table 85: Improved policy and regulatory framework (cumulative figures)........................................................94 Table 86: Number of HGSF Steering Committee and Technical Committee meetings held (annual figures) .94 Table 87: Number of government financing strategies developed (annual figures)..........................................94 Table 88: Number of government monitoring and evaluation systems established (annual figures).............94 Table 89: Number of district and national-level Ministry of Education (MINEDUC) and Ministry of Agriculture (MINAGRI) staff supported by new HGSF policies (annual figures)......................................................................95 Table 90: Number of child health and nutrition policies, regulations or administrative procedures in the following stages of development as a result of USDA assistance (stage 5) (annual figures).............................95 Table 91: Number of educational policies regulation, or administrative procedures in the following stages of development as a result of USDA assistance (stage 5) (annual figures) ..............................................................95 Table 92: Increased government support (annual figures) ...................................................................................95 Table 93: Increased engagement of local organizations and community groups (cumulative figures)...........96 Table 94: Anticipated methodological limitations and mitigation strategies; comments post-evaluation....139 Table 95: Ethical issues and safeguards implemented ........................................................................................143 Table 96: Key informant interviews conducted.....................................................................................................145 Table 97: Focus group discussions conducted......................................................................................................145 Table 98: Field schedule...........................................................................................................................................147 Table 99: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text.................................................................................148 Table 100: Reading, listening and comprehension of short story, midterm and endline................................148 Table 101: Percent of third grade students in words-correct-per-minute (WCPM) range, midterm and endline........................................................................................................................................................................148 Table 102: Reading and comprehension at endline, by district, midterm and endline ...................................148 Table 103: Students' home support and time allocation, midterm and endline ..............................................149 Table 104: Percentage of students in classroom identified as attentive by their teachers .............................149 Table 105: Student attentiveness, by grade; baseline, midterm, and endline ..................................................149 Table 106: School administrator use and application of teaching techniques, midterm and endline...........149 Table 107: Sources of food for school meals, midterm and endline..................................................................150 Table 108: Parents' knowledge of education benefit and storekeepers' knowledge of good practices; baseline, midterm and endline ...............................................................................................................................150 Table 109: School SGAC engagement, midterm and endline..............................................................................150 Table 110: Health and hygiene practices at endline, by district, midterm and endline ...................................151 Table 111: Health and hygiene practice utilization/application, midterm and endline ...................................151 Table 112: School water source and availability, midterm and endline.............................................................151 Table 113: External support to schools, midterm and endline ...........................................................................152 Date | Report Number i Executive Summary INTRODUCTION 1. This is the final evaluation report of the World Food Programme’s (WFP) United States Department of Agriculture (USDA) McGovern-Dole International Food for Education and Child Nutrition (“McGovern-Dole”) Programme’s Support in Rwanda (2016-2021).1, 2 This activity evaluation covers Phase I of the programme from January 2016 to March 2021 and all targeted districts. The evaluation was commissioned by WFP Rwanda. PURPOSE, OBJECTIVES, INTENDED AUDIENCE, AND CONTEXT 2. The evaluation’s objectives are accountability and learning. It examines the project’s impact and identifies lessons for future programming, including the baseline evaluation and design of Phase II (2021-2025). The key evaluation questions are: 1) Have literacy rates of school-age children improved over the duration of the programme? 2) Has the use of health and dietary practices increased? 3) What is the level of community involvement and participation in decision-making in school governance mechanisms? and 4) What are the key institutions and governance structures required to effectively deliver, implement, and sustain school meal interventions? 3. The evaluation applies OECD-DAC assessment criteria and reports history and end-of-project status of USDA-required indicators. 4. The primary users of the evaluation are WFP Rwanda and its partners, World Vision, Gardens for Health International, and Rwanda Biomedical Centre, to understand programme performance and obtain insights to inform future design; the Rwanda Ministry of Education (MINEDUC), Ministry of Agriculture and Animal Resources (MINAGRI), and district governments, to understand programme performance and alignment with government priorities; USDA, to assess programme performance, findings and lessons to inform other McGovern-Dole programmes; and WFP Regional Bureau Nairobi, WFP headquarters, WFP Office of Evaluation, and the WFP Executive Board, for wider organizational learning and accountability. 5. Rwanda is a small, densely populated country with a growing economy (though COVID-19 generated some economic setbacks) and a vision to become an upper-income country by 2050. It has made significant gains in reducing poverty but continues to experience high levels of food insecurity, malnourishment, and stunting, especially in rural areas. Nearly all children are enrolled in primary school; girls have a slightly higher enrolment rate than boys (98.0 percent and 97.3 percent, respectively) 3 and also a higher primary completion rate (101.8 percent and 89.0, respectively). 4 The COVID-19 pandemic resulted in school closures in March 2020; schools re-opened in November 2020 for grades 4-6, and in mid-late February 2021 for all grades. EVALUATION SUBJECT 6. WFP implemented Phase I (2015-2021) of the McGovern-Dole Programme with the original aims of supporting the Government to develop a national school feeding programme and to build government 1 The program in Rwanda is hereafter referred to as the “McGovern-Dole Programme.” 2 The terms of reference refer to the McGovern-Dole grant as the WFP Rwanda HGSF. The two terms are used interchangeably and in context, as there are cases where we mean to highlight the home-grown school feeding aspect. 3 Republic of Rwanda, Ministry of Education. 2018. Education Sector Strategic Plan, 2018/19-2023/24. 4 National Institute of Statistics of Rwanda. 2020. Rwanda Statistical Yearbook 2020. Secondary completion rates not reported. Date | Report Number ii capacity for a handover of activities by 2020. The Government officially named WFP as its main partner in school feeding in 2019. The focus of the Phase I programme, designed in 2015, has shifted from a handover of the programme to building the capacity of Phase I schools to serve as model schools for the national programme, and then to transition those schools into the national school feeding programme in 2023/24.5 This change followed the Government’s approval and funding of a national school feeding programme to begin in 2021. At present, the Government does not have the budgetary resources to adopt all aspects of the McGovern-Dole model nationally. The longer transition time will allow the Government to identify additional funding sources and more cost-efficient measures to support the national programme. 7. The McGovern Dole strategic objectives are (1) improved literacy of school-aged children and (2) increased use of health and dietary practices. The programme focuses on school feeding and education; water, sanitation and hygiene; health and nutrition; and capacity strengthening. It provides hot meals to primary school students, meeting a significant portion of a child’s daily nutritional requirements while providing an incentive to attend school. It supports an improved education environment, providing kitchen and sanitation infrastructure, training of school staff, literacy clubs, training on school gardens and nutrition, engagement of parents in education, strengthening the capacity of agricultural cooperatives to supply meal ingredients, modeling of low-cost meals from local foods, and delivering technical assistance to the Government on school feeding policies, strategies, and guidelines for a national programme. 8. The McGovern-Dole Programme targets 108 schools in four districts with some of the highest rates of poverty, food insecurity, and stunting in the country: Karongi, Rutsiro, Nyaruguru, and Nyamagabe.6 It is implemented by WFP in partnership with MINAGRI, MINEDUC, World Vision, Gardens for Health International, Rwanda Biomedical Centre, and district governments. Its implementation period (including three no-cost extensions) is October 1, 2015 – November 30, 2021. McGovern Dole funded the programme at $25 million with additional funding from the MasterCard, FEED and Caterpillar Foundations. METHODOLOGY 9. The evaluation employed a mixed-methods approach, using secondary and primary data to assess performance against targets, and analyzing the factors that affected performance, including the impact of COVID-19. Methods included a desk review; school survey and Early Grade Reading Assessment (EGRA) in 104 schools, semi-structured interviews with key informants and focus groups, observation of programme activities, and triangulation of data from WFP and partners. National and subnational lines of inquiry focused on programme alignment with government strategies and policies, transition planning, and strengthening government capacities for the national school feeding programme launching in 2021. School-level data collection focused on assessing achievements against targets and the quality and impact of activities. The main methodological limitation was that some of data from the EGRA baseline (conducted in June 2016) had limited comparability to the data from the midterm (conducted September-October 2018) and endline, owing to the presence of non-response data in the reading comprehension questions at baseline; the report thus primarily compares midterm and endline results. KEY FINDINGS: RELEVANCE 10. The McGovern-Dole Programme is highly relevant to the needs and goals of schools in some of Rwanda’s poorest areas. It supports improved learning outcomes and infrastructure and complementary activities to achieve those outcomes. The programme aligns with government strategies and priorities in education, food and nutrition, and school health; with WFP corporate objectives; and with United Nations Sustainable Development Goal (SDG) 2 (Zero Hunger), SDG 4 (Quality Education), and SDG 17 (Partnerships for Goals). 5 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Plan of Operations and Activities. 6 After the data collection phase, the program received a no-cost extension through September 2021 that added an additional school, bringing the total number of schools to 108.The evaluation does not extend to the additional school. Date | Report Number iii WFP’s work at the national level is highly relevant as WFP has identified where the McGovern-Dole Programme contributes to a government framework for a national school feeding programme. KEY FINDINGS: EFFECTIVENESS Key EQ 1: Have literacy rates of school-age children improved over the duration of the program? 11. A key outcome indicator is the percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text. The endline EGRA tested third-grade students to maintain comparability with the baseline. The results show that third-grade students improved or retained their reading performance and comprehension skills. Key informant and focus group responses indicate that community reading volunteers were the main reason for continued improvement during COVID-related school closures. Supportive learning environments at home – such as encouragement and assistance from parents and older siblings, and having reading materials available in the home – were viewed as contributing to students’ reading skills even before the pandemic. 12. The evaluation methodology does not allow conclusions regarding the direct attribution of student literacy results to programme activities; numerous variables outside the programme’s control could affect student outcomes. Nevertheless, based on qualitative data and the evaluation team’s assessment, programme activities designed to improve the quality of literacy instruction were both well-suited to the objective and indeed contributed to the observed improvements in student reading skills. These activities included providing a print-rich learning environment in schools, training teachers in new methods, and encouraging consistent teacher attendance. In addition, the training of head teachers and district and sector education officers on teacher coaching methods, and the training of administrators on school management, were considered highly beneficial, especially for school management. Providing schools with supplemental literacy material, and establishing reading clubs in school catchment areas supported by facilitators trained in the “Literacy Boost” approach, were also considered factors in instructional effectiveness. 13. Key informants noted the growing involvement of parents in various school activities, and a corresponding increased appreciation among parents of the importance of education, which supports student attendance and learning. The increase in girls’ attendance is of special note, and in addition to the factors noted above, may be related to the construction of girls’ sanitation rooms on school grounds. Key EQ 2: Has the use of health and dietary practices increased? 14. Health and hygiene infrastructure improved at schools, with improved water supply, latrines, girls’ sanitation rooms, and kitchen infrastructure in place at all schools. Due to the pandemic, permanent handwashing facilities were introduced. The programme has trained all parents on home health and hygiene practices. Qualitative interviews revealed few gains in knowledge among students and cooks on good practices though there is evidence that these practices are used: the percentage of students who can identify at least three key health and hygiene practices improved only slightly, from 49.2 percent at midterm to 52.9 percent at endline, 7 though students were observed washing their hands before entering class. The ability of cooks and storekeepers to identify at least three safe food preparation and storage practices declined. This was attributed to lack of practice during school closures, few refresher trainings, and turnover. Nevertheless, cooks were observed to use good personal and food preparation hygiene, though they were not always able to identify the good practices. Key EQ 3: What is the level of community-level involvement and participation in decision￾making in school governance mechanisms? 15. Head teachers and school committees state that parent engagement and understanding of the importance of education, along with parent trust in schools, is increasing. Three-quarters of parents (74.3 percent) 7 Source: midterm and endline EGRA surveys Date | Report Number iv could name at least three benefits of primary education at endline, a small increase since midterm 69.8 percent).8 School General Assembly Committees were strengthened through training on governance, school management, and conducting community dialogues. The level of parent contributions to the cost of school meals varies widely among schools and affects some schools’ ability to meet expenses. Key EQ 4: What are the key institutions and governance structures required to effectively deliver, implement, and sustain school meal interventions? 16. The Government officially named WFP as its main partner in school feeding in 2019. The partnership between the Government and WFP technical support has resulted in substantial gains in the capacity of local and national institutions in the design, management and implementation of school feeding. A national School Feeding Technical Working Group was established in 2019, and MINEDUC has institutionalized a School Feeding unit and a National School Feeding Steering Committee to guide the roll-out of a national programme in 2021. Stakeholders view WFP as a good partner, and there is strong collaboration with government institutions and non-governmental organization (NGO) partners. Partner capacity is strong, as evidenced by their delivery of results against targets; for example, World Vision’s work on WASH infrastructure and literacy has been a critical component of programme success. Key Findings: Efficiency 17. WFP and partners have achieved results in an efficient and timely manner. At the national level, the programme’s focus on technical assistance and capacity strengthening has been effective in assisting the Government to develop the skills for management, coordination, and oversight of a national programme. At the local level, the programme has effectively encouraged timely attendance and student retention, supporting improved teaching and learning outcomes and better school management, increased parent awareness and engagement on the importance of education, and resulted in greater knowledge about nutrition and the potential of school and kitchen gardens in communities. KEY FINDINGS: IMPACT AND SUSTAINABILITY 18. The sustainable impact of the McGovern-Dole Programme lies in its significant contributions to government policy and strategies for a national school feeding programme, and in the Government’s institutionalization of school feeding within government staffing structures. 19. The programme’s influence on national approaches to gender equality is reflected in the Government’s recognition of the value of girls’ sanitation rooms and the work to develop meals that meet adolescent girls’ nutritional needs. There is scope for greater gender sensitivity in the programme, and potentially for the scaled-up national programme. FACTORS AFFECTING THE RESULTS 20. A key internal factor influencing results is the quality, capacity and commitment of WFP staff, especially the dedication to learning and programme strengthening in its second half. The leadership, systems and processes that shape the programme have been greatly strengthened and WFP staff have grown in strategy, policy and facilitation skills. This has fostered strong partnerships with the Government, and the programme’s strategic relevance is now recognized. Improved coordination, communication, and role clarity within WFP Rwanda, expanded staffing, and greater support from technical units have increased the programme’s effectiveness and its credibility with Government. Government officials credit WFP’s willingness to collaborate and to support government priorities with engendering joint ownership of the programme. Partner inclusion is stronger, activities are better integrated, and operational collaboration remains strong. A procurement model that strengthens the capacity of local farmers to supply school meals is aligned with government priorities and is making good progress. Programme managers adapted activities during COVID-19-related school closures to continue its operations, including organizing take￾home rations for students, adopting remote training modalities for school personnel, providing literacy 8 Source: midterm and endline head teacher surveys. Date | Report Number v outreach to homes, extending gardening activities to communities, and addressing outstanding school infrastructure needs. 21. Another internal factor that supported strong results was the country office’s willingness to act on the recommendations from the midterm evaluation. At midpoint, WFP and the Government were building the systems needed for a sustainable school meals programme. The midterm evaluation noted the opportunity to more deliberately apply programme experience to support the Government in developing the national school feeding programme. This required WFP to better organize itself internally, better present programme results to the Government, and have a more structured approach to government partnership and capacity strengthening. 22. Key external factors favoring success are an organized education structure, a supportive policy environment, government commitment to funding, and recognition of the need for continued technical assistance to the Government to implement its national school feeding programme. The COVID-19 pandemic and related restrictions had a major impact on school closures; however, programme partners took adaptive actions to continue activities and trainings within pandemic parameters, helping ensure the continuity of the programme. CONCLUSION 23. Programme performance has been strong, especially since the midterm evaluation. Most endline targets are met or exceeded in nearly all activities. The programme’s transition from a focus on a small number of schools to include strengthening national capacity has effectively broadened its impact and influence on a national scale. The programme is well positioned for Phase II, with good internal management in WFP, strong working relations with Government and partners, and capacity at the implementation and policy levels to provide technical support to the roll-out of Rwanda’s national school feeding programme. RECOMMENDATIONS 24. Recommendations are presented by priority within strategic and operational categories. 25. Strategic Recommendation 1: Consolidate WFP internal staff capacity to be enablers rather than implementors. Many WFP Rwanda staff are skilled in operations whereas WFP’s expanded role requires staff to work at a policy and strategy level, and to manage government relations. WFP Rwanda has grown staff capability in capacity strengthening, partnership facilitation and relationship management for school feeding. These skills should be institutionalized and amplified across its broader portfolio. There is a strategic opportunity to formalize these capability requirements into human resources policies and processes, and to consolidate current experience into internal training models. It should also contribute its experience to WFP regionally and globally. 26. Strategic Recommendation 2: Consolidate and better document the country office’s capacity strengthening strategy, approach and priorities for application across the country strategic plan as a whole, building on the McGovern-Dole experience. Capacity strengthening needs to be considered a unique outcome to get the right level of design attention. It is essential that WFP’s efforts to evolve and scale up essential services are properly co-designed with the Government and other stakeholders, and reliably resourced for continuity. A specific strategic objective linked to government capacity strengthening will legitimize and publicly demonstrate WFP’s role in addressing school feeding risks and opportunities ahead, and position WFP to provide continued support. 27. Strategic Recommendation 3: Decide on WFP’s approach to supporting further development of a national procurement strategy so this can be resourced, planned, and efficiently executed. WFP has provided essential strategic and practical support to the Government’s consideration of its procurement model options. WFP needs to understand the Government’s specific expectations for WFP support so this can be effectively resourced and planned. Specifically, WFP needs to know how to position itself between a leadership role in presenting options and facilitating decisions, and a supporting technical role while Government works this through. Date | Report Number vi 28. Strategic Recommendation 4: Establish a learning agenda at country-office level to complement performance monitoring, test assumptions and document lessons from school feeding interventions. The McGovern-Dole Programme has built a strong performance monitoring system. In addition to output- and outcome-level results, it needs to capture learning regarding the underlying assumptions for change and determinants of success. Learning and reflection processes require dedicated systems, especially for programmes like McGovern-Dole that require real-time information for adaptive management. It is especially important for learning around gender equality in education. 29. The foundations of the national school feeding programme are established. However, much remains to be learned around scaling, equity and access to programme benefits, targeting across the nexus of schools￾community-local-government needs and capacities, purpose and boundaries of procurement models, alignment with related initiatives, resourcing, and sustainability of Phase 1 results. These issues need to be documented for stakeholders’ strategic and operational decision-making. Programme partners have contributed much to learning and should be included in developing and realizing the learning agenda. WFP Rwanda has a real contribution to make to the broader system on these issues, but process documentation is critical to realize this potential. 30. Operational Recommendation 1: Retain and continue to improve the effective McGovern-Dole Programme operating model developed after the Phase 1 midterm. An operating model framework considers the foundational elements that underpin strategic and implementation results. It commonly includes the following elements: people, governance, process, service delivery model, technology, and performance insights and data. It would be a useful framework for WFP to track performance of core organizational components as it continues to evolve its role. 31. Operational Recommendation 2: Ensure appropriate balance between capacity strengthening and direct implementation. A key factor that has enabled the programme to contribute to the development of the national school feeding programme is its technical credibility and grounded experience. It will be important for WFP to ensure its facilitation and technical support remain credible, by maintaining a high￾quality school feeding operation that demonstrates capacity to advise the Government on these issues. Some of this can come from WFP’s global experience and by connecting the Government to global school feeding networks. 32. Operational Recommendation 3: Maintain support to Phase I McGovern-Dole Programme schools, communities, and cooperatives. The Phase II design includes all Phase I schools until the second half of Phase II. It will be important that WFP continue to draw lessons, monitor and organize support directly and indirectly (e.g., through government and local partners) to Phase 1 project sites during the second half of Phase II. Concrete examples are supporting maintenance systems for the multitude of school infrastructure built in the last year of the programme and continuing to evolve the discussions with parents, and school, community, and local government representatives on expectations for parent contributions. 33. Operational Recommendation 4: Build adaptive management and agility into the Phase II design and implementation plan. A key strength in Phase I was making critical decisions efficiently and at the right time. Given that Phase II will focus more on the practical realities of the national school feeding programme, a similar level of agility and funding flexibility will be required to address emerging risks and opportunities. This can be facilitated by building in regular reflection points focusing on progress toward outcomes, routine performance monitoring, and maintaining a leadership culture that enables bold management, operational and funding decisions when necessary. 34. Operational Recommendation 5: Establish contingency planning at school and community level to prepare for and respond to shocks and stresses like COVID-19. WFP has effective contingency planning policies and processes that allowed management to quickly pivot the programme during the pandemic. The benefits of contingency plans and systems are clear. There is an opportunity now to embed contingency planning more directly into the local systems that WFP supports, which will contribute to broader resilience building and mitigate risks to the programme. Date | Report Number vii 35. Operational Recommendation 6: Implement appropriate gender analysis and approaches for Phase II. While the McGovern-Dole Programme design aligns with relevant gender policies and frameworks, efforts toward gender-equitable outcomes and gender-appropriate approaches would benefit from a broader lens that monitors how interventions influence GEWE. The absence of a comprehensive gender analysis, and the missed opportunity to implement relevant gender- sensitive approaches in interventions, are significant gaps. The programme should make learning on gender an explicit focus for Phase II. WFP should conduct a gender analysis at baseline and consider a gender audit at midterm and/or at endline. Date | Report Number 1 1. Introduction 1.1 EVALUATION FEATURES 1. This is the report of the independent endline evaluation of World Food Programme (WFP) Rwanda’s Home￾Grown School Feeding (HGSF) Programme 2016-2021.9 , 10 WFP Rwanda commissioned the evaluation to provide an evidence-based, independent assessment of programme performance. It has the dual objectives of accountability – to assess and report on programme performance and results, and learning – to determine why certain results occurred, and to derive lessons and good practices. The evaluation is timed and designed to inform the baseline evaluation of Phase II of the programme (2020-2025), operational and strategic decisions for Phase II, as well as learning in the context of COVID-19. 2. Specifically, the endline evaluation i) reviews the project’s relevance, effectiveness and efficiency, impact, and sustainability, ii) collects performance indicator data for strategic objectives and higher-level results, iii) assesses whether the project has succeeded in achieving McGovern Dole’s two strategic objectives, iv) investigates the project’s overall impact, and v) identifies meaningful lessons that stakeholders can apply to future programming. The main stakeholders with a direct interest in this evaluation and its recommendations are the WFP Rwanda Country Office (CO); its partner NGOs World Vision and Gardens for Health International (GHI); Government of Rwanda counterpart ministries, especially the Ministry of Education (MINEDUC), Ministry of Local Affairs (MINALOC), and Ministry of Agriculture and Animal Resources (MINAGRI); and the United States Department of Agriculture (USDA), the primary donor.11 Programme beneficiaries are also key stakeholders, in line with WFP’s and the ET’s commitments to accountability to affected populations. 3. The evaluation scope covers the entire implementation period of the McGovern-Dole Programme (2016- 2021). It covers the 107 schools in the four districts where the programme is implemented - Nyaruguru, Nyamagabe, Rutsiro and Karongi.12 The evaluation also covers WFP’s capacity strengthening activities at the national level, designed to support the Government’s development of a national school feeding programme. 4. The evaluation was conducted by TANGO International, which also conducted the midterm evaluation (MTE). The evaluation team (ET) was comprised of two international consultants (one female, one male) and four field-based national consultants (three female, one male), supported by a quantitative data analyst and a research associate. The team brings expertise in school feeding, food security, capacity strengthening, gender, quantitative and qualitative methods, and statistical analysis, as well as extensive experience evaluating WFP programmes. Both the national and international consultants have evaluated numerous similar programmes in Rwanda and the region. 5. Evaluation planning started in November 2020 and continued through the inception phase in December 2020. Due to school closures and other mitigation measures related to COVID-19, evaluation fieldwork was delayed till February 2021 and conducted by the local team over four weeks; the international consultants conducted remote interviews in the same approximate period. The cut-off date for data collection was March 14, 2021. 13 9 A no-cost extension through September 2021 was granted as this report was being finalized. 10 The terms of reference refer to the McGovern-Dole grant as the WFP Rwanda HGSF. The two terms are used interchangeably and in context, as there are cases where we mean to highlight the home-grown school feeding aspect. 11 See Annex 1 for complete stakeholder listing and analysis. 12 After the data collection phase, the program received a no-cost extension through September 2021 that added an additional school, bringing the total number of schools to 108.The evaluation does not extend to the additional school. 13 See further discussion and details in Section 1.4 and Annex 2. Date | Report Number 2 1.2 CONTEXT Poverty and food security 6. Rwanda is a small, hilly, landlocked, and densely populated country in East Africa with about 12.5 million people (2018). While Rwanda ranks 160th out of 185 countries in the 2020 Human Development Index,14 it is making progress in reducing food insecurity and poverty. Poverty decreased between 2011 and 2017 from almost 45 percent to just over 38 percent;15 extreme poverty declined from 24.1 to 16 percent.16 The highest rates of poverty and extreme poverty are in the south and west, including the four districts targeted by this programme – Karongi (approximately 53 percent), Rutsiro (approximately 49 percent), Nyaruguru (approximately 53 percent), and Nyamagabe (approximately 48 percent).17 7. The Government of Rwanda monitors food security through the Comprehensive Food Security and Vulnerability Analysis (CFSVA) conducted by WFP. The 2018 CFSVA found that 18.7 percent of households are food insecure.18 Among those, 1.7 percent are severely food insecure. Food insecurity is highest in Western Province (29.9 percent of households), followed by Southern Province (20.5 percent), Northern Province (17.8 percent), and Eastern Province (16.2 percent). The stunting rate is highest in Western Province (40.2 percent)19, which exceeds the World health Organization critical threshold for stunting (>40 percent). Food insecurity declined in the four targeted districts between 2015 and 2018 (Table 1). Table 1: Percentage of food insecure households, by district, 2015 and 2018 2015 2018 Rutsiro 57% 49% Nyamagabe 42% 30% Nyaruguru 37% 24% Karongi 35% 25% Source: CFSVA 2015 and 2018 Key data and trends related to SDG 2: Zero hunger 8. Forty-one percent of Rwandans are undernourished, and almost one fifth is food insecure (SDG 2.1.1).20 Thirty-three percent of children under five years of age (CU5) are stunted and nine percent are severely stunted.21 Stunting is inversely related to wealth quintile; 49 percent of children in the lowest wealth quintile are stunted, compared with 11 percent of children in the highest quintile. Low educational attainment and poor diet are common in mothers of stunted children. In addition, 12.6 percent of CU5 are underweight (low weight for age) and 2 percent are acutely malnourished (low weight for height). 9. Micronutrient deficiencies are also a public health concern; 37 percent of children under five and 19 percent of women of reproductive age are anemic. The most common causes of anemia are lack of iron in the diet and intestinal worms, preventing the absorption of micronutrients and minerals such as iron. Worm infections affect 65 percent of the population in Rwanda, and school-aged children are particularly affected. Main drivers of malnutrition are poor access to quality water, sanitation, and hygiene (WASH) as well as poor care practices such as not receiving antenatal care, even among those who can access a nutritional, balanced diet. 22 14 UNDP. 2020. Human Development Report 2020. 15 National Institute of Statistics Rwanda (NISR). 2015. Rwanda Poverty Profile Report 2013/14. 16 NISR. 2018. Rwanda Poverty Profile Report 2016/17-Results of Integrated Household Living Conditions Survey (EICV5). 17 NISR. 2018. EICV5. 18 WFP. 2018. Rwanda: Comprehensive Food Security and Vulnerability Analysis. 19 Republic of Rwanda. Rwanda Demographic and Health Survey 2019-20. Key Indicators Report. National Institute of Statistics of Rwanda, Ministry of Health, and ICF. 2020. 20 WFP. 2018. Rwanda: Comprehensive Food Security and Vulnerability Analysis. 21 National Institute of Statistics of Rwanda. 2020. Rwanda Demographic and Health Survey 2019-20 Key Indicators Report. October. https://dhsprogramme.com/pubs/pdf/PR124/PR124.pdf 22 United Nations Rwanda. 2017. Rwanda Common Country Analysis. Date | Report Number 3 10. Vision 2020 sought to modernize agriculture and increase productivity,23 and numerous national policies support agricultural improvements. Between 2012 and 2014, annual crop production increased by 5.7 percent, higher than the population growth rate (2.5 percent).24 Still, production is highly seasonal and poor families face higher prices and lower supply during the lean season. The majority of households in Rwanda are smallholder farmers with small plots of land. Women play an important role in farming, and 24 percent of the land is owned by women. However, women are mainly engaged in production rather than better￾paying value-added agricultural processing and marketing activities. Key data and trends related to SDG 4: Quality education 11. The government provides 12 years of free, compulsory, basic education to all children. The Education Sector Strategic Plan (2018/19-2023/24) emphasizes access to learning for disadvantaged students such as girls, children from poor families, and people with disabilities.25 Primary school enrolment is high (98.0 percent girls; 97.3 percent boys) and gender equity has been largely achieved due to government policies emphasizing the importance of universal primary education and gender equality in education. This includes the elimination of fees for basic education by government in 2003 that has enabled more children, especially girls, to attend school. The Gender Parity Index (the ratio of girls to boys enrolled in all primary schools) in Rwanda was 0.99 in 2018, indicating that girls are near parity with boys.26 The primary school completion rate has increased substantially in recent years, from 60.4 percent overall in 2015 to 95.4 percent in 2019 (females: 65.5 percent to 101.8 percent; males: 55.3 percent to 89.0 percent).27 Female enrolment in both lower and upper secondary has been slightly but consistently higher than male enrolment in all years from 2014-2019, with net enrolment decreasing for both sexes in this period. 28 12. According to an assessment of literacy and numeracy skills done in 2014, 45.3 percent of P2 students could read at grade level and 32.9 percent were numerate consistent with their grade level. 29 The adult literacy rate in Rwanda for people 15 years and older was 73.2 percent in 2018.30 13. Classrooms are often crowded, with over 60 students per teacher in the project schools, against a national standard of 45 students to one teacher. In 2016, 1,545 out of a total of 185,666 children enrolled in pre￾primary education were identified as having a disability; in primary schools, children with disabilities represent 0.75 percent of the total number of children enrolled, a similar percentage to the previous three years.31 This represents fewer than one percent of enrolled students and is significantly below the expected numbers within the population. Students with a disability represented only one percent of the total enrolled in secondary education in 2016. 14. The average primary school has one toilet for every 52 students (national targets: 40:1 for boys and 30:1 for girls). Only 54.1 percent of schools have piped water.32 MINEDUC estimates that menstruation accounts for an average of 50 days/girl/year in absences, and menstrual management continues to negatively affect girls, especially in the poorest districts. The education sector plan has a dedicated budget line to address education barriers for girls, including the provision of gender-sensitive WASH facilities. 23 Government of Rwanda. 2004. Rwanda Vision 2020. 24 NISR. 2015. EICV4. 25 Republic of Rwanda, Ministry of Education. 2018. Education Sector Strategic Plan, 2018/19-2023/24. 26 UNESCO Institute for Statistics. data.worldbank.org/indicator/SE.PRM.PRSL.ZS?locations=RW 27 National Institute of Statistics of Rwanda. 2020. Rwanda Statistical Yearbook 2020. Secondary completion rates not reported. 28 In the 2014-2019 period, the net enrolment rate secondary education for girls ranged from a high of 37.70 percent in 2014 to a low of 26.70 percent in 2019 to a high of 45.00 percent in 2019. For boys, it ranged from a high of 33.60 percent in 2014 to a low of 22.10 percent in 2019. National Institute of Statistics of Rwanda. 2020. Rwanda Statistical Yearbook 2020. 29 Republic of Rwanda. 2019. Rwanda Voluntary National Review Report. 30 World Bank. 2020. Data as of September 2020. https://data.worldbank.org/indicator/SE.ADT.LITR.ZS?locations=RW 31 Republic of Rwanda. MINEDUC. 2018. Revised Special Needs and Inclusive Education Policy. October. https://www.mineduc.gov.rw/fileadmin/user_upload/Mineduc/Publications/POLICIES/Special_Needs_Strategic_Plan.pdf 32 United Nations Rwanda. 2017. Rwanda Common Country Analysis. Date | Report Number 4 15. There are several early literacy initiatives in Rwanda in addition to the McGovern-Dole programme. MINEDUC, World Vision Rwanda, and WFP are partnering on Literacy Boost, a school feeding and literacy programme for 195,000 children in grades 1‐3 in 280 primary schools in central Rwanda.33 USAID funds Soma Umenye (2016-2021), a national early-grade reading intervention which focuses on Kinyarwanda reading skills for one million children in grades 1-3; and Mureke Dusome (2015-2020), a nationwide early￾grade literacy project. The Building Learning Foundations project funded by the UK Government seeks to enhance the quality of English and math teaching in grades 1-3 for 2.6 million children in public and Government primary schools. 16. Other relevant education programmes implemented by the Government are the One Cup of Milk per Child school feeding programme for pre-primary students and a cash transfer to secondary schools to provide a midday meal. Another relevant initiative is MINESANTE’s 12+ programme,34 which aims to reduce drop‐out rates by creating safe spaces in schools for girls; the programme targets 114,500 girls (ages 10‐12) in primary schools. Key data and trends related to SDG 17: Partnerships for the goals 17. Rwanda has made progress on a relevant key SDG 17 indicator: Government Health and Education spending (5.8 percent of GDP in 2020).35 The Government has been successful in funding 84 percent of its budget in 2018/2019 from domestic sources.36 External grants declined to 16 percent in 2017/2018, which is a promising trend for continued self-financing of education, health and other social sector programmes. 18. While the McGovern-Dole Programme does not explicitly mention human rights, the ET notes that the project supports two fundamental human rights as recognized in the 1948 Universal Declaration of Human Rights. The project activities, through its provision of a hot, healthy school meal, supports the Right to Food, which is interlinked with nutrition and health, and the ability to learn. The McGovern-Dole Programme activities centered around literacy, an improved learning environment at schools, and gender equality in education supports the Right to Education. Health 19. Nationally, the prevalence of stunting is 33 percent, a decrease of eight percentage points since 2012; however, stunting remains high in Western Province at 40 percent, and at 33 percent in Southern Province. 37 Stunting is influenced by several factors including diets low in nutrients and/or calories, illness, poor water quality, and the mother’s level of education. Stunting prevalence is 45 percent among mothers with no education versus 36 percent among mothers with a primary school education. Diets are high in starches and legumes and very low in animal protein, and only 17 percent of children aged 6 to 23 months have a minimum acceptable diet. Teenage pregnancies have increased over the past decade to 7.3 percent of women aged 15-19, attributed to violence against women, poverty, and lack of knowledge about reproductive health.38 20. Micronutrient deficiencies contribute to malnourishment. Anemia affects 37 percent of CU5, caused by iron-deficient diets, and intestinal worms. Intestinal worms prevent the absorption of nutrients and affect 65 percent of Rwandans, particularly school-aged children. Other contributors to malnutrition are poor access to WASH and health services and poor care practices. 39 A relevant initiative in this respect is the 33 (1) Evaluation TOR. (2) World Vision. 2016. Rwanda Annual Report 2016. 34 12+ is supported by DFID, Nike Foundation, and PSI. Source: Evaluation TOR. 35 SDG Center for Africa and Sustainable Development Solutions Network. 2018. 36 Republic of Rwanda. 2019. Rwanda Voluntary National Review Report. 37 Republic of Rwanda. Rwanda Demographic and Health Survey 2019-20. Key Indicators Report. National Institute of Statistics of Rwanda, Ministry of Health, and ICF. 2020. 38 Republic of Rwanda, Gender Monitoring Office. 2019. The State of Gender Equality in Rwanda from Transition to Transformation. March. 39 United Nations Rwanda. 2017. Rwanda Common Country Analysis. Date | Report Number 5 Ubuzima WASH project,40 which trains Community Health Workers and school‐based volunteers through the Community‐based Environmental Health Promotion Programme (CBEHPP), MINESANTE’s approved methodology of working with communities to help them identify and solve their own health and hygiene issues. World Vision is Rwanda’s largest implementer of CBEHPP and outside of the McGovern-Dole Programme is supporting MINESANTE in training 45,000 CHWs in 15,000 villages to reduce hygiene‐related diseases in communities and within schools. 21. The latest Integrated Household Living Conditions Survey (2016/17) indicates that the mean time needed to travel to a health center (on foot) is 56.2 minutes in southern provinces and 53.6 minutes in western provinces, slightly higher than the national average of 49.9 minutes; the report notes especially large reductions over the previous five years for rural areas and poor households.41 Gender 22. The Government of Rwanda has incorporated gender equality and gender equity as cross-cutting principles into its major legal and policy frameworks. Rwanda’s 2003 constitution (revised in 2015) recognizes women’s rights and gender equality, and women are legally guaranteed equal rights under land and inheritance laws. The National Strategy for Transformation (2018-2023) sets and tracks targets for the country’s development that include indicators for gender progress in education, employment, health, and other areas. Rwanda was ranked seventh in the 2021 Global Gender Gap Index by the World Economic Forum, the only Africa country to rank in the top ten in gender parity. The ranking recognizes Rwanda’s progress in reducing gender inequality, particularly in economic and political participation by women.42 However, there is less progress in other areas, and the 2020 Gender Inequality Index, which captures inequality in reproductive health, empowerment, and labor markets, ranks Rwanda 92nd out of 189 countries.43 23. Women continue to face inequities based on traditional gender roles that prioritize family and household duties for girls and women, as well as economic and health challenges. Female-headed households are 23 percent of all households and are more likely to be poorer and more food insecure than those headed by men. Around one-quarter of households in the four McGovern-Dole supported districts are female-headed. Between 51 percent and 77 percent of households in those districts are in the poorest wealth quintiles.44 Only 54 percent of female-headed households had some education versus 80 percent of male-headed households (2018).45 Half of households whose head completed primary school are food secure, while only 26 percent of households whose head had no schooling are food secure. Children in 29 percent of the poorest households were absent for one week or more over a three-month period compared to only 12 percent of children in wealthier households. Government policies and priorities relevant to the program46 24. The Government of Rwanda is guided by the national development plans Vision 2020, Vision 2035 and Vision 2050, which envision Rwanda transforming from an agrarian to an upper-income, knowledge-based economy. The country’s poverty reduction strategy prioritizes quality education for all as a prerequisite for a knowledge-based economy. 25. Vision 2050 aims to eliminate chronic malnutrition, guided by the national Food and Nutrition Policy (2018- 2024).47 The School Health and Nutrition (2014) policy states that all schoolchildren shall study in a healthy 40 Ubuzima operates via a cost‐sharing partnership between World Vision and Rwanda’s Water and Sanitation Corporation (WASAC), partnering with MINESANTE. Source: Evaluation TOR. 41 National Institute of Statistics of Rwanda (NISR). 2018. Rwanda Poverty Profile Report, 2016/17. November. 42 World Economic Forum. Global Gender Gap Report 2021. Insight Report. March 2021. 43 United Nations Development Programme. Human Development Report 2020. 44 WFP. 2018. Rwanda 2018: Comprehensive Food Security and Vulnerability Analysis. 45 “Some” education is not defined in the reference document (WFP. 2018. Rwanda 2018: Comprehensive Food Security and Vulnerability Analysis.) 46 See Annex 3 for a listing of additional relevant government policies. 47 WFP. 2018. Draft Country Strategic Plan (2019-2023). Date | Report Number 6 environment in child-friendly schools. Agriculture plays a key role in economic growth, poverty reduction, and food security. 26. Rwanda has improved the quality, coverage, and access to basic education through the Education Sector Strategic Plan (2018/19-2023/24). In 2012 the Government committed to providing 12 years of free, compulsory basic education. Rwanda has nearly achieved universal primary education and gender parity at pre-primary and primary levels. At the end of 2019, MINEDUC changed the language of instruction for lower (P1-P3) and upper primary (P4-P6) from Kinyarwanda to English.48 27. MINEDUC leads the education sector on policy, planning, coordination, regulation, monitoring and evaluation of the education activities sector. The Rwanda Education Board (REB) oversees education activities up to secondary level. District Administrations are responsible for the delivery of local education services. District Education Officers (DEOs), employed by the Ministry of Local Government (MINALOC), plan, deliver and monitor education services in their districts. Other international assistance in Rwanda relevant to the program 28. Total official development assistance to Rwanda and official aid received was USD 1.191 billion in 2019.49 Specific areas of assistance relevant to the programme, beyond education described above, include work under One UN and assistance to the health sector, as described below. 29. United Nations collaboration. WFP has worked in Rwanda since 1975.50 Rwanda is one of eight pilot countries in which United Nations agencies coordinate closely through One UN. United Nations agencies coordinate their activities toward economic transformation, social transformation, and transformational governance (the three pillars of the Government’s National Strategy for Transformation) through the United Nations Development Assistance Programme (UNDAP II) (2018-2023).51 United Nations inter-agency collaboration includes WFP and UNHCR’s joint assistance to Congolese and Burundian refugees in Rwanda52 and WFP, UNICEF and UNESCO collaboration on education activities such as creating Child‐ Friendly School standards and assessing literacy and numeracy. 30. Assistance to the health/nutrition sector. USAID Rwanda’s Feed the Future current multi-year strategy works to improve infrastructure, agricultural market linkages, nutrition, innovation, and policy through multiple activities.53 Since 2013, the Clinton Health Access Initiative (CHAI) has partnered with the Government to work with food producers to produce fortified blended foods to reduce malnutrition among infants and pregnant and lactating women.54 CHAI is supporting Africa Improved Foods on sourcing local agricultural produce to use in its factory which produces nutritious fortified blended foods that can be distributed throughout the country. USAID also supports programming in Rwanda under the President’s Emergency Plan for AIDS Relief, the President’s Malaria Initiative, and the Global Climate Change initiative.55 Effect of COVID-19 in Rwanda 31. COVID-19 infections and deaths have been low in Rwanda, largely due to the Government’s rapid action on prevention measures including lockdowns, movement restrictions, and school closures.56 Schools closed 48 MINEDUC. 2019. Communiqué: MINEDUC endorses the use of English language as a medium of instruction in lower primary. December. https://www.mineduc.gov.rw/news-detail/communique1 49 World Bank. Website consulted 7 June 2o21: https://data.worldbank.org/indicator/DT.ODA.ALLD.CD?locations=RW 50 WFP. 2017. WFP Rwanda Country Brief. November. 51 United Nations Rwanda. 2018. UNDAP 2018-2023 for Rwanda. Signed 31 July 2018. 52 Rwanda hosts approximately 74,500 refugees from the Democratic Republic of Congo (UNHCR 2021, DRC Situation, May 2021; figure from March 2021) and 47,800 from Burundi in six camps and in urban areas (UNHCR 2021, Burundi Situation, May 2021; figure from May 2021). A voluntary repatriation of Burundian refugees has been ongoing since August 2020. The protracted refugee situation strains limited resources of the Government of Rwanda and international donors. 53 USAID. 2018. Rwanda: Nutrition Profile. 54 Evaluation TOR; WFP. 2015. WFP Rwanda FY 2015 McGovern-Dole proposal. 55 History of USAID Rwanda. https://www.usaid.gov/history-usaidrwanda 56 University World News – Africa Edition. 2021. Smooth start as universities in Kigali reopen. 01 March. https://www.universityworldnews.com/post.php?story=20210228222455672 Date | Report Number 7 in March 2020; they re-opened in November 2020 for grades 4-6, and in January 2021 for grades 1-. In both cases, school feeding resumed immediately upon school re-opening. Comparing the most recent reporting periods around the pandemic for which attendance data are available,57 the decrease in attendance was greater for boys (see Table 2). Table 2: Attendance pre- and post- COVID-19 school closures Apr 2019 – Sep 2019 Oct 2019 – Mar 2020 Apr 2020 – Sep 2020 Oct 2020 – Mar 2021 All 95% Data not collected Data not collected 92.1% Boys 94% 88.3% Girls 96.2% 96% Source: WFP Rwanda, email communication 32. March 2021 cumulative figures indicate 25,311 COVID cases and 337 deaths.58 The Government has strengthened social protection programmes, set up remote learning programmes, and enacted an economic recovery plan for affected households and businesses. The pandemic prevention measures are expected to negatively affect the economy as output and employment have been reduced.59 1.3 SUBJECT BEING EVALUATED Table 3: McGovern-Dole Programme overview • Subject of evaluation: WFP's USDA McGovern-Dole International Food for Education and Child Nutrition Programme’s Support in Rwanda 2016-2021 • Type of evaluation: Activity evaluation • Timeline/duration: Proposal submitted FY 2015; approved for implementation period 1 October 2015 – 30 September 2020; granted a no-cost extension into FY2021 with a new end date of June 30, 2021; a second no-cost extension was later authorized through September 2021 • Budget: McGovern Dole (USDA): $25 million over five years (2016-2020); additional funding from the MasterCard, FEED and Caterpillar Foundations • Geographic areas: Initially 104 rural schools (up to 85,000 primary school students; currently approx. 83,000 students across Nyaruguru and Nyamagabe districts in the south (daily cooked meals), and in Rutsiro and Karongi districts in the west (daily porridge meals); through no-cost extensions, in 2020, WFP expanded the coverage of meals and select activities to three additional schools in Karongi, and in 2021 a fourth school was added, bringing the total to 108 schools • Thematic areas: School feeding and education; water, sanitation and hygiene, health, and dietary practices, national capacity strengthening • Transfer modality: In-kind food transfers (US food commodities and local and regional purchase of food commodities) and capacity strengthening • Partners: MINAGRI, MINEDUC, World Vision, GHI, Rwanda Biomedical Centre, districts Activity description 33. The strategic objectives of the WFP Rwanda HGSF Programme60 are (1) improved literacy of school-aged children and (2) increased use of health and dietary practices.. The programme started in October 2015; it was originally scheduled to end September 2020 but has received two no-cost extensions, taking it through 57 Due to covid, WFP did not collect or report attendance data for the October 2019 – March 2020 or the April – September 2020 reporting periods. We therefore compare the October 2020 – March 2021 period to April – September 2019. 58 Rwanda Biomedical Centre. 2021. https://www.rbc.gov.rw/fileadmin/user_upload/annoucement/Update-on-COVID-19-03- 05-2021-eng.jpg . Update dated 5 March. 59 World Bank. Rwanda Economic Update. Protect and Promote Human Capital in a Post-COVID-19 World. Edition No. 16, January 2021. 60 The program in Rwanda is hereafter referred to as the “McGovern-Dole Programme.” Date | Report Number 8 November 2021. The programme supports approximately 83,000 primary school61 children annually in 10862 schools in four of the poorest and most food-insecure districts in Rwanda (see map, Annex 4). The participating schools were selected by WFP and MINEDUC through a process that identified the target districts, selected the most food-insecure sectors within those districts based on primary data on poverty and health, and within those areas selected specific schools and communities that had the capacity to participate in the activities.63 34. In Nyamagabe and Nyaruguru districts in Southern Province, 40,000 children receive a daily meal during the school week consisting of 120 grams (g) of maize, 30g beans, 15g vitamin-A-fortified vegetable oil, and 3g iodized salt.64 In Karongi and Rutsiro districts in Western Province, 43,000 primary school students receive a porridge meal consisting of 120g of Supercereal and 15g of sugar. Approximately half of the students are female. Girls and boys receive the same meal, in keeping with gender equality principles. The food commodities selected for the programme are intended to fulfil a significant portion of each student’s daily nutritional requirements. The meal is also formulated to reduce micronutrient deficiencies and improve iron uptake when combined with deworming medications, as there is a high prevalence of anemia among schoolchildren in Rwanda. 35. Beyond providing a daily meal, the McGovern-Dole Programme supports an integrated set of activities to create an enhanced educational environment. The activities include piped water supply to schools, improved kitchens and storerooms, and school gardens. The programme design addresses gender equality in education by expanding and upgrading latrines to provide appropriate sanitation facilities for girls but does not otherwise have a strong gender focus. The physical improvements are complemented by training in literacy instruction methods for teachers, management training for head teachers, and literacy groups to promote students’ reading outside the classroom. Other programme activities include capacity strengthening for head teachers, teachers, School General Assembly Committees (SGACs), School Management Committees, cooks, and storekeepers. Funding 36. The US $25 million McGovern-Dole programme in Rwanda is funded by the United States Department of Agriculture (USDA) McGovern-Dole International Food for Education and Child Nutrition Programme.65 The McGovern-Dole programme, administered by the Foreign Agricultural Service, delivers international school feeding using donated in-kind commodities, usually supplied by the United States, and also delivers financial and technical assistance. The programme provides imported U.S. food commodities and financial support, and WFP procures maize meal, beans sugar and salt with separate funding from private donors such as MasterCard. Other programme donors are the FEED and Caterpillar Foundations. As of the last Annual Country Report available when this report was being finalized, the programme was confirmed in 2019 as more than 100 percent funded for multi-year use.66 Results framework 37. McGovern-Dole Programme in Rwanda aligns with the McGovern-Dole results framework and foundational results, which each describe causal pathways and assumptions (see Annex 5). With respect to the McGovern-Dole results framework, the programme addresses Strategic Objective (SO) 1, improved literacy of school-age children, and SO 2, increased use of health and dietary practices, with a view to supporting the Government in establishing a national school feeding programme, including strengthening capacity at national, district and school levels.67 Activities have been planned, sequenced, and implemented to ensure 61 Grades 1-6 62 Originally 104 schools. Three schools were added in Karongi in 2020 and one more was added mid-2021. 63 WFP. 2015. WFP Rwanda FY2015 McGovern-Dole proposal. 64 For a portion of the program, these commodities were purchased through the WFP Rwanda USDA’s Local and Regional Food Aid Procurement (LRP) Programme (2017-2019). See paragraphs 42, 79, and 111. 65 USDA McGovern-Dole Grant FFE-696-2015/007-00. 66 WFP. 2019. Rwanda Annual Country Report. Country Strategic Plan 2019-2023. 67 See Annex 5: Results Framework. Date | Report Number 9 achievement of SO1 and SO2, which both support improved student attendance (MGD 1.3). Activities for both SOs show a logical causal chain. 38. The programme contributes to the achievement of the McGovern-Dole foundational result, increased capacity of government institutions, by strengthening the national school feeding programme and by building capacity at national, district and school levels to eventually integrate elements of the McGovern-Dole Programme, including the strengthening of literacy and good health and hygiene practices among students and school personnel. It also supports the foundational result, increased engagement of local organizations and community groups, by building the capacity of local agricultural cooperatives to eventually supply food to school feeding programmes. Theory of change 39. The program design follows the McGovern-Dole Results Framework (see Annex 5). Activities have been planned, sequenced and implemented to ensure achievement of Strategic Objective 1, Improved Literacy of School-Age Children. Activities have also been aligned with the objectives of SO2, Increased Use of Health and Dietary Practices, to ensure that the broad range of interventions that support improved student attendance (MGD 1.3) are achieved. While there is no explicit theory of change for the program, the activities implemented to achieve both strategic objectives show a logical causal chain. The foundational results incorporate the actions that the McGovern-Dole initiative considers critical to the development of a sustainable, government-financed and -managed HGSF program for primary schools. This includes increased capacity of government institutions, community groups and local organizations. 40. While there is no theory of change for the programme, an important underlying assumption in an imputed theory of change is that parents will make consistent and adequate contributions to maintain school meal activities, such as helping fund cooks’ salaries and providing fuel wood. The national Government also views parent contributions, whether cash or in-kind, as an important element of sustainability. However, there is a question of whether this design aspect is appropriate to local capacity in poor areas. Schools must pay cooks, purchase firewood, buy soap and utensils, and pay for water, and they depend in part on cash contributions from the community to meet these costs. Individual schools may decide to allow parents to donate firewood, vegetables, labor or other in-kind contributions approved by the school if they are unable to provide cash. While parent contributions have increased since the MTE due to increased sensitization, most of the schools visited by the ET are still struggling to cover these expenses. This is not surprising given that the programme works in some of the most impoverished districts in Rwanda. WFP and partners have proactively managed this aspect of programme to minimize the risk to effectiveness and sustainability. Gender 41. While WFP has a strong institutional focus on gender equality, the McGovern-Dole results framework does not specifically mention gender equality as an objective. The project proposal does not reference a gender analysis. The CO’s Country Gender Action Plan 2016-2020 refers to a gender baseline study done in 2015- 2016 but the plan notes only limited challenges to the HGSF, in terms of the number of female cooks and female teachers. Past evaluations 42. The McGovern-Dole Programme baseline was conducted in June 2016 to inform programme design.68 The MTE69 covered the 2016-2018 (second quarter included); Table 10 in Section 2.5 details the MTE recommendations and actions taken to address them. Another relevant report is the endline evaluation of the WFP Rwanda Local and Regional Food Aid Procurement (LRP) Programme (2017-2019), which was 68 WFP Rwanda. 2016. Baseline Study: Home Grown School Feeding Programme. Conducted by IPSOS. 69 WFP Rwanda. 2019. WFP's USDA McGovern-Dole International Food for Education and Child Nutrition Program's Support in Rwanda 2016-2020: Evaluation Report, Midterm Evaluation. July. Date | Report Number 10 awarded by USDA to complement the McGovern-Dole Programme. 70 As noted above, a gender study was conducted in 2015-2016. Amendments to initial programme design 43. WFP implemented Phase I (2015-2021) of the McGovern-Dole Programme with the original aims of supporting the Government to develop a national school feeding programme and to build government capacity for a handover of activities by 2020. The Government officially named WFP as its main partner in school feeding in 2019. The focus of the Phase I programme, designed in 2015, has shifted from a handover of the programme to building the capacity of Phase I schools to serve as model schools for the national programme, and then to transition those schools into the national school feeding programme in 2023/24.71 This change followed the Government’s approval and funding of a national school feeding programme to begin in 2021. At present, the Government does not have the budgetary resources to adopt all aspects of the McGovern-Dole model nationally. The longer transition time will allow the Government to identify additional funding sources and more cost-efficient measures to support the national programme. 44. WFP’s implementation of the programme is consistent with the original programme design, with a few exceptions. First, the programme had initially planned to install lower-cost kitchen facilities to allow for the construction of a greater number of kitchens. However, the design did not consider the need for maintenance and training on maintenance. When quality problems arose in the lower-cost kitchens, the programme opted for a revised design at higher cost. Based on MTE recommendations, there has also been attention to improving reliable access to water for meal preparation and cleanup, as well as general hygiene and sanitation. After the MTE, World Vision allocated as much funding as possible from its literacy activities to WASH. Interviews show that the early proposal consultations between WFP and World Vision did include a robust WASH component but that this was not included in the final design due to funding limitations. 45. In 2020 and early 2021, the McGovern-Dole Programme was adapted to keep some important programme activities running, albeit at a reduced level, during the COVID-19 pandemic. No in-school meals could be provided between April and September 2020; 72 instead, WFP distributed take-home rations (THR) to ensure that vulnerable schoolchildren had access to food. Reading clubs were unable to meet, so Literacy Boost coordinators arranged book-lending activities so that children could continue to receive reading materials at home. Trainings for school administrators and teachers continued but were changed from in-person training to self-learning manuals, and SGACs met in socially distanced small groups. Trainings on health and hygiene incorporated the symptoms and transmission modes of COVID-19. Booklets and posters were distributed to inform people about COVID-19 prevention measures. GHI helped distribute vegetables from school gardens to vulnerable families who could not obtain fresh produce due to movement restrictions. MINEDUC also required that permanent handwashing stations be established at schools to prevent the spread of COVID-19. Renovations to school facilities continued, though the pandemic delayed construction of some kitchen and WASH facilities in some locations. 46. Another exception to the original programme design concerns an evolving understanding of requirements for a local purchase model to sustain school feeding over the longer term. The programme proposal defined the model in terms of proximity of farmers and farmer cooperatives. However, this assumption was made with limited analysis or stakeholder consultation at the time. At programme midpoint, the country office commissioned an evaluation of their LRP programme to inform ongoing consultation with the Government on suitable models and modalities. 73 70 Available here: https://www.wfp.org/publications/rwanda-local-regional-procurement-project-2017-2019-endline-evaluation 71 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Plan of Operations and Activities. 7272 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 73 WFP. 2020. Evaluation of USDA’s Local and Regional Food Aid Procurement Program (Rwanda 2017-19). Endline. Date | Report Number 11 Beneficiary data 47. Programme reach has been extensive, with 150,255 individuals benefiting directly from USDA-funded programme interventions (75,096 girls [122.6 percent of target]; 75,159 boys [108.8 percent of target), as of March 31, 2021, well over the target of 133,328 (112.7 percent of target). Similarly, the target is exceeded for the number of individuals benefiting indirectly from the programme with 238,872 indirect beneficiaries against a target of 175,379 (136.2 percent of target). 74 Complete sex-disaggregated data are given in Annex 6, Table 14 and Table 15 (cumulative figures) and Table 38 and Table 39 (annual figures). Commodity distribution data 48. Over the life of the activity, the programme distributed 98 percent of planned commodities in Southern province and 79.8 percent of planned commodities in Western province (Table 4). The lower distribution rate in Western districts was due largely to difficulty reaching schools during the rainy season, which caused delivery delays. At the time of the endline evaluation, food distribution was ongoing to allow for further consumption of commodities. Another relevant distribution was deworming treatment; annual targets were met or nearly met, and the life-of programme target was exceeded (see Annex 6, Table 76 and Table 77 and discussion in paragraph 138). (see table next page) 74 Direct beneficiaries include children, teachers, school administrators, parents, cooks, storekeepers, farmers, and government staff. To calculate indirect beneficiaries, to prevent the double-counting of siblings and parents on PTAs, WFP Rwanda uses a family multiplier of 3 instead of 5 (Rwandan national standard). Date | Report Number 12 Table 4: Commodity distributions, expected v. actual Reporting timeline (by fiscal year) Vegetable oil distributions: Southern provinces CSB+ distributions: Western provinces Expected (NMT) Actual a (NMT) % Achieved Expected (NMT) Actual a (NMT) % Achieved FY21 Q1-2 1/10/2020-31/03/2021 62.44 34.66 55.50%b 494.22 464.33 93.95% FY20 Q3-4 1/04/2020-30/09/2020 70.42 82.58 117.28c 564.99 678 120% FY20 Q1-2 1/10/2019-31/03/2020 42.21 42.21 100% 295.83 295.83 100% FY19 Q3-4 1/04/2019-30/09/2019 70.51 70.51 100% 564.85 564.85 100% FY19 Q1-2 1/10/2018-31/03/2019 190.30 190.30 100% 46.54 46.54 100% FY18 Q3-4 1/04/2018-30/09/2018 55.19 51.73 94% 706.95 444.46 63%h FY18 Q1-2 1/10/2017-31/03/2018 43.12 66.52 154%d 384.48 326.22 85%e FY 17 Q3-4 1/04/2017-30/09/2017 f 42.00 41.25 98% 349.44 326.22 93% FY 17 Q1-2 1/10/2016-31/03/2017 f 27.86 18.29 66% 181.88 135.21 74% FY 16 Q3-4 1/04/2016-30/09/2016 f 36.53 28.54 78%g 299.75 148.10 49%g FY 16 Q1-2 21/12/2015-31/03/2016 f Not started N/A Not started N/A Total: 640.58 626.60 98% 3,888.92 3,103.53 79.80% Source: WFP. Semi-annual donor reports 2016-2020 and ET consultation with WFP for FY2021 data. a The country office notes that delivery of USDA commodities to Rwanda is often delayed, which sometimes compels distribution of the foods before their expiration date, affecting actual vs planned volumes in some periods. b Lower achievement than planned due to COVID-related closures still in effect in September 2020. Upper primary re-opened in November 2020 and lower primary re-opened in January 2021; full feeding for all grades did not begin till January 2021. c Higher achievement than planned because when schools closed due to COVID-19, the programme distributed higher-than-usual volume of take-home rations to cover the anticipated 3-4 months when students would not be in school. This was also done to make sure the vegetable oil and CSB+ would be fully consumed before their expiration date; this was done with USDA approval and was a factor in the rationale for the no-cost extension granted for April-September 2021. d Higher achievement than planned because enrolment was higher than planned (total enrolment during this period was nearly 85,000 compared to the following year, when it dropped to 81,250); balances (14MT) for academic term starting April were distributed at schools to ensure no pipeline break. e Slower rate of consumption owing to structural challenges like access to schools during the rainy season, resulting in delivery delays. WFP is working with the Government to resolve. f Fluctuations in percentage achievement during this period due to fluctuations in enrolment. The programme kicked off ~June 2016, so some of the lower achievement rates are due to the distribution occurring in a partial year only. g Food had already been distributed for Q2 in southern provinces and in Q3 for western and southern provinces for the school year. h Lower achievement than planned because enrolment dropped below target in 2018, a drastic decrease from 2017. Date | Report Number 13 1.4 EVALUATION METHODOLOGY, LIMITATIONS, AND ETHICAL CONSIDERATIONS Evaluation criteria and questions 49. This evaluation was designed to address the OECD-DAC75 criteria of relevance, effectiveness, efficiency, impact, and sustainability of the McGovern-Dole Programme, per USDA monitoring and evaluation policy as specified in the evaluation Terms of Reference (TOR) (see Annex 7). Table 5 presents the OECD-DAC evaluation criteria and corresponding questions. Table 5: OECD-DAC criteria and evaluation questions Criteria Evaluation Questions Relevance To what extent is the programme in line with the needs of beneficiaries (boys and girls) and partners, including Government? To what extent is the activity aligned with WFP, partner, UN agency, and donor policies and priorities? To what extent is the intervention based on a sound gender analysis? To what extent is the design and implementation of the intervention gender-sensitive? *(new) Are the changes made to activities (design and implementation) due to COVID-19 relevant to beneficiaries, global, country, and partner/institution needs, policies, and priorities? Effectiveness To what extent are the outcomes or objectives of the intervention likely to be achieved? What are the major factors influencing progress in achievement or non-achievement of the outcomes/objectives of the intervention? To what extent does the intervention deliver results for boys and girls? Efficiency Is the programme implemented in a timely way? Are the activities cost-efficient? Is the programme implemented in the most efficient way compared to alternatives? Were the project strategies efficient in terms of financial and human resource inputs as compared to outputs? Does the monitoring system efficiently meet the needs and requirements of the project? Impact What are the medium-term effects on beneficiaries’ lives? What are the gender-specific medium-term impacts? Did the intervention influence the gender context? *(new) To what extent did COVID-19 affect project implementation and performance? Sustainability To what extent is the Government taking ownership of the programme (e.g. demonstrated commitment and contributions)? What is the demonstrated capacity at central and sub-national levels to manage the program? Are local communities (School General Assembly Committees, farmers’ groups, etc.) fully involved in and contributing toward school feeding and education activities? Has the policy framework supporting the National School Feeding Programme been strengthened within the project period? What are the major factors influencing the achievement or non-achievement of sustainability of the program? 75 Organisation for Economic Co-operation and Development (OECD) – Development Assistance Committee (DAC) Date | Report Number 14 50. The TOR also presented key evaluation questions to highlight key lessons and performance to inform future strategic and operational decisions (see Table 6). Table 6: Key evaluation questions ) Have literacy rates of school-age children improved over the duration of the program? If so, how and why? For example, are students able to read grade-level text? Are teachers demonstrating new methods of teaching? 2) Has the use of health and dietary practices increased? If so, how? Has illness-related absence decreased? Are students washing their hands? Are schools and school kitchens clean? How are school gardens being used? 3) What is the level of community-level involvement and participation in decision-making in school governance mechanisms (School General Assembly Committees and School Management Committees [SMCs])? Particularly, what is the level of involvement and participation of women? Also, what is the level and sustained continuity of community contributions in cash and in kind? 4) What are the key institutions and governance structures required to effectively deliver, implement, and sustain school meal interventions? What relationship structures among these institutions yield the most successful and effective school meal programmes? Is WFP’s capacity support to smallholder farmers and key line ministries appropriate/sufficient to effectively facilitate national ownership? Has the provided capacity support increased the Government’s capacity to own and sustain a national school meals program? 51. In addition, the country office has emphasized the following evaluation priorities: • Use of primary quantitative data to assess results in literacy and WASH (e.g., via the Early Grade Reading Assessment (EGRA) tool); and • How can WFP work more strategically in its capacity development efforts with Government? What capacity development approaches/activities are working/not working? Where do things stand in terms of being on track for transition of the programme into the national school feeding programme and government ownership? 52. There was also interest in understanding the impact of COVID-19 prevention measures and school closures in 2020. 53. To meet evaluation expectations across these multiple areas of inquiry, the ET tailored the OECD-DAC evaluation questions and sub-questions to the country context and operating environment, and combined these with the key evaluation questions and priorities into an evaluation matrix. The matrix enumerates all questions and their corresponding analysis methods. This was initially done for the midterm evaluation, and again reviewed and validated by WFP in the endline inception phase and finalized before fieldwork. Special attention was given to gender equality in the formulation and administration of the questions and data collection tools (see Annex 8). The evaluation matrix designed at midterm has been reorganized at endline to more clearly harmonize the interrelationships of the OECD-DAC criteria and the key evaluation questions.76 The evaluation matrix (Annex 9) reflects the sum of the questions, criteria and priorities expressed in the TOR and by the country office; it streamlines the organization and content of the original matrix, while retaining all original questions. The matrix now organizes all questions under the appropriate OECD-DAC criteria, with an additional section on factors affecting results, and is the basis for the organization of findings reported in Section 2 of this report. 76 The ET acknowledges that the evalation matrix typically does not change at the reporting stage and that this diverges from Decentralized Evaluation Quality Assurance System (DEQAS) guidelines. However, the endline inception report (including the evaluation matrix) did not undergo DEQAS review, a country office decision based on the need to move forward with the evaluation after COVID-related delays and to complete the endine evaluation in time to inform the baseline study for Phase II of the programme. The reorganization of the evaluation matrix responds to comments from the external DEQAS review of the draft endline report around harmonizing the various sets of questions presented in the TOR. The reorganization was made with the approval of the CO, which reviewed and approved the final version of the matrix. Date | Report Number 15 Methodology77 54. The evaluation used a mixed-methods approach to assess programme performance and the factors that affected performance. This approach combined a desk review; a school headmaster survey, EGRA survey,78 qualitative interviews with key informants and focus groups (see list in Annex 11); observation at schools; and examination of quantitative data from WFP and partner reports. National and subnational lines of qualitative inquiry focused on the programme’s alignment with government strategies and policies, transition planning, and strengthening capacities for the national school feeding programme. School-level data collection focused on assessing progress toward targets and quality and impact of activities, outputs and outcomes. 55. The endline sampling methodology for school-based data collection activities replicated the sampling methodology used at midterm.79 The first stage of the sample selected 20 schools from the 104 schools receiving support at the time of the midterm evaluation, using probability-proportional-to-size (PPS) systematic random sampling. The ET acknowledges that the programme has since added three schools (plus a fourth, in a no-cost extension in mid-2021); this was a very small percentage increase in the sample frame and thus did not require a new calculation. Moreover, the three new schools only received a meal and a few selected activities,80 thus other areas of the programme that were measured at midterm could be not assessed such as reading comprehension and WASH in the new schools. The same 20 schools were visited at midterm and endline, allowing a panel study at school level. The EGRA and additional health/WASH questions were administered to 440 third-grade students (220 M and 220 F), maintaining grade-level comparability with the baseline. Analysis 56. Qualitative analysis involved first entering all interview notes into a data matrix organized according to the categories explicit in the evaluation questions. This data matrix is the core set of information used for the analysis. Content analysis was then used to identify key messages around each evaluation theme as well as variability in the message from one stakeholder group to another. 57. Quantitative analysis was conducted using STATA version 15.0 software. This included descriptive analysis such as cross-tabulations, means, and significance tests to provide additional context to the underlying components of key indicators. The manner in which data for the programme was monitored and collected does not allow for sophisticated statistical analysis or cross-analysis of select project indicators, e.g., the installation of WASH facilities and girls’ attendance. Results are presented for the sample as a whole; while in some cases results are also presented at district level, the methodology does not allow for statistically valid comparisons across districts. 81 Due to budget constraints, the evaluation design did not allow for advanced analysis beyond calculation of point estimates and longitudinal performance comparisons for key indicators, and did not compensate for programme monitoring deficits. That said, in the view of the ET the midterm and endline design are considered an efficient and fit-for-purpose evaluation option for the programme. 58. The evaluation report includes outcome and output data for all McGovern-Dole indicators in Annex 6 in the interest of providing complete performance data for all stakeholders, especially USDA. The data for these indicators are extensive and serve as reference; the analysis in this report focuses on those indicators most relevant to answering the evaluation questions – primarily the outcome indicators. 77 See full details of the methodology in Annex 10. 78 We report comparative data from the EGRA baseline survey (data collected June 2016), EGRA midterm survey (data collected September – October 2018), and EGRA endline survey (data collected February 2021; see field schedule in Annex 12). 79 Details on school and student sampling methodology are given in Annex 10. 80 The new schools receive food and new kitchens, storerooms and stoves. 81 This would require a larger sample size, which was beyond the budget available. Date | Report Number 16 Gender considerations 59. All methods of inquiry included review of evidence of the programme’s attention to gender equity and women’s empowerment (GEWE). This included but was not limited to the review of GEWE aspects of programme design, strategy, and implementation as reflected in programme and government documents, inclusion of GEWE questions in interviews and focus groups, and assessment of gender-disaggregated data. Triangulation checks on validity and reliability 60. Primary quantitative data from the EGRA and school surveys82 were triangulated with quarterly and annual report data and with qualitative findings to assess the validity, reliability and consistency of the data and concurrence of all evidence sources regarding results. The ET also checked for internal consistency in WFP reporting on McGovern-Dole indicators, identified quantitative data discrepancies, and worked with the country office to make needed clarifications and corrections, which are reflected in this report. 61. The evaluation also examined all McGovern Dole outcome indicators where end-of-project achievement varies more than 10 percent from the target in either direction and sought to explain any variances beyond those parameters. This often entailed requesting clarification from the CO. Where the variance could be explained, this was either noted in a table note and/or, where there were implications for answering the evaluation questions, incorporated into the analysis in the main narrative. 62. The ET faced some challenges in interpreting the indicator names and calculations; several indicators are phrased identically but are considered outcome indicators when data are computed on a cumulative basis, and are considered output indicators when (the same) data are computed on an annual basis. This is a non-standard distinction between outcome and output indicators. There is also some redundancy in indicators; for example, “Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance” (Table 33) and Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance” (Table 34) report exactly the same data. The ET has discussed these issues of definitions and efficiencies with the country office, and the country office has indicated its intent to apply these lessons in creating a more streamlined indicator set in Phase II. Ethical considerations 63. The ET observed ethical research practices such as informed consent, protecting personally identifying information, and child protection protocols. The ethical safeguards used by the ET are discussed in Annex 10, Section J. Constraints and limitations 64. The implementation of the original evaluation plan encountered two main constraints. First, the inception phase was delayed from mid-2020 until November 2020 due to COVID-19, and data collection was delayed until February 2021 due to pandemic-related in-country movement restrictions. Second, the international consultants could not travel to Rwanda due to COVID-19 restrictions. Instead, they conducted remote interviews and the national team collected data in the field. This approach was effective for data collection but limited the informal interaction that takes place during in-country work. To address this challenge, weekly check-ins were organized to enable essential space for iteration and reflection between the ET and the McGovern-Dole Programme. 65. The main methodological limitation, also noted in the midterm report, was that some of the baseline data had limitations that did not allow comparability with the midterm and endline evaluations. This was due largely to the presence, in the baseline, of non-response data in the reading comprehension questions. 82 The last EGRA was conducted at midterm using Tangerine software; the complete midterm EGRA source files were no longer available and had to be rebuilt for the endline. This challenge was successfully resolved through support from World Vision and extra resourcing provided by TANGO. Date | Report Number 17 Therefore, most of the comparisons in this report are to the MTE results. The ET did re-estimate baseline values for EGRA scores to allow comparability on reading comprehension.83 2. Evaluation Findings 2.1 RELEVANCE Relevance to needs 66. WFP’s targeting of the most food insecure districts is appropriate to geographic needs and to the needs of primary schools, which are in some of the poorest areas in those districts. The original school selection had input from DEOs, local mayors, and World Vision, and was verified by WFP, MINEDUC, and World Vision. Teachers, DEOs, and WFP staff cited school meals as relevant to children’s nutrition needs and having increased regular attendance and reduced dropouts. Outcome data from the project shows that regular school attendance84 increased from 67.1 percent at baseline to 96 percent at endline (percentages calculated against final attendance targets) (Annex 6, Table 48). District officials interviewed at endline indicate the project is aligned with the Government’s goal of ensuring student attendance and retention. The programme provides the same meal content and portions to girls and boys. However, going beyond the basics of school meal content, WFP’s work on nutrient gap analysis for the national school feeding programme has expanded thinking on school nutrition and has increased gender sensitivity toward the nutritional needs of adolescent girls, who may still be matriculating in primary school. WFP has worked with MINEDUC on constructing local school meal menus to address the specific nutritional needs of adolescent girls (e.g., iron-rich foods). 67. In addition to supporting food security and access to education, the programme has provided access to water, sanitation facilities, nutrition education, and literacy training in very poor areas, all of which are highly relevant to children’s health and increases their ability to participate in an improved learning environment. These activities are relevant to local needs, in that rural schools in food insecure areas lack basic infrastructure, including WASH facilities, students have limited access to reading material and few resources for reading outside of school, and families struggle to feed their children an adequate diet. 68. The relevance of the programme was clear in interviews with MINEDUC and district officials, who credited a noticeable improvement in test scores in schools supported by the McGovern-Dole Programme to the school meals and the literacy activities.85 MINEDUC sees the programme as very important to the retention and health of primary school students, though small in scope in relation to the national school system. Notwithstanding the challenges to engaging community support for school feeding mentioned above, this strategy is nonetheless relevant to create greater ownership and interest among parents in the components of the McGovern-Dole Programme and their importance to a quality education. 69. In terms of the programme’s work with smallholder farmers, it provides support to both women and men, including women-run cooperatives, which is consistent with Rwanda’s National Cooperative Policy promoting membership for women and gender equality in cooperatives.86 Cooperatives are owned by their members; in the WFP Rwanda school feeding portfolio, women comprise 52 percent of the total membership of supported cooperatives. WFP’s support for female cooperatives and members is important as, according to a recent government report, women’s membership (nationally) is only 42 percent 83 The ET was unable to replicate the results reported in the baseline report, and therefore re-estimated the baseline values by applying the same methodology used at midterm to the raw baseline data. 84 “Regular” is defined as attending 80 percent or more of the total days school is in session. 85 This is consistent with the improved results measured by the EGRA, discussed in Section 2.2, Key EQ 1. 86 Republic of Rwanda, Ministry of Education. 2014. National School Health Policy, Kigali 2014. Republic of Rwanda. National Cooperative Policy in Rwanda Toward Private Cooperative Enterprises and Business Entities for Socio-Economic Transformation. Revised version [1]. Kigali, January 15, 2018. Date | Report Number 18 compared to 52 percent for men (2015). 87 Women’s lower rate of membership is attributed to their disproportionate share of household duties, which limits their time to participate in outside activities. Coherence with national policies, strategies, and government initiative 70. National school feeding policy. The McGovern-Dole Programme contributes to national development goals on education, food and nutrition security and capacity strengthening. There is a strong coherence with government policies and plans including increasing the literacy rate to 99 percent by 2050 and universal pre-primary enrolment rates by 2050 from 17.5 percent (2016).88 In addition, the programme has provided appropriate and flexible support to school feeding policy development, which has been ongoing throughout the programme, and supported the establishment of associated governance systems and functional processes, as described in the paragraphs below. 71. The McGovern-Dole Programme was designed to align with the Government of Rwanda’s implicit and subsequently declared policy priority to scale up a national school feeding programme. The Government finalized its school feeding policy and strategy in 2019. In anticipation of this action, WFP began work in 2019 with the Government on the School Feeding Operational Guidelines to support the Government’s school feeding strategy. The policy, strategy, and guidelines were approved by MINEDUC in 2021 and form the basis of trainings for key stakeholders in the national scale-up. A School Feeding Programme unit was created in MINEDUC to support the roll-out, with WFP assisting with the institutionalization of the unit by developing job profiles for its staff. Following its adoption of a National School Feeding Policy and strategy in 2019, in 2020 the Government demonstrated its commitment to scaling up school feeding to reach 3.3 million pre-primary, primary, and secondary students.89 It increased the school feeding budget from US $5 million in 2014 to US$ 37 million for school meals and US $19 million for kitchen and stove construction in 2020-2021. 72. Interviews across a range of stakeholders confirm the view among senior government officials that the McGovern-Dole Programme has been an important contributor to developing the national programme. Government officials stated that the work with WFP has connected education officials to a global school feeding network that allows the Government to benchmark its progress against other countries as the economy of Rwanda grows. At the local level, district officials interviewed indicate the project is aligned with the Government’s Vision 2050 goal of poverty reduction. 73. Previously, the main coordination mechanisms between the programme and its government partners were semi-annual steering committee meetings led by MINEDUC and quarterly coordination meetings led by MINEDUC and WFP. In late 2019, WFP supported the Government to establish a National School Feeding Technical Working Group (TWG) under the Education Sector Working Group. The TWG is chaired by MINEDUC and co-chaired by WFP and meets quarterly. It has grown into the main coordination mechanism to guide not only the McGovern-Dole Programme but all school feeding efforts in Rwanda. In October 2020, WFP reported that with the end of the project, all programme governance would shift from the Steering Committee to the National School Feeding TWG90 and to the new National School Feeding Steering Committee.91 87 Republic of Rwanda, Gender Monitoring Office. 2019. The State of Gender Equality in Rwanda from Transition to Transformation. March. 88 Republic of Rwanda. Ministry of Finance and Economic Planning. Vision 2050. https://www.nirda.gov.rw/uploads/tx_dce/Vision_English_Version_2050_-31_Dec_2020.pdf 89 Draft review comment from WFP Rwanda. 90 Membership of National School Feeding TWG: MINEDUC (chair), MINALOC, MINAGRI, Ministry of Health, Ministry of Trade and Industry, Rwanda Basic Education Board, National Childhood Development Agency, Rwanda Cooperative Agency, Rwanda Food and Drugs Authority, and Rwanda Biomedical Center, WFP (co-chair), FAO, UNICEF, USAID, World Vision International, Japan International Cooperation Agency, European Union, Africa New Life Ministries, FH Association Rwanda, Movement for the Fight Against Hunger in the World, The Wellspring Foundation for Education, Water Aid. Source: Rwanda School Feeding Operational Guidelines Summary. 91 Membership of National School Feeding Steering Committee: MINEDUC (chair), MINALOC, Ministry of Gender and Family Promotion, MINAGRI (co-chair), Ministry of Health, Ministry of Finance and Economic Planning, Ministry of Trade and Industry, Ministry of Environment, Ministry of Infrastructure, Rwanda Basic Education Board, Rwanda Public Procurement Authority, Date | Report Number 19 74. Below the national level, programme coherence is monitored regularly through quarterly technical meetings with all stakeholders at district level and regular monitoring meetings, also at district level. These district meetings have played an important role in bridging the national – subnational gap identified as a sustainability risk during the MTE. In Karongi district, a recent resolution from a technical committee meeting called for the inclusion of school feeding in 2019-2020 imihigo (performance contracts) in each sector where the programme operates. This is an encouraging development as it formalizes local commitments to school feeding 92 though it remains to be seen if other districts adopt the same approach. 75. Gender policies. The McGovern-Dole Programme is aligned with the Government’s policies on gender equality, which are well articulated in Rwanda’s legal, policy and strategic frameworks. The programme is also aligned with the WFP Gender Policy 2015-2020. Equal access to education is addressed in Rwanda’s Girls’ Education Policy (2008) which aims to eliminate gender disparities in education.93 Rwanda has achieved gender parity in education, with a net national enrolment rate of 98.6 percent for girls and 98.4 percent for boys (2019).94 The programme is further aligned with the National Gender Policy (2010) whose goal is to integrate gender equality and equity into government planning and programmes for social and economic development, including addressing the social, cultural and economic factors that prevent girls from enjoying equal access to education.95 The McGovern-Dole Programme provides equal access to school meals, handwashing, literacy clubs, gardens and nutrition education for girls and boys and sensitizes communities and parents to the importance of education for girls as well as boys. The programme has made a significant contribution to the retention of girls in school by constructing girls’ sanitation rooms which enable adolescent girls to continue to attend school during their menstrual cycle. 76. The McGovern-Dole Programme support to agricultural cooperatives aligns with the Government’s Strategic Plan for Agriculture Transformation (2018-2024), which contains specific strategies to address gender issues in agriculture through its inclusion of women-run cooperatives and cooperatives with a majority of female members, enabling women to build capacity to provision local schools. Complementarity to the policies, priorities, and interventions of donors, relevant government entities, and NGO partners 77. United States Government. The McGovern-Dole Programme is consistent with the United States Foreign Agricultural Service Strategic Plan, specifically Goal 4, Objective 4.1: Implement non-trade-focused congressionally mandated programmes. 96 The programme contributes to the Foreign Agricultural Service’s sustainability goal by strengthening government capacity to implement a national school feeding programme and to assume full management of the school meals activity implemented under the McGovern-Dole Programme, although on a more limited scale. By helping local agricultural cooperatives build their capacity to supply food for school meals, the McGovern-Dole Programme helps to ensure a sustainable source of supply for school feeding programmes that will also help communities to develop new markets and greater resilience.97, 98 In order to build on complementary partnerships and ensure coordination among development actors, WFP has signed a Memorandum of Understanding (MoU) with Rwanda TVET Board, National Childhood Development Agency, Rwanda Cooperative Agency, Rwanda Food and Drugs Authority, Rwanda Agriculture Board, WFP (co-chair), FAO, UNICEF, USAID, DFID, World Vision International. Source: Rwanda School Feeding Operational Guidelines Summary. 92 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 93 Republic of Rwanda, Gender Monitoring Office. 2019.The State of Gender Equality in Rwanda from Transition to Transformation. March. 94 National Institute of Statistics of Rwanda. 2020. Rwanda Statistical Yearbook 2020, December 2020. 95 Republic of Rwanda, Ministry of Gender and Family Promotion. 2010. National Gender Policy, July 2010. 96 United States Department of Agriculture, Foreign Agricultural Service. NDa. Foreign Agricultural Service Strategic Plan (2019- 2022). 97 United States Department of Agriculture, Foreign Agricultural Service. NDb. FY 2019 Food Assistance Proposal Guidance and Request for Applications. 98 See also the discussion on impact and sustainability in Section 3.1. Date | Report Number 20 the USAID Feed the Future ‘Hinga Weze’ programme. Of the 10 districts targeted by Hinga Weze, three districts overlap with the McGovern-Dole Programme (Rutsiro, Karongi and Nyamagabe).99 78. Government of Rwanda. The WFP Rwanda 2015 McGovern-Dole proposal states that the programme was designed to support the National School Feeding Programme and contribute to its continued development and sustainability. The Government’s National School Feeding policy endorses a school feeding programme based on local purchase of commodities with a view to eventual nationwide implementation without external support. The Education Sector Strategic Plan 2018/2019-2023/2024 and the National School Health Policy 2014 mention the Government’s intention to create a National School Feeding Programme. The focus on primary school aligns closely with the Government’s goal of reducing stunting and malnutrition, and to improve literacy and learning outcomes. In addition, the National School Health Policy 2014 recognizes the importance of deworming to prevent malnutrition and anemia in school-age children. Deworming services are normally provided to schools by local health officials. The McGovern-Dole Programme supported this health objective by partnering with the Rwanda Biomedical Centre to pilot the distribution of deworming medicine by teachers. This teacher distribution model has been adopted by the Government and scaled up nationally. 79. As part of the Rwanda Country Strategic Plan 2019—2023, WFP interventions in Rwanda also include Phase 3 of the Sustainable Market Alliance and Assets Creation for Resilient Communities and Gender Transformation project (2020 to 2023), and the Sustainable Agricultural Productivity and Market Linkages project (2021-2024). WFP and the Food and Agriculture Organization (FAO) have longstanding collaboration to support smallholder farmers to access markets, such as through Purchase for Progress (P4P), which was a separate activity under the country office during the first part of the programme but has since merged with other agricultural support activities. The Government has integrated lessons learned from P4P to create the Common P4P (CP4P), which purchases food from smallholder cooperatives for the National Strategic Reserve.100 In March 2020, WFP transitioned from the USDA-funded LRP project to approach working with the private sector to develop marketing and distribution models that are accessible to farmers. This is a general approach for all WFP-supported cooperatives in the country which also applies to MGD interventions.101 80. NGOs. Synergies with NGO partners have been strategic and, as a result, successful in meeting evolving beneficiary and stakeholder needs. World Vision, a primary partner in the McGovern-Dole Programme, targets vulnerable children and their families in a number of areas that are congruent with McGovern-Dole interventions, including education, literacy, health and nutrition, and water, sanitation and hygiene. The partnership with World Vision was critical for the programme to meet learner needs during the COVID-19 suspension of school, for example, by reallocating programme resources to strengthen community literacy activities. World Vision’s experience with WASH interventions strengthened that component of the McGovern-Dole Programme in the second half of the programme, with separate latrines for girls and disabled students, piped water, permanent handwashing stations, and sanitation rooms for girls. 81. There has been positive adaptive management of partnerships and partner focus throughout the programme. A partnership with the Adventist Development and Relief Association (ADRA) early in the programme to build kitchens was dropped due to problems with the quality of the construction. WFP then coordinated construction with private firms and directly managed the completion of remaining. Earlier in the programme, World Vision focused on building or enhancing rainwater collection systems; after the MTE, the programme shifted to connecting schools to a permanent water source in close collaboration with the Government, i.e., MINALOC. World Vision also demonstrated adaptive management during the school closure by shifting reading clubs to book lending and replacing temporary handwashing stations with permanent structures. 99 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 100 WFP. 2014. Purchase for Progress - P4P Rwanda. 101 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). Date | Report Number 21 82. In another example of partnership management, FAO was part of the initial design for school gardens but did not participate due to budgetary limitations, which delayed the start of this activity. This workstream was then shifted to a new partner GHI, who brought significant value added with experience in practical nutrition education. GHI works to address the root causes of chronic malnutrition, especially among children. Its activities include the integration of agriculture and nutrition among Rwandan families, many of whom are farmers but who lack knowledge on nutrition. GHI promotes skills and knowledge needed to achieve healthy diets through the cultivation of home gardens and school gardens using readily available local resources that are accessible to poor families. Coherence with WFP and UN-wide strategies, policies and normative guidance 83. WFP Corporate Strategic Plan (CSP) and Sustainable Development Goals (SDGs). The McGovern-Dole Programme aligns with the WFP CSP (2017-2021), which is guided by SDGs 2 and 17 of the 2030 Agenda for Sustainable Development. The programme supports WFP’s corporate SO1, end hunger by protecting access to food, improve nutrition, and achieve food security, which aligns with SDG 2. 84. The programme also supports WFP CSP SO2, partner to support SDG implementation, which includes strengthening country capacities for implementation and aligns with SDG 17. This is reflected in part in WFP’s strategy of embedding staff in MINEDUC and MINAGRI to coordinate USDA partners and track implementation and is a useful way to foster collaboration, ensure alignment of policies and strategies, and build government capacity for school feeding. The selection of technically skilled partners to implement the literacy, WASH, and nutrition components further supports the principle of partnering to achieve SDG results. 85. The programme component that develops the capacity of smallholder cooperatives to eventually supply commodities for school feeding is aligned with WFP CSP SO3, to achieve food security. The ET noted in the MTE that the USDA requirement to use imported food is not aligned with WFP’s policy supporting local procurement. The McGovern-Dole Programme was developing the capacity of local smallholder cooperatives to provide maize directly to local agro-processors producing CSB+, which would enable processors to source maize in-country instead of importing it. WFP is continuing its work to strengthen agricultural cooperatives to play an important role in the provisioning of school meals. 86. As achieving middle-income status is part of Rwanda’s national vision, the McGovern-Dole Programme also aligns with WFP’s strategic concern about assisting vulnerable populations within middle-income countries. 87. WFP policies. The McGovern-Dole Programme in Rwanda is aligned with WFP’s School Feeding Policy (2013) which focuses on helping countries to establish and maintain nationally owned programmes linked to local agricultural production. 102 It also aligns with the WFP School Feeding Strategy 2020-2030 focus to establish and maintain government-led school feeding programmes through technical support and capacity development, and to develop links with smallholder agricultural producers to supply schools, support livelihoods, and strengthen market linkages. 88. WFP’s support to government in strengthening capacity for the national school feeding programme is highly coherent with WFP’s shift to an upstream role and its global strategic focus on helping countries to strengthen national policies and systems.103 89. The McGovern-Dole Programme is still guided by the current WFP 2015-2020 Gender Policy:104 it promotes the participation of girls and indicator data are disaggregated by gender. The programme provides for the construction of gender-sensitive toilet facilities for girls (as well separate, non-gendered toilets for disabled students). WFP has worked to sensitize MINEDUC, school administrators, teachers, and cooks on gender equality and to encourage greater female participation in the programme. In 2019, WFP commissioned two studies on gender dynamics and access to credit within the bean value chain, which led to gender sensitization training for WFP supported cooperatives, including those currently participating in the 102 WFP November 2013. Revised School Feeding Policy. 103 WFP. 2017. WFP Strategic Plan (2017-2021). 104 Initially the WFP 2009 policy, later subsumed by the WFP 2015-20 Gender Policy. Evaluation TOR. Date | Report Number 22 McGovern-Dole Programme. The MTE noted that the McGovern-Dole Programme did not include a specific approach to address gender equality and women’s empowerment. This was still the case at endline. 90. United Nations. The United Nations agencies in Rwanda coordinate closely through One UN initiative. Since 2008, agencies have coordinated their activities for economic and social transformation and transformational governance through the United Nations Development Assistance Programme (now UNDAP II, 2018-2023). United Nations inter-agency collaboration includes WFP and UNHCR’s joint work on refugee issues and WFP, UNICEF and UNESCO collaboration on education activities such as creating Child‐ Friendly School standards and assessing literacy and numeracy. UNICEF has been an important partner in WFP’s work in education. UNICEF is co-chair with MINEDUC of the Education Sector Working Group (ESWG), which provides a forum for cross-sector planning. WFP is an active member of the ESWG and supported the Government to establish a TWG on school feeding under the ESWG in 2019.105 UNICEF collaborates closely with WFP in education policy and technical approaches to nutrition and WASH. UNICEF partnered with WFP, World Vision, and MINEDUC to design low-cost permanent handwashing stations for schools in response to the pandemic, and to jointly advocate for construction funds. WFP worked closely with FAO on a framework for the NSFP operational guidelines. Key Findings and Conclusions – Relevance 91. The McGovern-Dole Programme is highly relevant to the needs of schools and their educational goals in some of Rwanda’s poorest areas, supporting not only improved learning, but providing the infrastructure and complementary activities to support better learning outcomes and improved teaching and school management. 92. The McGovern-Dole Programme aligns with national priorities on education, school meals, improved food security and nutrition, community participation, gender equity, and agricultural development. 93. WFP has effectively identified where the McGovern-Dole Programme fits with government strategy on school feeding and has been able to amplify the impact of the government programme by providing technical assistance for the national school feeding programme expansion through appropriate capacity strengthening and accompaniment models. 2.2 EFFECTIVENESS Key EQ 1: Have literacy rates of school-age children improved over the duration of the program? 94. As measured by changes in key indicators between the start of the programme and the final evaluation, literacy results and performance have been strong, with most targets met or exceeded, despite the prolonged closure of schools due to COVID-19. One important outcome indicator is the percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text. The endline EGRA survey finds that the end-of-programme target for girls and boys combined was well exceeded (77.7 percent actual versus 69 percent target), as were the gender￾disaggregated targets (Annex 13, Table 99). As shown in Figure 1, there was strong, steady growth in students’ reading comprehension relative to programme expectations over time. Figure 1 further shows that girls outperformed boys in reading comprehension over the life of the programme. Qualitative interviews yielded several explanations for better performance by girls. District education officials in Rutsiro said that girls participate more in reading clubs than boys. Teachers confirmed that girls perform better at reading than boys and said that boys spend more time playing games. SGAC members in Rutsiro said that since the school feeding programme started, girls attend classes more frequently than boys, which contributes to better reading scores. Parents interviewed in Rutsiro confirmed this view, as did MINEDUC officials. 105 WFP. 2019. MGD Semi-Annual Report (Oct 1, 2019 – March 31, 2020). Date | Report Number 23 Figure 1: Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade￾level text Source: Baseline. midterm and endline EGRA surveys 95. The analysis in Table 7 indicates that third-graders’ reading performance – as measured by various reading tasks – improved at both midterm and endline: all changes were positive and statistically significant compared to baseline. The data in Table 8 also show improvements in third-graders’ reading comprehension, as measured by the percentage of students who could answer more questions correctly: at midterm, 55.1 percent of third-graders exceeded the standard and at endline, 83.2 percent. Key informant and focus group responses indicate that community reading volunteers were the main reason for continued improvement while schools were closed due to COVID-19. The role of community volunteers was critical in distributing reading materials at home and encouraging parents to read with their children; or, if parents were illiterate, to have the children tell them what they read. In some cases, older siblings who were home from boarding school also played a key role in establishing a supportive reading environment in the household. Table 7: Reading performance for third-grade students at endline Indicator Baseline Midterm Endline Reported Re￾estimated a # of Tasks Mean per Minute # of Tasks Mean per Minute # of Tasks Reading letter-sounds 11.0 16.4 100 49.0*** 95 56.8*** 101 Reading syllables 25.0 45.6 100 55.8*** 100 59.5*** 101 Familiar words 11.0 21.1 50 29.9*** 50 26.9*** 50 Unfamiliar words 7.0 14.8 50 33.7*** 50 32.5*** 50 Correct words in text/story 10.0 20.1 65 22.8** 56 56.2*** 79 n n/a 402 441 435 Source: Baseline, midterm and endline EGRA surveys. Statistically significant at the .10 (*), .05 (**) or .01 (***) level with the re-estimated baseline data a The presence of non-response data for reading comprehension in the baseline data reduced the comparability from endline to baseline. Baseline values were re-estimated to address this constraint. 41.5 39.8 43.3 59 56.8 62.9 77.7 70.2 85 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 Total Boys girls BL (2016) MT (2019) EL (2021) Date | Report Number 24 Table 8: Reading comprehension, third grade (all students to read and/or listen) Total correct answers to comprehension questions Rating (aligned to 2012 NSC proposal)1 Midterm Endline % of students to achieve Total % % of students to achieve Total % 0 (0% of total questions) Does not meet standard (0-44% correct) 4.3 17.3 0 1 (20%) 4.9 Does not meet standard (0-44% correct) 3.2 0.5 2 (40%) Does not meet standard (0-44% correct) 9.8 4.4 3 (60%) Meets standard (45-69% correct) 27.7 27.7 12 12 4 (80%) Exceeds standard (70+% correct) 37.6 55.1 30.1 83.2 5 (100%) 17.5 53.1 n= 441 n= 435 Source: Midterm and endline EGRA survey. 1 NSC = National Standards Committee 96. The EGRA survey results also indicate midterm-to-endline improvements in students’ ability to read and comprehend a short story, in reading speed (as measured by number of words correct per minute), and in the ability to read and understand grade-level text (see Annex 13, Table 100 through Table 102). Midterm￾to-endline increases are also seen in the percentage of students who get reading help from their parents, and in students’ time available to study and do homework (see Annex 13, Table 103). Across all grades (1-6), there were sizeable increases in the percent of students identified by their teachers as attentive (see Annex 13, Table 104). 97. While EGRA results for the McGovern-Dole project are positive, they cannot be compared with non-project schools or national averages as comparable studies are not available.106 The McGovern-Dole Programme does not have performance benchmarks for reading and comprehension and national benchmarks are not yet established. In 2019, the USAID Soma Umenye project and the Rwanda Education Board developed early grade reading benchmarks for Kinyarwanda oral reading fluency and reading comprehension for grades 1, 2, and 3, and the Local Early Grade Reading Assessment (LEGRA) to measure literacy performance against the benchmarks.107 However, these results are not directly comparable with the midterm and endline EGRA survey data used in the current evaluation due to methodology, survey timing, sample size, and geographic location of the EGRA and the LEGRA pilots. MGD 1.1: Improved Quality of Literacy Instruction 98. The programme focused on improving the learning environment in schools by training teachers in new, higher quality teaching techniques and tools and by promoting a more consistent classroom presence by teachers. Teachers who showed exemplary teaching performance were recognized with teaching awards by World Vision in 2018 and 2019 (see Annex 6, Table 19).108,109 This likely contributed to exceeding programme targets, with 1,240 teachers using new techniques and tools (126 percent of target), and 1,680 teachers attending school at least 90 percent of the time (257.3 percent of target) (Annex 6, Table 17 to Table 19). 99. The programme has effectively increased access to books and other learning materials, which were in critically short supply at the outset of the programme. Consequently, World Vision distributed all planned 106 USAID’s Soma Umenye project refers to early grade reading benchmarks; see paragraph 97. 107 USAID. 2020. “Early Grade Reading in Rwanda: What Does ‘Good’ Look Like and How Do We Measure It?” October 21. https://www.globalreadingnetwork.net/learning/early-grade-reading-what-does-good-look-and-how-do-we-measure-it 108 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 109 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). Date | Report Number 25 school supplies to the 104 schools in 2017.110 For the 2017 distribution, World Vision provided more than double the planned number of textbooks and teaching and learning materials (learning starter kits) to schools (Annex 6, Table 20 and Table 21). The number of students benefitting from the starter kits far exceeded the annual target of 20,750 in all five years since the distribution; at the highest, 96,041 students benefitted (2018; 463 percent of target) and at the lowest, 41,877 students (2020; 202 percent of target). The ET observed a print-rich environment in school classrooms during both the midterm and final evaluations. 100.Since FY2017, all programme schools were provided with supplemental reading material. (Annex 6, Table 23 through Table 27). Literacy materials enabled the programme to reach into the communities, with 312 weekend reading clubs established in the school catchment areas (Annex 6, Table 58). Each club is supported by three facilitators (though some clubs have fewer), trained by World Vision on the Literacy Boost Community Action model.111 School librarians were trained on library management techniques and on the new Reading club Toolkit. To encourage reading club facilitators to continue volunteering, World Vision distributed smart phones for communication between facilitators, Kinyarwanda model teachers and project staff. 101.Due to COVID-19 restrictions, reading clubs were unable to meet during the school closure. To ensure continued access to reading materials, World Vision organized door-to-door book lending with grade 1-3 students.112 Reading facilitators told ET interviewers that outreach and follow-up was not easy, especially to distant households, but parents ensured that books loaned to students were kept in good condition at home, and literate parents are helping their children with their reading and writing skills, a finding consistent with the EGRA survey data (see Annex 13, Table 103). 102.To improve the teaching of literacy skills, the programme trained and certified 1,509 teachers, educators, and teaching assistants (target: 816) (Annex 6, Table 28). Peer observations by teachers and classroom observations by programme staff, Sector Education Inspectors and headmasters indicate a yearly increase in the number of teachers demonstrating improved literacy-teaching skills. (Annex 6, Table 28 and Table 29). 103.World Vision conducts quarterly refresher trainings for teachers. Classroom observations confirmed that every trained teacher visited is using at least three to four Literacy Boost Techniques in each lesson.113 Interviews with teachers confirm they received training from World Vision on reading techniques and teaching methods, while some have received refresher trainings. During school closure, trainings were adapted and rolled out using a self-learning manual developed by World Vision’s Literacy Boost Coordinators and shared with teachers through WhatsApp groups. Hard copies were also distributed.114 104.Head teachers, school-based mentors, deans of studies, sector education officers, and district education officers were trained and certified on teacher coaching methods, again far exceeding the target (262 trained versus 139 target, an achievement of 188.5 percent) (Annex 6, Table 30). Coaching techniques included the creation of teacher development plans, classroom observation, giving feedback, the facilitation of learning circles and peer-to-peer observation.115 The endline school survey confirms that head teachers apply new techniques (Annex 13, Table 106). Trainings were adapted to COVID-19 restrictions using a self-learning approach, and ET interviews with administrators confirm they received training from World Vision on school management and planning. District officials and school personnel told the ET that the training of teachers and school administrators, especially the teacher coaching, are perceived as a strength of the McGovern-Dole Programme. 110 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 111 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 112 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 113 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 114 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 115 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). Date | Report Number 26 MGD 1.2: Improved Attentiveness 105.Attentiveness. The EGRA survey indicates that at endline, 74.1 percent of students in classrooms were identified as attentive by their teacher (Annex 13, Table 104). This is below the target of 80 percent, but still an improvement over 57.4 percent at midterm, with similar midterm-endline improvements at all grade levels (Annex 13, Table 105). Interviews with school administrators and teachers indicate that attention levels were slightly lower after schools reopened but were still higher than at midterm. 106.Beneficiaries and meals data. The endline beneficiary target is exceeded with 147,798 students receiving daily school meals against a target of 127,650 (115.8 percent of target) (Annex 6, Table 34). This kept up with enrolment, which rose steadily during the project period to 115.8 percent of target at endline (73,403 girls vs target 66,378 and 74,395 boys vs target 61,272 (Annex 6, Table 54). Yearly beneficiary targets decreased after 2017 (Annex 6, Table 36) due to new schools being established in the programme catchment areas and student being transferred to schools outside of the McGovern-Dole Program; 116 in those years the programme continued to exceed or nearly meet yearly targets for number of students served, with the lowest achievement level still high, at 94 percent in FY2020. 107.The overall number of school meals provided achieved 80.4 percent of target (see Annex 6, Table 32). The yearly targets for meals were almost met in 2017-2019, with over 14 million school meals provided in each of these three years (see Annex 6, Table 35). No in-school meals were provided between April and September 2020 because schools closed due to COVID-19.117 To enable continued support, WFP organized two take-home-ration (THR) distributions to students in all118 schools supported by the McGovern-Dole Programme.119 The THR were meant only for students, but interviews with SGACs indicate that THRs were shared within household and usually lasted a week or less (2-7 days maximum, depending on family size). From October 2020 to March 2021, fewer meals were distributed than in previous years due to COVID￾related school closures (see Annex 6, Table 35). 108.Planned commodities were provided to 58 schools in Karongi and Rutsiro districts, where students receive a daily meal of CSB+ and sugar while students in 49 schools in Nyaruguru and Nyamagabe districts receive a meal of beans, salt, fortified oil and fortified maize meal. USDA provides in-kind commodities of CSB+ and fortified oil while Mastercard provided funds to purchase beans, salt, sugar, and fortified maize meal. 109.Procurement. Beans and maize meal are procured locally while salt is procured regionally and sugar internationally.120 Endline school survey findings show small decreases in the percentages of schools receiving food from school gardens (50 percent at midterm vs 45 percent at endline), NGOs (100 percent at midterm vs 90 percent at endline)121, and parents (5 percent at midterm vs 0 percent at endline) while provision from local markets increased from 40 percent at midterm to 55 percent at endline (Annex 13, Table 107). 110.Meal timeliness and quality. Students and school personnel interviewed by the ET said they appreciated receiving timely meals. Students like the maize and porridge meals but complain that sometimes the beans are not well cooked. 111.Capacity strengthening. WFP provided capacity strengthening to 35 McGovern-Dole-supported cooperatives in southern and western provinces.122 This includes 16 cooperatives supported under the LRP 116 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 117 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 118 Three new schools were added in January 2020 in Karongi. Further, a satellite school that was previously created at one of the McGovern-Dole-supported schools in Karongi, has now been established as an official school. This former satellite school brings the total number of schools with these THR to 108. 119 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 120 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). 121 The survey question is Where is the food for the children obtained? This response option is worded NGOs provided; there is no response option for WFP. NGOs include WFP, and there are no agencies providing food to the schools. 122 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). Date | Report Number 27 Programme, which closed in September 2019.123 McGovern-Dole training encompasses business plan development, facilitating access to formal financial institutions, coaching, cooperative governance, financial management, access to markets, training on Good Agriculture Practices (GAP) and post-harvest handling and storage.124 112.The number of smallholder farmers benefiting from partnerships with schools grew from zero in 2016 (against a target of 5,935) to 3,922 at the midterm in 2018 (34 percent of the target of 11,500). At endline, 9,771 smallholder farmers benefited from partnerships with schools, 85 percent of the target of 11,500 farmers (Annex 6, Table 44). Due to time limitations, the ET visited four cooperatives, two of which had not yet sold products to WFP or schools, though the cooperatives view WFP and schools as their most promising market linkages. Cooperative members interviewed by the ET confirmed having received training, though some said only a few members were trained. Seventeen cooperatives have developed business plans. Interviews with MINAGRI indicated that farmer groups need comprehensive support in strengthening production and commercial capacity in order to supply schools, in addition to strengthening local-to-national market systems. 113.School gardens. The target for the establishment and maintenance of school gardens, led by GHI, was reached in 2018 (Annex 6, Table 46). The activity has expanded considerably since the midterm and now also includes nutrition activities/training for parents, school personnel, local government, and community members125. It further includes the development and distribution of educational material in schools; the creation of Nutrition Oversight Committees in all schools; meetings with cell leaders to develop supervision plans and the development of performance contracts with schools (Imihigo) to include nutrition-sensitive activities; the provision of seeds and garden support packages to schools126; cooking demonstrations and nutrition trainings; and support to schools to organize parent nutrition days. 127 114.During the school closure, GHI partnered with cell leaders and school authorities to identify and distribute vegetables grown in the school gardens to 4,498128 vulnerable families. A survey of 371 home gardens maintained by students who received seeds from 45 programme schools was conducted by GHI in December 2020. The survey found that schools provided nine types of vegetable seeds, with amaranth, kale, onion, and beets the most distributed, while the best-producing vegetables were spinach, chayote, celery, leeks, and nightshade. Students were growing an average of 3.4 types of vegetables.129 MGD 1.3: Improved Student Attendance 115.Attendance. The programme target for overall regular school attendance was nearly achieved, at 96 percent actual vs planned (Annex 6, Table 48). As shown in Figure 2, attendance climbed fairly steadily. By FY2021, attendance was 101.3 percent of target for boys and 91.2 percent for girls in FY2021. Teachers and SGACs told the ET that school meals motivate children to attend regularly. 123 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 124 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 125 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 126 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). 127 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 128 Number reflects 770 in April – September 2020 and 3,728 in October – December. Distribution was mostly in the Western Region (Rutsiro and Karongi). Source: GHI Quarterly Report October- December 2020. 129 GHI. Student Home Garden Survey. December 2020. Date | Report Number 28 Figure 2: Number of students regularly (80%) attending USDA- supported classroom/ school Source: WFP. Semi-annual donor reports 2016-2020 and ET consultation with WFP for FY2021 data. 116.The final target of 5 percent, for the percentage of students who miss more than 10 school days per year due to illness, was met every year from FY2017 to FY2021 inclusive (Annex 6, Table 49). In FY2021, the value for girls was 1.72 percent and 2.4 percent for boys. The small increase between FY2020 and FY2021 is likely due to COVID-19, as parents and teachers were more cautious about sending students who show signs of illness to school.130 At endline, teachers and SGAC members interviewed said that students look healthier and that dropout rates have decreased significantly. 117.GEWE. The programme and the Government view gender equity largely in terms of enrolment, attendance, retention, and access to education. In this respect, it has been successful; school administrators and staff told the EGRA/school survey team that girls’ attendance has increased over the course of the programme. Girls are active in the literacy clubs and perform better than boys on EGRA reading tests. 118.One programme contribution singled out by government, WFP, and school-based KIs as influencing gender equality and protection are the girls’ sanitation rooms, which reduce absenteeism because they enable girls to attend school during their menstrual period. Thirty-six percent of female students in the McGovern-Dole Programme are of adolescent age.131 Teachers noted that prior to having the sanitation rooms, some girls would fall behind in their studies and drop out of school. Key informants also said that teachers guide girls in the use of the sanitation rooms and are available to talk to the girls, so that the girls feel more secure at school. Nevertheless, there is room for a more gender-sensitive approach in establishing access to the sanitation rooms. The ET found that girls often must request the sanitation room key from a male teacher, which was awkward for the girls, and that sanitation rooms were located too close to classrooms and thus not in a private space. The Government has adopted the provision of girls’ sanitation rooms for all schools, contingent on available funding. However other stakeholders are also prioritizing sanitation rooms; Action Aid and World Vision have both constructed these in the McGovern-Dole Programme schools. World Vision has constructed 33 girls’ rooms in the McGovern-Dole schools, 72 of the 108 schools (67 percent) have girls' rooms. 132 130 WFP. 2021. MGD Semi-Annual Report (October 1, 2020 – March 31, 2021). 131 WFP Rwanda. 2021. Semi-annual student survey. March. 132 Per information provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. 42,226 48,943 55,135 55,137 60,507 43,409 50,327 56,707 56,707 62,079 85,635 99,201 111,841 111,841 122,586 127,650 40,000 60,000 80,000 100,000 120,000 2017 2018 2019 2020 2021 Female Male Total Target (total) Date | Report Number 29 119.Also, adoption in national schools is not only based on government budget but on other stakeholders also prioritizing girls' rooms. For example, Action Aid has constructed girls' rooms in a number of schools in Rwanda, even in some McGovern-Dole schools. While World Vision has constructed 33 girls' rooms in the programme, 72 of the 107 schools have girls' rooms (67% of our schools).133 120.Timeliness. Interviews with WFP staff cited internal qualitative monitoring that shows school meals decrease tardiness among students.134 Teachers and administrators said that the meals helped girls attend school on time, as before they were delayed preparing meals at home. Interviews with parents and SGACs confirm children’s eagerness to attend school on time knowing they will receive a meal. With children taken care of at school parents also indicated having more time for work. 121.Kitchens. At endline, all 104 schools have access to improved food preparation and storage equipment including new or rehabilitated kitchens, cook areas, storerooms, stoves and kitchen utensils. Kitchens, storerooms, and stoves were built under the supervision of two commercial contractors, district officials and WFP technical staff.135 Targets for the provision of fuel-efficient stoves are exceeded. (Annex 6, Table 50 and Table 52). 122.Latrines. At endline, targets for the construction or facilitation of latrines are exceeded: 476 stalls vs a target of 364 (130.8 percent achievement) and 34 latrines vs a target of 26 (130.8 percent achievement) (Annex 6, Table 51). The target was met in 2018 but due to the ongoing need for latrines at many programme schools, World Vision raised an additional US $90,000 from private partners to construct more latrines.136 123.Parent awareness of the importance of education. The percentage of parents in target communities who can name at least three benefits of primary education increased from 69.8 percent at midterm to 74.3 percent at endline (Annex 6, Table 108), but fell short of the target of 90 percent.137 World Vision completed the planned parent awareness workshops in early 2019, but planned additional activities based on the low progress on this indicator at midterm, including four radio programmes (two in the south and two in the west) aired by World Vision during COVID-19.138 The programmes covered reading awareness, the importance of education, parental engagement in children’s education, and good health and hygiene practices. A quick assessment conducted by reading club facilitators during house-to-house lending activities found that 4,038 parents (54 percent male, 46 percent female) listened to one or more of the programmes during this period. 124.Monthly workshops were organized to raise awareness on the importance of education among parents of children attending the programme’s reading clubs.139 The programme decided to include all parents of children enrolled in community reading clubs,140 which dramatically increased the number of parents trained in the last two years of the program: from 1,231 parents (85 percent of the annual target) in 2019 to 4,038 in 2020 (277 percent of target) and 19,134 in 2021 (1,314 percent of target) (Annex 6, Table 57). From 2017-2021, the programme trained 42,464 parents. Additionally, during house-to-house book lending visits, reading club facilitators distributed flyers to 19,134 parents (47 percent men, 53 percent women) stressing the importance of literacy and parents’ reading awareness.141 125.Community trainings. Between 2017 and 2021, trainings were provided to SGACs in all project schools (Annex 6, Table 62 and Table 63). Trainings and refreshers were provided by World Vision to SGAC members on the use of the Citizen Voice and Action (CVA) model to conduct community dialogue at 133 Per information provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. 134 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 135 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 136 WFP. 2020. MGD Semi-Annual Report ((Oct 1, 2019 – March 31, 2020). 137 Endline school survey conducted by the evaluation team. 138 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 139 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 140 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 141 WFP. 2021. MGD Semi-Annual Report (November 2020 – April 2021). Advance excerpt provided by WFP. Date | Report Number 30 schools. 142CVA training equips SGAC members to discuss issues related to school meal preparation and distribution, WASH, literacy, parent contributions to school feeding, and student and teacher attendance. SGACs then develop action plans to address challenges and to guide advocacy efforts for school improvements. Regular school monitoring exercises revealed the growing involvement of parents in school management through the use of school scorecards to monitor progress.143 The school endline survey indicates that SGACs received training on nutrition, school feeding, school management and governance, health, school infrastructure, and school gardens (Annex 13, Table 109). 126.In some schools, parents have contributed to replacing and repairing handwashing stations, purchasing new kitchen utensils, and providing manure for school gardens.144Parent contributions are also a critical input to pay cooks’ salaries, and to purchase produce and firewood. Each school administration and SGAC determine the amount of the parent contribution; this ranges from 300 Rwf (US$ 0.32) to 1,500 Rwf (US$ 1.62) per student per term, though in some schools, parents may instead make in-kind contributions of firewood or volunteer labor in the kitchen or garden. The parent contribution fund is managed by either a teacher, the SGAC or the school administrator and 98 percent of schools record this amount in a financial record book.145 The MTE report indicated that some parents had concerns with how these funds were managed, i.e., the risk of funds disappearing or being used for other purposes. Interviews at endline show that these concerns were mostly addressed through discussion between parents, community leaders and school staff, which was facilitated by the programme. 127.FGDs with SGACs indicate that parents’ contributions are slowly increasing but are estimated to vary from 40 to 80 percent of parents; the ability to contribute is more challenging in certain schools and for very poor families. Students told the ET that, in some cases, students do not receive a meal when their parents do not contribute. The country office reports that this practice is highly discouraged by WFP, parents and MINEDUC but some schools continue to practice it and not report it.146 Key EQ 2: Has the use of health and dietary practices increased? MGD 2.1: Improved Knowledge of Health and Hygiene Practices 128.EGRA findings indicate a small increase in knowledge of health and hygiene practices in all project districts (53 percent at endline vs 49 percent at midterm) except Karongi where the midterm value (48 percent) decreased to 41 percent at endline (Annex 13, Table 110). Despite this increase, the target for students who can identify at least three key health and hygiene practices (80 percent) is not met (Table 9 and Annex 6, Table 64). 129.The EGRA findings show an improvement in students’ health and hygiene practices at endline, including handwashing with soap after toilet and before eating, brushing teeth, eating a balanced diet, and avoiding open defecation. However, the findings also indicate a decrease in other personal hygiene (e.g., bathing) from 91 percent at midterm to 81 percent at endline (Annex 13, Table 111). District officials and the majority of teachers interviewed told the ET that they noticed an improvement in handwashing practices. The ET noted that students are continually reminded by teachers to practice good hygiene, so may have forgotten some practices during the long absence from school. However, students were observed washing their hands before entering class.147 142 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 143 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 144 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 145 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 146 Per information provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. 147 The ET observed permanent handwashing stations at the school entrances where all students wash their hands. They also observed “step and wash” technology (kandagira ukarabe) in classrooms (though some did not have water) and by the latrines, though some students did not adopt the practice. Date | Report Number 31 Table 9: Student health and hygiene knowledge, baseline, midterm and endline Indicator Target Baseline Midterm n Endline n Percent of students who can identify at least 3 key health and hygiene practices 80.0 n/a 49.2 441 52.9 435 Male students 80.0 48.0 48.6 220 44.7 215 Female students 80.0 47.0 49.8 221 60.9 220 Source: Baseline, midterm and endline EGRA surveys No significant difference between male and female students at midterm is observed at a p<0.05 level or lower. 130.Training for parents on good health and hygiene practices focused on the importance of improved hygiene at home, including personal hygiene, safe drinking water, handwashing, skin diseases, safe food handling, latrine sanitation, and preventing acute respiratory infections (ARI).148 The target was exceeded across all project years because the project decided to train all parents, as children are more likely to contract hygiene-related diseases at home (Annex 6, Table 66). 131.The project also organized health and hygiene campaigns in all 104 schools. Targets were met in 2017-2018 and almost met in 2019-2021, with achievement ranging from 91 – 98 percent (Annex 6, Table 65). Interviews conducted by World Vision showed increased child awareness of the importance of good health and hygiene practices. At the beginning of the project, World Vision provided manufactured handwashing stations to each school but over time these ceased to function. After the MTE, World Vision and WFP designed a handwashing station model for schools that can be made from readily available, low-cost materials. 149 During COVID-19, permanent improved handwashing stations were added in 38 schools. Each school now has at least one handwashing station outside of each classroom, latrine, and kitchen. To ensure that students received health and hygiene messages while schools were closed, World Vision’s Reading Club Facilitators distributed booklets containing good hygiene practices messages to children when they were at home during lockdown.150 Interviews show that COVID prevention messaging from the Government and other sources further reinforced knowledge on handwashing practices. MGD 2.2: Increased Knowledge of Safe Food Prep and Storage Practices 132.Cooks and storekeepers in the four districts received training on food handling, meals, safe food preparation, storage and WASH,151 with bi-annual refresher trainings as cooks and storekeepers frequently change in the programme schools.152 The number of trained cooks and storekeepers thus varies across project years; the number trained was highest in 2018 (469 trained; 90 percent of target) and 2019 (459 trained; 88 percent of target). While training numbers dipped to a low of 164 (32 percent of target) in 2020 due to the pandemic-related suspension of group meetings and trainings, they were back on track by April 2021, with 309 cooks and storekeepers trained against a FY target of 520, representing 59 percent achievement at midyear (Annex 6, Table 68). 133.The endline school survey shows a large decrease from baseline to endline in cooks/storekeepers’ ability to identify at least three safe preparation and storage practices, from 92 percent at baseline to 85 percent to 40 percent at midline and endline, respectively (final target 95 percent) (Annex 13, Table 108). This is attributed to trained cooks leaving the schools, schools with no cooks, and to newly hired cooks who took part in the endline assessment but have limited skills or training from the project, along with a lack of practice for cooks during school closure. The ET did observe that cooks practiced good hygiene both 148 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 149 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 150 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020); draft review comment from World Vision 151 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). Information also confirmed during key informant interviews with cooks and storekeepers. 152 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). Date | Report Number 32 personally and in kitchens, including handwashing, though the cooks were not always able to identify the good practices. MGD 2.3: Increased Knowledge of Nutrition 134.The indicator used to measure nutrition knowledge relates to training: number of individuals trained in child health and nutrition as a result of USDA assistance. The number of individuals trained between 2017-2021 was 25,590 against an endline target of 208 (Annex 6, Table 69). This does not necessarily represent vast overachievement – rather, it is a measurement issue explained by a change in training model, whereby the indicator target was based on the original model. Initially, training was to be conducted using a training-of￾trainers approach, which would be consistent with a target of 208 individuals, as these individuals would in turn train many (hundreds or thousands) others. Given GHI’s solid expertise in direct capacity strengthening in health and sanitation, the programme approach shifted from training trainers to training beneficiaries directly; in a decision agreed with WFP, World Vision also changed its training model to include all parents. While it is possible or even likely that the programme’s training efforts achieved the intended scope, this indicator no longer measures this accurately. MGD 2.4: Increased Access to Clean Water and Sanitation Services 135.The project met or exceeded the target number of schools with improved sanitation facilities in all project years (Annex 6, Table 72). The project built or reconstructed 34 latrines, exceeding the target. World Vision reallocated funds to meet an increased need for latrines due to higher student numbers.153,154 136.The target for all 104 programme schools to have an improved water source and sanitation facilities was reached in 2018 (Annex 6, Table 70). The endline school survey confirmed that all schools in the sample use piped water (Annex 13, Table 112). Building/rehabilitating water collection systems included the provision of rainwater catchment systems but as rainwater supplies were inconsistent, in 2019 the National School Feeding Steering Committee approved connecting schools to piped water sources.155 WFP reallocated funds for this purpose and in late 2019, eight additional schools were connected to piped water156 and in 2020, 12 additional schools were connected, bringing to 20 the total number of schools connected to water by the McGovern-Dole Programme.157 School staff told the ET that some schools with piped water receive only intermittent supply, and some schools struggle to pay water bills. 137.With COVID-19, the challenge of adequate handwashing facilities in schools nationwide was widely discussed. MINEDUC issued a requirement to schools to construct permanent handwashing stations. In response, WFP and RBC constructed 10 permanent handwashing stations and World Vision constructed 28 stations in select project schools.158 Of the two joint designs by WFP, UNICEF, MINEDUC and World Vision, the Government has adopted one nationwide for construction in over 1,300 schools.159 ET interviews with district officials indicate there is still a need for WASH material, especially soap. In addition, schools with new permanent structures must contend with much higher water bills. MGD 2.5: Increased Access to Preventative Health Interventions 138.Deworming is organized twice a year by the Rwanda Biomedical Centre (RBC) as part of the national Integrated Health Week (IHW). WFP has had a partnership with RBC since 2016 and supports the delivery of the annual campaign.160 Yearly targets for the number of students receiving deworming treatment were 153 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 154 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 155 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). 156 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 157 2020 data provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. 158 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 159 Additional information on handwashing facilities construction provided by WFP Rwanda country office in Evaluation Reference Group review of draft report.. 160 WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). Date | Report Number 33 met in 2017-2018 and achieved 94 percent or higher in 2019-2021 (Annex 6, Table 77. In 2020, RBC transitioned the IHW campaign to a decentralized model with deworming conducted at school and community levels. World Vision also supported RBC in the development of tools and materials for the prevention of Neglected Tropical Diseases and parasitic diseases for use in RBC’s new campaign model.161 MGD 2.6: Increased Access to Requisite Food Prep and Storage Tools and Equipment 139.As reported in the discussion on improved school infrastructure (paragraph 121) all 104 programme schools have improved food preparation and storage equipment. The kitchen facilities constructed in Rutsiro and Karongi districts in 2017-2018 were rehabilitated to reduce smoke production. Due to COVID￾19 lockdowns in Rwanda, there were some delays in planned construction and handover, but these were completed by programme end.162 Key EQ 3: What is the level of community-level involvement and participation in decision￾making in school governance mechanisms? 140.Communities’ and parents’ engagement with schools has strengthened since the MTE. Parents are more engaged with schools as they are better informed, are familiar with the reading club activities, and value the school meals. District officials confirmed that parents have a greater recognition of their role in their children’s education and an increased sense of responsibility regarding their contribution to the programme. Several factors contribute to this expanded involvement, including greater outreach to parents through parent and school committee meetings, capacity strengthening of teachers, volunteers, and cooks, and the involvement of cooperatives. GHI’s outreach with school gardens, seeds, and nutrition education has also generated more community engagement with the schools, and World Vision aired radio programmes during COVID-19 to raise awareness on the importance of education. Increased awareness generated by the reading clubs and strengthened community literacy activities through better alignment with school curricula, have been both drivers and results of overall parent engagement. The success of these efforts is borne out in endline EGRA indicators, as well: the percent of parents in target communities who can name at least three benefits of primary education rose from 69.8 percent at midline to 74.3 percent in FY2021 (Annex 13, Table 108 and Annex 6, Table 55). 141.Parents support the school by contributing to school decision-making, school management, financial and in-kind contributions to the school feeding programme, direct purchase of vegetables from school gardens, and supporting assistant teachers with childcare. The SGAC mobilizes and sensitizes parents, especially on contributions, and represent parents in decisions relating to education and school development. Teachers and school administrators stated that more parents are attending school meetings. Interviews with parents and SGAC members indicate that school committees are working with parents to facilitate collaboration between the parents and the school and are working with local leaders to reduce dropouts. The EGRA/school survey team noted that the FGDs with parents and school committees were gender-balanced in terms of participation by men and women, though women are mainly engaged in the school management committees. 142.Teachers and school administrators stated that parent contributions have improved since the MTE, and now, according to KIIs and FGDs with parents and SGAC members, 40-80 percent of parents are providing cash or in-kind support to school meals. However, KIs in many schools said that parent contributions remain a significant challenge and often are not sufficient to enable schools to pay cooks and to buy firewood and vegetables. This is attributed to parents’ view that school meals are the responsibility of the Government, the effect of COVID-19 on incomes, and poverty. Schools continue to sensitize parents to their role in the programme and to the reasons for contributing. 143.Job creation and training for cooks and cleaners is valued by the community. During the MTE the ET noted that few cooks at the schools were women. The number of female cooks is slowly increasing, and in 2019, 161 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). 162 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). Date | Report Number 34 73 percent of WFP-assisted schools have at least one female cook.163 Some barriers remain to women’s employment as cooks, including the attitude of many men that women are not strong enough to stir the large pots of food. Another barrier in the western region is the need for cooks to walk to school before sunrise, which is not safe for women, in order to prepare porridge in the morning. Key EQ 4: What are the key institutions and governance structures required to effectively deliver, implement, and sustain school meal interventions? Increased Capacity of Government Institutions 144.The number of government staff trained and certified exceeded life-of-project targets (Annex 6, Table 82). At national level, training was given on managing programme implementation, linking programme partners and ministries, and coordination and oversight. At the district level, training focused on increasing ownership, and sustainability mechanisms including an emphasis on producing sufficient vegetables in the next programme phase. 164 Improved Policy and Regulatory Framework 145.The national School Feeding Policy and strategy, first written in 2016 with WFP support, were updated with WFP technical assistance and approved by MINEDUC in 2019. 165 While a new national nutrition policy was being formulated in 2020, WFP held meetings with stakeholders to ensure school feeding would be reflected in the new policy. Both policies are awaiting full approval by the Cabinet, which is projected for 2021. WFP has committed to support MINEDUC in drafting a financing strategy for the national scale up of school feeding in 2020/2021 (Annex 6, Table 85 and Table 89). 146.WFP supported MINEDUC to finalize the draft School Feeding Operational Guidelines in 2019 and 2020 and MINEDUC approved the guidelines in 2021. The guidelines provide comprehensive guidance for the implementation of school feeding including governance, reporting, food handling, preparation, nutrition, storage and safety, and procurement.166 WFP also created a summary of the guidelines in English and Kinyarwanda which has been distributed nationwide to all schools. These guidelines are to be used as the official government instructions on school feeding. The guidelines build upon WFP’s support to MINEDUC in developing menu modelling for the national school feeding programme using nutritious, locally sourced foods. The model menus influenced the national budget planning process and resulted in MINEDUC proposing a higher government budget for meal allocations. 167 It also provided the Government with a cost benchmark for a nutritious meal for the first time.168 This enables MINEDUC to compare its current subsidy for a nutritious meal, which covers about 40 percent of the cost, with the actual cost. 147.The project planned to hold six meetings per year with the former HGSF Steering Committee and Technical Committee.169 Sixty-four meetings were held between 2017 and 2021, indicating strong communication and coordination among committee members. WFP reports that quarterly meetings were held at district level, and yearly meetings at national level (Annex 6, Table 86). 148.To support more robust data collection and monitoring of school feeding programmes, WFP supported two government workshops in 2020 to incorporate improved indicators into MINEDUC’s management information system. WFP assisted MINEDUC to pilot the new school feeding module in 10 McGovern-Dole schools in 2020. The target for establishing government monitoring and evaluation systems has been met 163 WFP. 2019. Rwanda Annual Country Report. Country Strategic Plan 2019-2023. 164 Per communication with country office during review of M&E data. 165 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 166 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 167 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 168Per information provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. 169 Details on these committees are discussed in Section 2.1, paragraph 73. Date | Report Number 35 (Annex 6, Table 88); WFP trained all 107 schools in May 2021 to be users and a larger pilot is ongoing with national roll-out planned for September 2021.170 149.The McGovern-Dole Programme supports three staff at the national level, one in MINAGRI and two in MINEDUC, who provide coordination, management, supervision, monitoring and reporting of project activities at the national level using existing government structures and resources. 171 In August 2020, following continuous advocacy by WFP, MINEDUC established a four-person school feeding unit within the ministry. WFP supported the development of job profiles for the unit 172 (Annex 6, Table 89). 150.In each programme district, the McGovern-Dole Programme supports a district coordinator to monitor, coordinate and provide technical assistance to programme activities, and to advocate for the programme with district Government.173 District coordinators are key participants in WFP workshops on the development of operational guidelines and school nutrition indicators, and on the review of national training materials created by WFP and MINEDUC. The district coordinators are also important resources for administrators and schools implementing the government school feeding programme. 151.WFP, district coordinators and MINEDUC adapted existing global training guides to develop national guidelines for food quality and safety for all schools in Rwanda.174 Training materials will support implementation of the guidelines and include a video series to complement future in-person trainings.175 Increased Government Support 152.The Government has approved a large budget increase for the NSFP. Prior to the introduction of the NSFP, it was envisioned that public-private investments would be leveraged as a result of USDA assistance and that the Government would allocate part of its budget to the national programme. With the advent of the NSFP, the original indicator to attract financial support is no longer relevant;176 the target and actual values for this indicator are zero. (Annex 6, Table 92). Increased Engagement of Local Organizations and Community Groups 153.The objective to increase engagement of local organizations and community groups is reported under paragraphs 111-112 on partnerships with farmer groups to supply food to schools. WFP provided support and capacity building to 35 McGovern-Dole-supported cooperatives in southern and western provinces177 (Annex 6, Table 43). In addition, the target of 104 SGACs supported as a result of USDA assistance was 100 percent achieved in 2017 (Annex 6, Table 93). The number of public-private partnerships formed as a result of USDA assistance was achieved in 2020, with a similar result in 2021, with 35 partnerships formed (113 percent). In addition, community outreach efforts of World Vision resulted in 936 community volunteers leading weekly Reading Clubs throughout the project.178 170 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). Updated info provided by country office via email correspondence 2 August 2021. 171 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 172 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 173 WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). 174 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). 175 WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). 176 Note received by the ET from WFP as part of the review of outcome performance data (24 March 2021). 177 WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). 178 Per information provided by World Vision in Evaluation Reference Group review of draft report. Date | Report Number 36 Key Findings and Conclusions – Effectiveness 154.Over the life of the programme, the McGovern-Dole Programme consistently exceeded or nearly met its annual targets for the number of school-aged children receiving daily meals (see Annex 6, Table 36), despite the constraints imposed by the pandemic. To complement food assistance, the programme has supported activities in nutrition education, improved WASH, deworming, training of cooks and other areas that contribute to improved nutrition. 155.The McGovern-Dole Programme has been effective in significantly improving children’s reading skills. EGRA scores and interviews with school administrators and teachers confirmed the improvement in children’s reading capacities, with girls better performing than boys. Key informant and focus group responses indicate that community reading volunteers were the main reason for continued improvement while schools were closed due to COVID-19. 156.The programme provides equal access to benefits for girls and boys and has supported and influenced the Government’s policies on gender equality. There is scope for greater gender sensitivity in some aspects of the programme, and potentially for the scaled-up national programme. 2.3 EFFICIENCY 157.As noted throughout Section 2, programme targets were often exceeded to a considerable degree. Part of the McGovern-Dole Programme’s success in meeting (and exceeding) targets is that it is an intensive programme with layered interventions that focuses on a small number of schools. At the same time, interviews with programme staff indicate that some targets were also set too low at the beginning of the programme and were not sufficiently adjusted to account for changes to the implementation approach during the programme. Details for significant over- and under-achievement against indicator targets are provided as table notes in Annex 6. 158. In the second half of the programme, WFP and partners expanded the programme’s value for money through their integrated support and input to national school feeding policies and strategies. This was paired successfully with learning from implementation at district, school, and community level that could be applied to context-appropriate scaling. The programme, especially since the midterm, has reached beyond its four target districts to support the development of national school feeding policy and guidelines that will apply to the entire national school system. This transition from a focus on implementation to a broader role in strengthening national capacity effectively broadened the impact of the McGovern-Dole Programme. Significantly, WFP staff acknowledged that the pandemic restrictions allowed them more time to focus on discussions at the national level with government partners and on technical support for a national school feeding programme. 159.The reorientation of specific programme components indicates an increase in the efficient use of resources, particularly in the second half of the programme. For instance, the expanded implementation of the school garden component in particular addresses questions raised in the MTE about this activity’s efficient use of resources; the inclusion of community outreach and nutrition information in the school garden activity indicates that its efficiency has improved. In another example, the project literacy component was adapted to provide better complementarity to Soma Umenya, another literacy project that was modelled after the McGovern-Dole Programme and had more comprehensive coverage. The active engagement of World Vision and WFP in the education sector helped to assure understanding and alignment with existing national initiatives. 160.Programme and partner integration have improved since the midterm, increasing the efficiency of operations. Internally, WFP has instituted greater coordination among the HGSF, logistics, nutrition, and smallholder agricultural market support units in the country office since the midterm, creating an internal working group to bring unit heads together on strategic issues. Partners are now included in strategic discussions with the Government on school feeding, whereas at midterm, programme partners were Date | Report Number 37 mainly engaged in implementation. Monitoring and evaluation coordination has also improved through the involvement of the internal working group. 161.The introduction of the CSP and SO managers has brought more experienced staff into programme strategy and management. This, along with hiring more staff after the midterm, has improved management and operational efficiency. Staff interviews indicated some concerns that the introduction of additional staffing levels poses a risk to the direct engagement of senior leadership with the programme. In addition, as a result of intermediary staffing levels, senior leadership may receive fewer details and less key information to make informed decisions, and less input from staff who are most closely linked to understanding the work. The specific concern here is that this may dilute the level of senior leadership attention to the programme, which was a main factor in programme strengthening after the MTE. 162.WFP has worked with MINEDUC to improve the efficient use of resources by developing a low-cost design for school kitchens, and a design for handwashing stations that can be constructed from local materials. These two designs have fed into the national construction of more than 2,600 kitchens in government schools and more than 1,300 permanent handwashing stations.179 School administrators are aware that they are responsible for kitchen maintenance, though whether they will have sufficient budget to do so is unclear. The programme has also financed piped water supply to many schools to replace unreliable water catchment systems, but some schools report intermittent supply from piped systems, indicating that reliable water supply continues to be a challenge despite this investment. Key Findings and Conclusions – Efficiency 163.Program targets were exceeded due to the programme’s high-intensity programming across a small number of schools. However, in some cases targets were also set and kept too low. 164.WFP and partners expanded the programme’s value-for-money through their integrated support and input to national school feeding policies and strategies. Pandemic restrictions on direct implementation activities provided further opportunity for strategic engagement with the Government. WFP and partners have demonstrated efficiency through the realization of programme results in an economic and timely way, and through a high-level of adaptive management that resulted in efficiency improvements, especially in the second half of the programme, i.e., improvements to the school garden and construction activities. 165.Improvements to the internal operating model of WFP and strengthening of partnership arrangements increased efficiency across the programme as a whole. 2.4 IMPACT AND SUSTAINABILITY 166.The sustainable impact of the McGovern-Dole Programme is evident in its contributions to national policy and strategies, and in the Government’s decision to institutionalize school feeding management functions. These institutionalized functions include the establishment of a National School Feeding Steering Committee, a Technical Working Group, and a School Feeding unit within MINEDUC. Both are now part of the government structure, with their own budgets. Along with budgetary support, the strong ownership and leadership of the school feeding programme by senior government officials is an important contributor in sustainability. WFP has played a significant role in establishing these structures and government ownership, in part through WFP’s ability to demonstrate results and use lessons from the McGovern-Dole Programme to support the government’s own education priorities. WFP is a member of the National School Feeding Steering Committee, which enables it to continue to contribute to the design and implementation of a sustainable national school feeding programme that is appropriate to the capacities and budget of the Government. The task of capacity strengthening of the new institutions and their staff – from the national to the district level – is a longer-term challenge, particularly in the areas of procurement of nutritious local foods and addressing, maintaining water infrastructure, and safe meal preparation in 179 Per information provided by WFP Rwanda country office in Evaluation Reference Group review of draft report. Date | Report Number 38 schools. National capacity building for sustainability extends beyond partner ministries; in Phase II WFP will work with national academic and training institutions to train a cadre of managers in the future with the skills to implement the programme. WFP will continue to provide technical support to the Government in these and other areas in Phase II. 167.An important contribution to the sustainability of school feeding was WFP’s work with MINEDUC to develop model meal menus for the NSFP.180 The model menus provide a seasonally based, cost-effective school meal using locally available foods. The meal is designed to meet 50 – 70 percent of a child’s micronutrient requirements and one-third of daily energy, protein, and fat requirements. 181 Menu modeling also addressed the additional nutritional requirements of adolescent girls. This was well received by the Government, which saw it as a relevant technical solution which contributed to understanding of how school feeding could address education, nutrition, and agricultural needs. 168.The Government has also shown its commitment to a national school feeding programme by allocating 10 percent of its education budget to school feeding, with additional money set aside for kitchens and other infrastructure. However, the current government budget of US $0.06 per day provides only 38 percent of the cost of a model meal. The model meal aims for a base cost of US $0.14 per day. The government plans to provide a cash transfer to cover US $0.10 of the meal and will rely on a parent contribution of US $0.04 per child per day to cover the full cost. The model menus have prompted MINEDUC to increase its budget request for school meals. WFP also continues to advocate with the Government for an increased allocation. 169.WFP continues to learn from the McGovern-Dole experience and adapt its approach. The programme currently relies in part on imported food, an inherently unsustainable strategy, but sources beans and maize for school meals locally, in alignment with government policy to support local production. To support a sustainable school feeding programme, WFP developed a procurement strategy for the Government based on local purchase that would eliminate reliance on imported food while supporting local and regional market development. The strategy to provide more nutritious meals aims to promote greater diversity in crop production while providing a reliable school-based market for small farmers and contributing to greater resilience among farmers. The Government was very supportive of a procurement strategy linked to smallholder farmers, which is in line with the Government’s designation of agricultural development as a national priority. Another contribution to sustainability in Phase II will be the integration of cash transfers for the purchase of food locally and will move to serve only hot meals in line with the Government's request.182 170.A key challenge to sustainability is the development of a national school feeding model that is appropriate to needs and resources. The McGovern-Dole Programme has been a key contributor to this process, with WFP and World Vision designing low-cost kitchen and handwashing facilities for schools, which have been adopted by the Government for all schools. While the Government does not have the budget to adopt the entire McGovern-Dole model, it has recognized the value of its integrated education, nutrition, and WASH components; for instance, on a national basis, it is encouraging school gardens and the training of cooks and storekeepers. Education officials acknowledge that there are still many challenges to implementing a national school feeding programme, and that poor areas will still need a subsidy and partners even after five years. 180 MGD Semi-Annual Report Apr-Sept 2020. 181 WFP. School Feeding in Rwanda. Menu Modelling and Meal Options. April 2020. 182 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Introduction and Strategic Analysis. Date | Report Number 39 Key Findings and Conclusions – Impact and Sustainability 171.After the MTE, the McGovern-Dole Programme expanded its impact from direct results in targeted schools to larger scale systems-level impact through targeted support for a national school feeding programme and central and decentralized levels. This support encompasses an effective combination of technical input to national policies and strategies paired with context-specific learning from implementation. 172.The sustainable impact of the McGovern-Dole Programme is evident in its substantial contributions to a national policy and strategies to scale up a national school feeding programme, which is now seeing increased government budget support, and in its contributions to the institutionalization of school feeding within government structures. 2.5 FACTORS AFFECTING THE RESULTS Internal factors 173.A key internal factor that contributed to the strong results achieved by the McGovern-Dole Programme since the MTE was the country office’s willingness and ability to act on the recommendations from that evaluation. At midpoint, WFP and the Government were building the systems needed for a sustainable school meals programme. The MTE noted the opportunity for the programme to more deliberately apply its experience to support and guide the government in developing the national school feeding programme. This required WFP to better organize itself internally into a credible partner, better present its results to Government and have a more structured approach to its government partnership and capacity strengthening to facilitate easier government engagement. Table 10 shows the headline MTE recommendations and the actions taken by the country office, which are discussed in greater detail below. Table 10: Midterm evaluation recommendations1 and actions Operational Recommendations Country Office Actions R1: Strengthen WFP management, role clarity and staff capacity for functions related to the McGovern-Dole programme [high priority] This recommendation was fully implemented and provided the foundation for the programme’s success in the second half of its implementation period. Country office management collaborated closely with the HGSF unit to resolve this recommendation within a 12-month period. As discussed in multiple conversations with the country office, WFP country office leadership, including the new Country Director, were strongly engaged in operations and strategy in the second half of the programme, which allowed for more adaptive management. Regular briefings with country office management helped ensure that management was kept abreast of new developments. The roles and responsibilities between the HGSF unit, technical units and senior leadership were clarified. Coordination and collaborative planning were increased, and informal day-to-day communication was regularized. Additional country office positions around capacity strengthening and strategic outcomes and increased technical capacity at HQ/RB level were leveraged to provide direct support to the programme. Key country office personnel with strategic skills sets were positioned to provide support to the HGSF programme as a whole, including for government advocacy and partnership functions – which was a specific emphasis of this recommendation. Field monitoring staff was increased so schools were visited monthly instead of only twice per semester. This provided more regular programme monitoring to maintain implementation quality and allowed more frequent reporting and review with government on challenges and follow-up actions. Relationships with partners were strengthened at both senior leadership and operational levels. Date | Report Number 40 Table 10: Midterm evaluation recommendations1 and actions Operational Recommendations Country Office Actions R2: Organize reflection meetings to inform knowledge management, advocacy, and strategic thinking. [high priority] This recommendation was largely implemented. An HGSF Internal Working Group was created specifically in response to this recommendation. It is chaired by the Deputy Country Director and includes all key head of units and other relevant colleagues. Senior country office leadership placed particular emphasis on reflection activities that considered a whole-of-programme approach to supporting government to develop a national school feeding programme. Such reflection was organized through regular semi-annual meetings, monthly meetings as well as incorporated into regular planning and performance monitoring sessions between the HGSF unit and country office senior leadership. Through these activities, the HGSF programme increasingly recognized the relevance and potential of its partnership and capacity strengthening modalities, and it matured to consider strategic decisions and results of equal importance to those at operational level. The caveat to this recommendation is that the country office did not institutionalize or properly resource a knowledge management function to capture, organize and document the process and results learning generated through its reflection activities. While this information was accessible to the ET through interviews, it was not documented for efficient and effective future use. R3: Contribute to the development of a school kitchen model that integrates primary and secondary school kitchen infrastructure and can still be supported by parent and community contributions. [medium priority] This recommendation was largely implemented. The programme shifted resources from COVID-19 affected activities to scale up and improve its infrastructure activities, including rehabilitation and building of kitchens. WFP and MINEDUC designed a low-cost kitchen model for the national school feeding programme. Integration of primary and secondary school kitchens was considered where relevant, and new kitchen infrastructure always took into account facilities across both primary and secondary grades if present in the same school. The issue of parent and community school engagement contributions was elevated to a strategic priority by country office leadership, with positive progress noted at endline and a clear line of sight to further improve on this issue. Strategic Recommendations Country Office Actions R4: Initiate a structured transition or continuation planning process with the Government [high priority] This recommendation was fully implemented, especially the recommendation to establish an intersectoral working group, led by the government with backstopping from WFP as necessary, that focuses on the school feeding and education nexus. In 2019, WFP supported the Government to establish a Technical Working Group on school feeding under the Education Sector Working Group, which is now the main coordination mechanism to guide McGovern-Dole Programme to government efforts. Overall, WFP action to support the Government to develop a national school feeding programme that builds on the McGovern-Dole results has become more needs-based, deliberate and structured. For Government, this has made the relationship with WFP and management of WFP support easier to manage and apply. 1 WFP Rwanda. 2019. WFP's USDA McGovern-Dole International Food for Education and Child Nutrition Programme’s Support in Rwanda 2016-2020: Evaluation Report, Midterm Evaluation. July. 174.The quality and commitment of McGovern-Dole Programme staff has played a key role in the success of the programme. Senior leadership capacity and engagement, noted as a key challenge at midterm, have improved and been a key driver of success in the second half of the programme. Across KIIs, WFP and partner senior leadership were consistently highlighted as internal and external champions of the McGovern-Dole Programme. This leadership and high-level support provided McGovern-Dole Programme Date | Report Number 41 staff with the confidence and organizational support to intensify programme activities where needed, establish and subsequently achieve higher standards of quality, and facilitate stronger partnerships with key government counterparts. 175.Leadership also played an important part in establishing a culture reset, especially in WFP, around the McGovern-Dole Programme. At the time of the midline evaluation, the strategic relevance of the McGovern￾Dole Programme was not fully understood and appreciated. At endline, it is very evident that the McGovern-Dole Programme is a flagship initiative for WFP. The government partnership and capacity strengthening elements of the programme now reflect what WFP in Rwanda wants to be known for. 176.Roles, skills and capabilities across McGovern-Dole Programme functions have been strengthened. WFP made changes that increased competency, credibility and trust with partners, specifically with the Government. These changes included ensuring high-quality staffing and a reduction in turnover at the country-office level, supported by the re-establishment of a dedicated school feeding unit at WFP HQ. Further changes were made at RB level to increase capacity and accessibility on core and supporting programme functions - including nutrition-sensitive approaches, school feeding modalities, gender and social inclusion, and supply chain management. 177.Coordination and communication have improved significantly since midterm. Within WFP, role clarity and regular coordination/communication channels between the McGovern-Dole Programme team and country office technical and functional support units, and with senior leadership, has improved. Day-to-day decision-making is facilitated through frequent communication on an as-needed basis, instead of being addressed at monthly or bi-monthly meetings as was the case at midterm. Such meetings are now instead organized as reflection points, which is more appropriate to their frequency. 178.Despite some staffing changes, relationships and coordination among WFP, World Vision and Garden Health International have improved decision-making processes since midterm. More regular formal and informal engagement at senior leadership level among these programme partners has improved functional collaboration, facilitated a whole-of-programme approach and enabled presentation of the consolidated McGovern-Dole Programme model to external stakeholders. The absence of such an approach was a particular gap noted at midterm. Operational collaboration among partners, which was already fit-for￾purpose at midterm, has been maintained. 179.The McGovern-Dole Programme service delivery model has been further tailored to the local context. Building on the solid technical capacity already in place at midline, McGovern-Dole Programme partners have continued to refine service delivery functions across all programme objectives to ensure that capacity strengthening, direct implementation by schools and communities, and outreach and advocacy activities are as efficient and effective as possible. The endline qualitative data indicate that satisfaction with McGovern-Dole Programme activities has, in general, increased or remained at a similar level as midterm. The lack of appropriate WASH infrastructure, noted as a critical deficit at midterm, has been addressed with available means. Reliable access to water is still a work in progress, but advances have been made in collaboration with Government. Provision of sanitation rooms for adolescent girls has contributed to regular attendance and retention of girls in school, according to project partners. Parent and community engagement has improved, mainly due to increased trust in the McGovern-Dole Programme and the schools associated with the programme: the programme has successfully managed to settle many of the concerns of parents, particularly related to the need for and the use of cash and in-kind contributions that schools were requesting. 180.Similarly, the programme has made meaningful progress in developing a procurement model that aligns with government needs and capacities. While this progress is notable, it is the McGovern-Dole Programme way-of-working that stands out most: the programme has successfully positioned itself as a partner that listens carefully to Government and addresses government priorities for McGovern-Dole Programme support. The programme has a positive reputation within Government and has become a strategic and technical partner of choice for current and future opportunities. Date | Report Number 42 181.Adaptive management in a dynamic operating environment. The programme managers successfully adapted interventions to remain relevant in a dynamic and complex environment. For example, programme managers adjusted the activities of implementing partners early in the programme. Especially notable was the positive manner in which managers responded comprehensively to the recommendations of the midterm review. Programme managers further demonstrated their agility through several highly effective responses to the challenges of COVID-19 by reallocating programme resources to provide take￾home rations during school closures, extending literacy services to students at home, continuing teacher trainings using a self-learning manual, extending outreach on school gardens and nutrition to communities around schools, and highlighting the importance of WASH infrastructure in schools. 182.The programme adapted its monitoring system to improve capture of programme results, which has added credibility to the McGovern-Dole Programme and to the value it has offered to Government. The programme has made significant investments in its performance monitoring function, which was evident all the way through to the collaboration with the ET at endline. For example, the McGovern-Dole Programme baseline study report (July 2016) focused on indicators that could be measured before project implementation. Some baseline indicators could not be measured because their definition is linked to the implementation of project activities. Following the baseline study, WFP reviewed and realigned the targets. In preparation for the midterm and endline evaluations, WFP put in place a strong programme monitoring component that collected and compiled data from each of the beneficiary schools and related activities on a regular basis. An improvement in the adjusted system is that all indicators, including policy-related indicators, require specific project records. 183.The ability of the programme to give accurate and reliable insight into its progress was cited by government and other sector stakeholders as a key strength, which positioned the McGovern-Dole Programme as a dependable stakeholder across its implementation domains. In addition, the programme has been credited for enhancing the utilization of its insights and performance data through strong visual representation of results, succinct and easy-to-use briefs on a range of technical topics, and readily available coaching by programme staff when additional explanation or information was required. 184.Interviews with programme staff indicated that a next step would be to complement the performance monitoring function with a learning function. The establishment of a learning agenda to proactively explore the validity of programme assumptions, and identify risks and opportunities to programme effectiveness, sustainability and scale-up was considered critical for continued success. Interviews indicate that this was especially relevant given WFP’s evolving role in Rwanda and the focus on government capacity strengthening, which is still a relatively new workstream for WFP. Interviews further indicated that key elements of a successful learning agenda are already in place, i.e., a learning culture, good coordination and communication within the programme and with stakeholders, and focus on evidence-based decision making. A formal learning agenda would tie all these elements together into an effective system for proper resourcing and utilization. 185.Partnership and relationship management are, appropriately, central to the McGovern-Dole Programme model. WFP and the programme partners have a large network of relationships across the many domains covered by this programme. These are critical to the strategic and operational success of the programme, and the programme has demonstrated that it takes these relationships seriously. The programme was cited as relationship-driven as opposed to seeking short-term wins, which was a major contributor to positive perceptions of the programme by government agencies, communities, and schools. 186.The programme adapted effectively to the risks and opportunities posed by COVID-19. The impacts of COVID-19, including the impacts of COVID response decisions has been discussed throughout this report. COVID-19 has severely disrupted the ability of the programme to conduct its school feeding and school￾based education activities, which posed a significant risk to the ability of the programme to achieve its intended outcome-level results. However, as the endline results have shown, these risks did not emerge - mainly due to the enabling factors already described in this section but also due to COVID-related opportunities taken by the programme that offset these risks. There are three key examples that demonstrate these opportunities. Date | Report Number 43 187.First, somewhat unexpectedly to the ET, the programme achieved strong literacy results despite schools being closed. Qualitative information attributes this result to the reallocation of resources from school￾based support to community literacy activities. Weaknesses in community-based literacy activities were corrected after the midterm evaluation, and the community literacy activities (along with school-based activities) were considered essential to achieving outcomes. COVID-19 gave the programme time and resources to address a programme weakness that provided critical support during school closures. 188.Second, with schools closed the programme was able to focus on urgent infrastructure deficits, which had also been noted at midterm. The programme shifted resources to the construction and rehabilitation of WASH facilities, kitchens, and storerooms. Where possible, this was timed to coincide with government efforts to increase reliable access to water. For many schools this resulted in a dramatic change in water availability for meal preparation and clean-up, and for hygiene and sanitation facilities. Some concerns remain over the capability of schools to maintain this new infrastructure, but this risk is already being reviewed by programme staff. 189.Third, with schools closed both government and McGovern-Dole Programme partners had the time and bandwidth to focus on ensuring the policy framework, partnership and capacity investment were aligned with the potential of the programme to contribute to a national school feeding programme in Rwanda. The time gained from school closures gave all parties the opportunity to focus on preparing good practices and lessons learned for consideration in policy and decision-making processes. This much-needed pause enabled to programme to not only demonstrate programmatic results but also to present insights and options to continue scaling at national level, including through its Phase 2 programme. External factors 190.The enabling environment for the McGovern-Dole Programme in Rwanda continues to be strong. Several of the enabling external factors noted at midterm remain valid at endline. Schools remain generally well organized with regular support from district officials and have been able to successfully apply the practices promoted by the project. The supportive policy environment, noted at midterm, has only strengthened since then. When combined with increasing budgetary readiness, leadership and capacity of Government, this is a key driver of programme success. Government capacity at national and subnational levels, which was already noted as strong at midterm, has significantly improved. Both levels of Government have a mature understanding of their capacity limitations and that further growth of the national school feeding programme must be facilitated in partnership with other stakeholders. The partnership between the McGovern-Dole Programme and Government can best be described as highly collaborative, even edging toward a co-created school feeding policy framework in Rwanda. 191.One external challenge is the change in the language of instruction in schools from Kinyarwanda to English in 2020. The full impact of this change is not yet evident as schools were closed for much of 2020. The change poses a challenge to primary school teachers – a 2018 study found that 38 percent of grade 1-3 teachers are able to teach in English - and to students who are still mastering their Kinyarwanda skills.183 The change potentially jeopardizes curriculum delivery and inclusion and may negatively impact learning outcomes, especially in rural areas where teachers are less likely to have English language skills. 184 The ET notes that World Vision has successfully piloted a toolkit through the McGovern-Dole Programme that supports teachers to build skills and techniques for teaching in English while building the English language skills of students. The toolkit will be used with lower primary teachers in Phase II. 192.COVID-19 and preventive measures such as lockdowns have pushed Rwanda into a recession and slowed economic growth.185 The gross domestic product fell by 3.4 percent in 2020 against a pre-pandemic growth 183 World Bank. Quality Basic Education for Human Capital Development in Rwanda. Project Information Document. 7 Nov 2018. 184 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Plan of Operations and Activities. 185 Government of Rwanda. Rwanda Economic Update. Protect and Promote Human Capital in a Post-COVID-19 World. January 2021, Edition no. 16. Date | Report Number 44 rate of 9.5 percent in 2019.186 As a result, poverty is expected to rise by 5.1 percentage points in 2021 with the greatest impact on women’s employment and in rural areas. While the government response included increased social protection measures, reduced revenues may impact the government’s ability to fully fund education activities at planned levels. The budget for the 2021-2022 academic year has been reduced, and the impact is expected to show up in the baseline for Phase II.187 193.The pandemic has had an impact on agricultural cooperatives, which are important to the sustainability of school feeding programmes as well as to the national food supply. One study of 90 cooperatives in eight African countries found that by mid-2020, most had lost income due to the market disruption caused by COVID-19.188 In Rwanda, the drop was due primarily to a decrease in consumer spending and decreases in the selling price for maize and other crops. 194.Other external factors noted as challenges at midterm have been addressed. As discussed under Internal Factors, communities have become more trusting of school management, which was a deliberate McGovern-Dole Programme focus since midterm. The Government has also taken steps to address the gap in coordination between national and subnational levels, including improving information flow and increasing participation in planning and progress meetings. Qualitative findings indicate that WFP has played a supporting role in closing this gap. Key Findings and Conclusions – Factors Affecting the Results 195.The McGovern-Dole Programme operating model has been strengthened since midterm. The programme investments in people, governance, process, service delivery model, and performance insights and data have contributed significantly to programme results at endline. 196.Improvements in performance monitoring have added credibility to the McGovern-Dole Programme. The ability of the programme to give accurate and reliable insight into its progress was cited by government and other sector stakeholders as a key strength, which positioned the McGovern-Dole Programme as a dependable stakeholder across its implementation domains. Related improvements include the strong visual representation of results, succinct and easy-to-use briefs on a range of technical topics, and readily available coaching by programme staff when additional explanation or information was required. 197.Partnership and relationship management are, appropriately, front and center to the McGovern￾Dole Programme model. The investment in and prioritization of long-term relationships over short￾term wins will continue to drive programme impact and sustainability. 198.There continues to be a strong enabling environment for the McGovern-Dole Programme in Rwanda. Several of the enabling external factors noted at midterm remain valid at endline, i.e., capacity of schools and Government. 199.Of great benefit to the programme were the strengthened capacity at county-office, RB and HQ levels. This included ensuring high-quality staffing and a reduction in turnover at the country-office level, the re-establishment of a dedicated school feeding unit at WFP HQ, and changes at RB level to increase capacity and accessibility on core and supporting a range of programme functions relevant to school feeding programme design and implementation. 200.Key external factors that affected programme performance at midterm have been addressed to the extent possible. Communities have become more engaged with schools and in the process have developed greater understanding of parents’ role in education, and greater trust in school management of parent contributions. Government has taken steps to address the gap in coordination between national and subnational levels. 186 National Institute of Statistics of Rwanda. Statistical article. 18 March 2021. 187 Personal communication, WFP Rwanda. July 2021. 188 USAID. Marketlinks. Resilient to Crises: How the Adaptive Nature of Cooperatives has Aided in Overcoming COVID-19- related Challenges. Venture 37. October 7, 2020. Date | Report Number 45 201.COVID-19 presented both risks and opportunities for the programme. Expected setbacks in literacy results due to school closure did not materialize. Instead, COVID-19 presented several opportunities that the McGovern-Dole Programme was able to leverage to amplify overall results. With schools closed the programme shifted resources to community literacy activities, the programme was able to focus on urgent infrastructure deficits, and McGovern-Dole Programme partners had the time and bandwidth to focus on ensuring the policy framework, partnership and capacity investment was aligned with the decision by Government to scale up to a national school feeding programme. 3. Conclusions and recommendations 3.1 CONCLUSIONS Relevance 202.The McGovern-Dole Programme is aligned with government priorities in education and child nutrition and is highly relevant to the needs of primary schools in some of the poorest and most food-insecure areas of Rwanda. School meals are relevant to national goals for improved child nutrition and increased student attendance, attention, and retention. Programme activities to improve access to water, sanitation facilities, nutrition education, and literacy training are highly relevant to children’s health and their ability to get the most value from an improved learning environment. 203.The programme was agile in adapting to changing circumstances and addressing complex needs to maintain a high level of relevance of programme activities. In the first half of the programme, partnership arrangements were effectively reorganized for both the education and agriculture components of the programme. In the second half, the programme proactively addressed the critical assumption of parent contributions required to support school feeding implementation. Programme staff worked closely with school and community stakeholders and parents to clarify expectations for parent support, which helped settle uncertainties for many parents. 204.The school closures and movement restrictions due to COVID-19 preventive measures presented both risks and opportunities to the McGovern-Dole Programme. The programme was able to adapt to school closures by shifting reading activities to the household level and engaging parents. In addition, it implemented needed infrastructure improvements, and school garden activities expanded to support home vegetable production and provide nutrition information. Programme partners and Government had more time to focus on policy, strategy, and capacity strengthening. 205.The training of teachers, head teachers, district education officials, parents and others through the programme is relevant to the need for improved teaching methods, better school management, and stronger community engagement. The resulting improvements in the quality of teaching and teacher attendance, and the greater engagement of local education officials, parents and other stakeholders offer lessons that are relevant to the broader primary education sector. Local education officials perceive these trainings as a strength of the McGovern-Dole Programme and credit the trainings with improving school management. 206.The programme is relevant to the needs of the Government of Rwanda. It was designed to align with the Government of Rwanda’s implicit and subsequently declared policy priority to scale up a national school feeding programme. As such, it has provided critical technical expertise and capacity strengthening to support the development and roll-out of the Government’s national school feeding programme. It further enables the Government to connect to a global school feeding network through WFP, which allows the Government to learn from others’ experience and to benchmark its progress against other countries. WFP’s work with Government to develop several low-cost options to kitchen construction, handwashing stations, nutritious meals using seasonal, locally available foods, and meals that meet the nutritional needs of Date | Report Number 46 adolescent girls, is highly relevant to helping the Government meet its goals for a national school feeding programme within its current budget limitations. 207.The development of a capacity strengthening approach and a procurement strategy linked to smallholder farmers who would supply the national school feeding programme is relevant to government priorities to buy food locally, support agricultural development and improve the livelihoods of small farmers. Capacity strengthening is provided to both women and men, including women-run cooperatives, consistent with national policy supporting gender equality in cooperative membership. While the McGovern-Dole Programme aligns with relevant gender policies pertaining to the programme components, the design and any subsequent adaptations did not include any specific approaches to address gender equality and women’s empowerment. 208.The McGovern-Dole Programme is aligned with WFP corporate objectives and relevant policies, in particular the WFP School Feeding Strategy 2020-2030. The programme also aligns with the United Nations commitments to SDG 2 (Zero Hunger), SDG 4 (Quality Education), and SDG 17 (Partnerships for Goals). Effectiveness 209.The McGovern-Dole Programme, with its integrated approach to an improved learning environment, has been effective in significantly improving children’s reading skills. Officials state that the programme has improved attendance and performance, and reduced dropouts. Officials cited several additional benefits of the programme, including the literacy clubs, improved school management due to training, additional training of teachers and cooks who often have limited experience, improved hygiene, the involvement of parents and SGACs, and the links to cooperatives. The programme strengthened its monitoring system, improving its ability to demonstrate the results of the model to the Government. 210.WFP has effectively leveraged the learning and experience from the McGovern-Dole Programme to provide a model for the Government’s National School Feeding Programme (NSFP). 189 WFP strengthened government capacity through training, staff secondments, establishing committees, and participation in international meetings such as the Global Child Nutrition Forums in 2018 and 2019. WFP established strong working relationships with MINEDUC on policy and implementation and with MINAGRI on capacity building of small farmers for local food production. WFP has provided technical assistance, tools, accountability frameworks, and south-south learning opportunities to build management skills for the NSFP. WFP supported the Government to establish an intersectoral working group on school feeding and education, and a National School Feeding TWG. The TWG has played a key role in operationalizing the school feeding policy, coordinating stakeholders and investments, and aligning the programme with long-term government strategy. Collaboration between WFP and MINEDUC has produced school feeding operational guidelines, a national nutrition policy, and a training video for storekeepers and cooks. Health and literacy results from the McGovern-Dole model have provided evidence for the Government to request more funding for school meals. WFP continues to support government efforts to implement the NSFP using a HGSF approach. WFP’s close collaboration with the Government on the McGovern-Dole Programme has increased government ownership of the programme and has resulted in school feeding being elevated to a national priority by the Government. 211.WFP’s technical assistance and work in capacity strengthening, and the Government of Rwanda’s active engagement in school feeding policies, strategies, and programmes, have supported efforts to mainstream lessons from the McGovern-Dole Programme. MINEDUC officials underscored the willingness of the Government to learn, and WFP’s willingness to listen and collaborate, as key to adapting the learning from the McGovern-Dole Programme into the NSFP. This flexibility has fostered collaboration on low-cost models for kitchens, WASH facilities and school meal menus that consider the Government’s budget constraints. Programme partners adapted their interventions to local needs, e.g., moving from school￾based to community literacy activities, and adopting a community outreach and engagement model for 189 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Date | Report Number 47 school gardens and nutrition. During school closures, the programme addressed outstanding infrastructure needs for WASH facilities, kitchens and storerooms in some schools. 212.Girls enjoy equal access to McGovern-Dole Programme benefits and exhibit better attendance and higher performance in primary school than boys. WFP’s work with the Government on nutritional analysis has helped expand potential options for school meals that can help meet the specific nutritional needs of adolescent girls. However, the support for GEWE has focused largely on equal enrolment and matriculation. Current efforts do not adequately enable girls’ continued academic achievement, job opportunities and leadership roles in society, nor do they address social norms that expect girls to prioritize household care-taking responsibilities over education among families, teachers, and students. Sensitization of teachers to gender attitudes is especially important, as only 30 percent of school staff are female.190 Efficiency 213.The programme has demonstrated efficiency in meeting or substantially exceeding most of its targets within its approved budget, and in a timely way. This is evidenced by performance indicator results as well as internal progress reports, EGRA survey results, and key informant interviews with local and national stakeholders. 214.Changes within WFP Rwanda have improved programme and partner integration since the midterm and increased the efficiency of operations. WFP has instituted greater coordination among McGovern-Dole Programme staff and the logistics, nutrition, and smallholder agricultural market support units in the Country Office since the midterm, creating an internal working group to bring unit heads together on strategic issues. 215.The programme adjusted specific interventions based on the midterm findings, which created greater efficiencies in the use of programme resources. The literacy component was integrated into the Government’s national literacy initiative, resulting in changed McGovern-Dole activities that ensured better efficiency and collaboration with another project that began operating in the same geographical area. This integration had the additional benefit of giving greater visibility to the McGovern-Dole Programme’s work in improving literacy. The expanded and intensified approach to the implementation of the school garden component since the midterm has justified the use of programme resources for this activity. 216.During the school closure and movement restrictions related to the COVID-19 pandemic, WFP and partners were able to adapt a few activities so that the programme continued to reach students and communities. Literacy club facilitators visited homes to loan books so children could continue reading activities and distributed health and hygiene booklets to households. Radio programmes were broadcast in each programme district to sensitize parents on the importance of education and literacy, and flyers were given to parents to maintain awareness of these important topics. GHI distributed 29,465 fruit and agroforestry trees to schools and 12,992 vegetable seedlings to parents during COVID-19 pandemic, and offered training on kitchen gardens. WFP organized two THR distributions for students. Impact and Sustainability 217.The McGovern-Dole Programme has had a degree of influence on the Government’s approach to the national school feeding programme. The demonstrated willingness of the Government to commit to and expand school feeding as a way to address food insecurity and help ensure the future development of the country is evidence of the Government’s intention to sustain a school feeding programme. While the Government does not have the resources to replicate the McGovern-Dole Programme model in all schools, it has adopted some of its approaches for the national programme, such as kitchen infrastructure and WASH components, and is working with WFP on increasing the capacity of smallholder farmers to provision the schools. 190 WFP.2019. Rwanda Annual Country Report. Country Strategic Plan 2019-2023. Date | Report Number 48 218.As of 2021, the national government has increased its budget commitment to school feeding eight-fold, investing in kitchens and other infrastructure in over 2,600 schools. Despite increased budget allocation, the Government cannot yet support the entire cost of a national school meal programme. To address budget constraints, WFP has developed model menus for the national school feeding programme that will provide a nutritious meal based on low-cost, seasonal, locally available foods, at a cost of 148 Rwf (US$ 0.15) for a base meal.191 Such practical solutions also support the government priority to reduce malnutrition among children. 219.WFP has worked with MINEDUC on sustainable solutions to the budgetary and logistic challenges of a national programme by modeling low-cost school meal menus that use locally available, seasonal foods, constructing local menus to address the specific nutritional needs of adolescent girls, and collaborating on models for low-cost kitchens and handwashing stations that can be constructed from locally available materials. WFP’s capacity strengthening of smallholder farmer cooperatives, including women-run cooperatives, to eventually supply school meals programmes will be a significant contribution to the sustainability of the national programme. 220.The ET notes that the Phase II proposal includes a plan for WFP and national-level academic and training institutions to jointly develop and implement training in aspects of NSFP management and include to NSFP skills into curricula.192 The institutionalization of training in NSFP management skills is an important step in ensuring there will be a broad, sustainable base of management professionals who can implement the programme over the long term. WFP has also drawn on Phase I experience and recognized that capacity at district, cell, school and community levels will need to be strengthened during Phase II to manage the greatly expanded NSFP. 221.As the McGovern-Dole Programme phases out of current schools and begins operations in new schools, sustainability in programme schools will vary between sites. The parent contribution remains a challenge to programme sustainability. In cases of families with multiple school-age children, schools will face problems paying for cooks, firewood and vegetables as poor parents will not be able to provide contributions for all their children. The programme schools know that they are responsible for maintenance of their new infrastructure, but several school heads expressed uncertainty about how they would pay for maintenance. District education officials are very supportive of the McGovern-Dole Programme achievements, which will help encourage the continued use of methods and approaches introduced to school staff by the program; nevertheless, funding will continue to be a challenge. Impact and Key Lessons Learned 222.The McGovern-Dole Programme has provided technical expertise and capacity strengthening critical to the development of the Government’s national school feeding programme. 223.The McGovern-Dole Programme model has influenced the Government to adopt some of its approaches in the National School Feeding Programme, specifically kitchen infrastructure and WASH facilities. The McGovern-Dole Programme has helped the Government to create sustainable solutions to implementation challenges by developing low-cost kitchen models and handwashing stations, and by modeling low-cost school meal menus based on local foods. 224.Adaptive management and a proactive approach to learning can yield significant performance improvements, as evidenced by the changes made after the MTE. After the MTE, the country office strengthened internal management, added staff capacity and clarified roles and responsibilities between the HGSF unit, technical units and senior leadership. Coordination, communication, and planning among technical units was formalized. This adaptive management approach was evident in the programmatic responses to COVID-19 challenges. 225.Leadership and organizational culture are critical for effective adaptive management. The country office demonstrated these qualities by acting on the MTE recommendations. This included 191 WFP Rwanda. School Feeding Newsletter. April-September 2020. 192 WFP. 2021. WFP Rwanda Phase II proposal for USDA McGovern-Dole Grant for WFP Home-Grown School Feeding Programme in Rwanda. Date | Report Number 49 emphasizing learning and knowledge management and strengthening strategic and technical input to the Government to aid with the transition process. 226.Effective capacity strengthening and authentic government partnership need to be based on credibility, relationships, and trust. Along with better internal programme management, the programme improved monitoring and reporting of results to the Government to better demonstrate programme achievements. The country office intentionally pursued a more structured approach to its engagement with the Government on developing the National School Feeding Programme. The Government’s appreciation of WFP acting as a facilitator rather than a driver of change has engendered a stronger partnership. 227.The original timeframe of the programme was too short to achieve the expected result of full transition into a national school feeding programme in a five-year period. Future designs should use a phased approach from the outset, similar to how Phase 2 was designed to build on Phase 1, with markers or benchmarks that identify readiness for the next phase to begin. Date | Report Number 50 3.2 RECOMMENDATIONS 228.The recommendations are presented by priority (1 = highest priority) within strategic and operational categories, with suggested implementation timelines and responsible parties. The strategic recommendations draw on insights from the Phase I experience. The operational recommendations pull through those insights into practical options for Phase II. Ultimately, WFP and government capacity to undertake the work required to move forward and assist the Government in its scale-up of the national school feeding programme for primary and pre-primary schools will determine the priority and sequencing of actions. # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when Str￾1 Recommendation 1: Consolidate WFP internal staff capacity to be enablers rather than implementors. WFP is assuming a new role globally in strengthening the capacity of government institutions to carry out many activities previously implemented by WFP. Many WFP Rwanda staff are skilled in operations whereas WFP’s new role requires staff to work at a policy and strategy level and to manage government relations. WFP Rwanda has meaningfully grown staff capability around capacity-strengthening modalities, partnership facilitation and relationship management for school feeding. These skills should be institutionalized and amplified across its broader programme of work. WFP Rwanda management has a strategic opportunity to formalize these capability requirements into its human resources policies and processes, and to consolidate current experience into internal training/induction models. It should contribute this experience to WFP regionally and globally, where many country offices are on similar journeys in terms of changing the nature of WFP’s impact through the CSPs. Strategic WFP Rwanda country office RBN, HQ High 2021-22 Str￾2 Recommendation 2: Consolidate and better document the country office’s capacity-strengthening strategy, approach and priorities for application across the CSP as a whole, building on the McGovern-Dole experience. It is essential that WFP create an enabling environment within the CSP to continue to evolve and grow its capacity-strengthening model in partnership with the Government and other Strategic WFP Rwanda country office RBN, HQ with input from government counterparts Medium 2021-22 Date | Report Number 51 # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when stakeholders. Consolidated success across the CSP will also strengthen the country office’s resource mobilization strategy for continuity and scale. Similar to Recommendation 1, this recommendation focuses on keeping the momentum of WFP’s current credibility and role as a key government partner in Rwanda. While WFP and the McGovern-Dole Programme have greatly contributed to strategic thinking and planning on a national school feeding programme, roll-out of this programme at scale will be a large undertaking that will require an expanded government partnership over the next several years. Capacity-strengthening results at the CSP level will legitimize and publicly demonstrate WFP’s capacity as a key partner for the Government of Rwanda. Str￾3 Recommendation 3: Decide on WFP’s approach to supporting further development of a national procurement strategy so this can be resourced, planned, and efficiently executed. WFP has provided essential strategic and practical support to the Government’s consideration of its procurement model options. To remain an effective government partner, WFP needs to develop and present a clear understanding of existing systems, as well as understand the Government’s specific expectations for WFP support so this can be effectively resourced and planned. Specifically, WFP needs to know how to position itself between a leadership role in presenting options and facilitating decisions, and a supporting technical role, while the Government works this through. Across all options, WFP will need to have capacity and resources in place to engage with multiple ministries and local government over an extended period. Such engagement will necessarily deal with policy development, public financing, consultation and co-creation processes, pilot initiatives, and models for scaling. In addition, any WFP approach will need to develop a position within the debate on imported versus locally sourced food commodities, which includes donor parameters for local purchase models. Strategic WFP Rwanda country office RBN and HQ In collaboration with programme partners, especially government counterparts Medium 2021-2023 Date | Report Number 52 # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when Str￾4 Recommendation 4: Establish a learning agenda at country￾office level to complement performance monitoring, test assumptions and document lessons from school feeding interventions. The McGovern-Dole Programme has built a strong performance monitoring system and contributed to strengthened M&E at country-office level. While the system adequately tracks output- and outcome-level results, it does not capture critical learning regarding the underlying assumptions for change and determinants of success to date. This documentation is not readily available for the country office to draw on for further strategic and programme design decisions. Learning and reflection processes require dedicated systems, processes and resources if they are to be utilized, especially for rapidly evolving programmes like the McGovern-Dole Programme, which require real-time information for adaptive management. It is especially important for learning around gender equality in education, specifically learning that focusses on what approaches work to address risks and opportunities for change. The foundations of the NSFP and WFP’s role in supporting its continued evolution have been established. However, much remains to be learned around issues like scaling, equity and access to programme benefits; targeting across the nexus of schools, community, and local government needs and capacities; purpose and boundaries of procurement models; alignment with related initiatives; resourcing; and sustainability of Phase 1 results. While WFP Rwanda is contributing to USDA’s school feeding learning agenda through special studies, these issues also need to be tracked, documented, and made fit-for-purpose for strategic and operational decision-making by a range of audiences. Programme partners have contributed much to learning and should be included in developing and carrying out the learning agenda. Finally, as noted under Recommendation 1, WFP Rwanda has a real contribution to make to the broader Strategic WFP Rwanda country office With input from RBN, HQ and programme partners Medium 2022 Date | Report Number 53 # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when WFP system. To fully realize this potential, process documentation is critical. Op￾1 Recommendation 1: Retain and continue to improve the effective McGovern-Dole Programme operating model developed after the Phase 1 midterm. An operating model framework considers the foundational elements that underpin strategic and implementation results. It commonly includes the following elements: people, governance, process, service delivery model, technology, and performance insights and data. It is especially helpful for organizations and programmes in transition and would be a useful framework for WFP to track performance of core organizational components as it continues to evolve its role. Operational WFP Rwanda country office With input from RBN or HQ, if required by country office High by Phase II midpoint* Op￾2 Recommendation 2: Ensure appropriate balance between capacity strengthening and direct implementation. A key factor that has enabled the McGovern-Dole Programme to make such a positive contribution to the development of the NSFP is its technical credibility and grounded experience. It will be important for WFP to ensure its facilitation and technical support to strengthening government policies and processes remains credible by maintaining a high-quality school feeding operation that demonstrates capacity to advise the Government on these issues. Some of this can come from WFP’s global experience and by connecting the Government to global school feeding networks, but a critical mass of implementation across key components of school feeding in Rwanda will need to remain. Operational WFP Rwanda country office With input from RBN or HQ Medium by Phase II midpoint* Op￾3 Recommendation 3: Maintain support to Phase I McGovern-Dole Programme schools, communities, and cooperatives. The programme came to its full fruition after its midpoint. Two years of comprehensive implementation that built on the foundations set in the earlier part of the programme have yielded good results, but this is insufficient Operational WFP Rwanda country office With input from programme partners High by Phase II midpoint* Date | Report Number 54 # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when to ingrain the sustainability of the behavior and systems changes currently observed. The Phase II design includes all Phase I schools until the second half of Phase II. It will be important that WFP continue to draw lessons, monitor, and organize support directly and indirectly (e.g., through government and local partners) to Phase 1 project sites during the second half of Phase II. Concrete examples are supporting maintenance systems for the multitude of school infrastructure built in the last year of the programme, and continuing to evolve the discussions with parents and representatives from schools, communities, and local government on expectations for parent contributions. Op￾4 Recommendation 4: Build adaptive management and agility into the Phase II design and implementation plan. A key strength of Phase I was its ability to make critical decisions efficiently and at the right time. Given that Phase II will focus more on the practical realities of how the NSFP will run, it is safe to assume that at minimum, a similar level of agility and funding flexibility will be required to address emerging risks and opportunities. This can be facilitated by building in regular reflection points that focus on progress toward outcomes in addition to routine performance monitoring, and maintaining a leadership culture that enables bold management, operational and funding decisions when necessary. Operational WFP Rwanda country office With input from programme partners High by Phase II midpoint* Op￾5 Recommendation 5: Establish contingency planning at school and community level to prepare for and respond to shocks and stresses like COVID-19. WFP has effective contingency planning policies and processes in place that allowed it to quickly pivot the programme during the COVID-19 pandemic, such as distributing THR to students. The benefits of contingency plans and systems are clear. There is an opportunity now to embed contingency planning more directly into the local systems that WFP facilitates and Operational WFP Rwanda country office With input from programme partners High by Phase II midpoint* Date | Report Number 55 # Recommendation Recommendation grouping Responsibility Other contributing entities Priority: High/medium By when supports, which will contribute to broader resilience building and also mitigate material risk to WFP’s programmes. Op￾6 Recommendation 6: Implement appropriate gender analysis and approaches for Phase II. While the McGovern￾Dole Programme design establishes alignment with relevant gender policies, frameworks and initiatives, WFP’s efforts toward gender-equitable outcomes and gender-appropriate approaches would benefit from a broader lens that understands and monitors how interventions influence GEWE. The absence of a comprehensive gender analysis, and the missed opportunity to implement gender- sensitive approaches in interventions where they are relevant, were significant gaps in Phase I. The programme should make learning on gender an explicit focus for Phase II, drawing on what has been learned in Phase I. The ET notes the progress WFP has made at a corporate level in driving the gender, social inclusion, and disability agenda forward. This gives WFP Rwanda the opportunity to draw on the corporate momentum, experience, and technical resources to establish the McGovern-Dole Phase II programme as gender-sensitive to start with. In particular, the WFP Gender and Age marker is a useful starting tool to measure the extent that gender and age are integrated into programme design and monitoring. For Phase II, WFP should conduct a gender analysis at baseline and consider a gender audit at midterm and/or at endline. WFP should also consider conducting a social norms analysis to determine if barriers to behavior change around gender (and health and dietary practices) are rooted in social norms that can be addressed by the programme. This will help direct the country office to appropriate tools and approaches for further integration of gender and other inclusion domains into Phase II strategy and operations. Operational WFP Rwanda country office With input from RBN, HQ and programme partners High by Phase II midpoint* *with the opportunity to use the mid-term review as a validation exercise of the relevance of any outstanding issues Date | Report Number 56 Date | Report Number 57 Annex 1: Stakeholder Analysis Table 11: Stakeholder analysis and mapping Stakeholders Interest in the evaluations and likely use of evaluation reports for this stakeholder WFP STAKEHOLDERS Country Office Rwanda Responsible for the country-level planning and operations implementation, WFP Rwanda has a direct stake in the evaluations and an interest in learning from experience to inform decision-making. It is also called upon to account internally as well as to its beneficiaries and partners for performance and results of its operation. Regional Bureau (RB) Nairobi Responsible for both oversight of country office and technical guidance and support, the RB management has an interest in an independent/impartial account of the operational performance as well as in learning from the evaluation findings to apply this learning to other country offices. The Regional Evaluation Officers supports country office /RB management to ensure quality, credible and useful decentralized evaluations. Office of Evaluation (OEV) OEV has a stake in ensuring that decentralized evaluations deliver quality, credible and useful evaluations respecting provisions for impartiality as well as roles and accountabilities of various decentralized evaluation stakeholders as identified in the evaluation policy. WFP Executive Board (EB) The WFP governing body has an interest in being informed about the effectiveness of WFP programmes. These evaluations will not be presented to the Board but their findings may feed into thematic and/or regional syntheses and corporate learning processes. OTHER STAKEHOLDERS Beneficiaries As the ultimate recipients of food assistance, beneficiaries have a stake in WFP determining whether its assistance is appropriate and effective. Consequently, students, teachers, and SGACs are considered key stakeholders. The level of participation of women and men, boys and girls in the evaluations through interviews, surveys and focus-group discussions in the evaluations and their perspectives will be sought. Available data will be disaggregated by sex and age when relevant. Government As WFP is implementing the McGovern-Dole Programme to support the Government in setting up a national school feeding programme, MINEDUC, MINAGRI, the Ministry of Infrastructure (MININFRA), Ministry of Local Affairs (MINALOC) and the Ministry of Gender and Family Promotion (MIGEPROF) as well as the four implementing districts have a direct interest in knowing whether activities are aligned with their priorities, harmonized with the actions of other partners, and attain the expected results. Issues related to capacity strengthening, handover and sustainability will be of interest. Date | Report Number 58 Main donor USDA USDA funds WFP’s McGovern-Dole Programme through a McGovern-Dole Grant (FFE-696-2015/007-00) and so has a strong interest in knowing whether their funds have been spent efficiently and if WFP’s work has been effective and contributed to USDA’s own strategies and programmes. Other donors MasterCard funds some of the food commodities used in the programme and contributed to overall implementation of complementary activities. UN Country Team The UNDAP 2018-23 contributes to the national goal of “developing Rwandans into a capable and skilled people with quality standards of living and a stable and secure society” through Outcome 3 which reads as follows: By 2023 people in Rwanda, particularly the most vulnerable, enjoy increased and equitable access to quality education, health, nutrition and WASH services.” The UNCT therefore has a shared interest in the evaluation findings, particularly UNICEF, UNESCO, WHO, FAO, UNFPA and UNHCR whose work in this area is interconnected with that of WFP. School Feeding Technical Working Group members NGOs are WFP’s partners for the implementation of some activities while at the same time having their own interventions. The results of the evaluation might affect future implementation modalities, strategic orientations and partnerships. The School Feeding Technical Working Group is a key group of stakeholders, including members World Vision (sub-grantee focusing on literacy and health), Gardens for Health International and Rwanda Biomedical Centre (key implementing partners), and others. Date | Report Number 59 Annex 2: Evaluation Timeline Table 12 shows the original (planned) timeline presented during the inception phase. The timeline was adjusted numerous times given repeated delays related to COVID-19, e.g., school closures and repeated changes to re-opening dates, the need to procure official permissions for local travel, and scheduling interviews with government officials and other stakeholders who were occupied with the pandemic. These delays had cascading effects on the analysis and reporting stages. The ET has noted in the “actual” column the dates that the listed tasks were realized; precise dates are given where available however given the numerous changes, other dates are approximated. Table 12: Timeline of tasks and deliverables, by phase Task Dec 14-20 Dec 21-27 Dec 28 - Jan 3 Jan 4-10 Jan 11-17 Jan 18-24 Jan 25-31 Feb 1-7 Feb 8-14 Feb 15-21 Actual Inception/ Planning Planned Desk review of documents x x x x approx. the same Inception meetings/ communications x x x approx. the same Intro mtgs w/ World Vision to modify EGRA tool (WV confirmed they will send updated EGRA tool by Dec 23) x x approx. the same Review any other surveys that need to be programmed (headmaster); revise as needed x x x approx. the same Translate headmaster, SGAC, teacher, community tools into Kinyarwanda x x x approx. the same Prepare and submit brief inception report: methodology, workplan/ field schedule, survey tools and topical outlines x draft submitted Dec 23 Dec 29: WFP returns feedback on inception report x Jan 07 Revise/ finalize inception report x Jan 15 (slight delay given ongoing conversations re: changing possibilities for fieldwork start dates due to COVID-19 protocols in Rwanda) Data Collection Data collection task schedule delayed due to COVID-related constraints: extended school closures and need to secure travel permissions Date | Report Number 60 Table 12: Timeline of tasks and deliverables, by phase Task Dec 14-20 Dec 21-27 Dec 28 - Jan 3 Jan 4-10 Jan 11-17 Jan 18-24 Jan 25-31 Feb 1-7 Feb 8-14 Feb 15-21 Actual Inception/ Planning Planned TANGO works with World Vision to modify and finalize EGRA tool x approx. the same Jan 4-5 TBD: Inception day w/ local team in WFP office and int'l team virtual (+ prep) x these meetings happened virtually throughout January Schedule remote interviews x x late Jan into Feb Conduct and write up remote interviews x x x some in Jan, most in Feb through mid-Mar Field test tools (in Kigali) (EGRA and headmaster) and quant analyst review x late Jan; desk-based review testing as field team awaits GoR travel clearance Feb 03: field team receives GoR travel clearance Schedule school visits and KIIs x early Feb School, community, district, cooperative data collection and write-up; ongoing consults/debriefs with local & int'l teams x x x Feb 04-24 Ongoing remote QA of EGRA and headmaster surveys and tech support; ongoing updates to local consultants as data are analyzed x x x Feb 04-24 Follow-up interviews and consultations x x x Feb-Mar In-country debrief Mar 23 Analysis and Reporting Analyze EGRA and headmaster survey results (incl. combining w/BL data and conducting statistical tests) x x late Mar – Apr 20 Write up EGRA and headmaster survey results x x late Mar – Apr 20 Local consultants clean up notes and generate summary reports (interim product for internal use); review by int'l team x late Mar – Apr 20 Submit draft evaluation report (ER) x Apr 21 Date | Report Number 61 Table 12: Timeline of tasks and deliverables, by phase Task Dec 14-20 Dec 21-27 Dec 28 - Jan 3 Jan 4-10 Jan 11-17 Jan 18-24 Jan 25-31 Feb 1-7 Feb 8-14 Feb 15-21 Actual Inception/ Planning Planned Circulation of draft Evaluation Report for review to HQ and ERG two weeks (per TOR) DEQAS comments received May 19 ERG comments received May 28 DEQAS comments review call May 24 extensive consultation with country office to clarify indicator data and other questions arising from the review process Submit revised ER based on HQ & ERG feedback three weeks to incorporate feedback (per TOR) Aug 20 Circulation of draft Evaluation Report for review to USDA three weeks (per TOR) pending Submit final ER based on USDA feedback three weeks to incorporate feedback (per TOR) pending Prepare and submit 2-page brief March - early April, pending status of report finalization pending Date | Report Number 62 Annex 3: Government Policies and Priorities Related to Food Security, Social Protection and Education Table 13: Government policies and priorities related to food security, social protection and education Vision 2020 Vision 2020 is the main policy framing Rwanda’s development priorities. It seeks to Vision 2020 seeks to modernize agriculture and increase productivity,193 and transform Rwanda into a middle-income country with healthier, educated and more prosperous Rwandans by 2020. Vision 2020 considers gender a cross-cutting issue and commits to establishing a gender friendly legal and policy framework. Multi-Sectoral Food and Nutrition Policy and Strategic Plan (2013– 2018) This plan addresses stunting through multi-sectoral nutrition activities, including nutrition screening of children under age five.194 A more recent strategic plan for agriculture investment (PSTA IV 2018–2023) aims to mainstream food security and nutrition throughout strategic programmes. Multi-stakeholder platforms such as the Inter-Ministerial Coordination Committee have been established to expand nutrition activities jointly implemented by the Government and development partners. National Social Protection Strategy (2011) The NSPS prioritizes development of the social protection sector to ensure that all poor and vulnerable people are guaranteed a minimum income and access to core public services. The Vision 2020 Umurenge Programme (VUP) is a large-scale Government social protection programme that aims to eradicate extreme poverty by 2020.195 Implemented by the Local Administrative Entities Development Agency (LODA) under the Ministry of Local Government (MINALOC), VUP targets schools in the poorest areas. It is part of the Economic Development and Poverty Reduction Strategy (EDPRS 2), which includes indicators and targets to reduce chronic malnutrition among children under two, reduce poverty, and create employment, exports, and 11.5 percent economic growth.196 National Strategy for Transformation (2018-2023) Rwanda’s key poverty reduction strategy focuses on three pillars, of which the social transformation pillar has a priority area to ensure quality of education for all, aiming at building a knowledge-based economy.197 The other two pillars are economic transformation and transformational governance. Education Sector Strategic Plan (ESSP) (2010‐2015) The ESSP aims to improve the quality of education. It prioritizes school feeding as a key component of school health and nutrition. The Rwanda Education Board (REB) supports the ESSP and works to build teachers’ capacities. The 193 Government of Rwanda. 2004. Rwanda Vision 2020. 194 USAID. 2018. Rwanda: Nutrition Profile. 195 NISR. 2018. Vision 2020 Umurenge Program (VUP) - Baseline Survey. 196 Government of Rwanda. 2013. Economic Development and Poverty Reduction Strategy 2013 – 2018. 197 United Nations Rwanda. United Nations Development Assistance Plan UNDAP II 2018 -2023 A summary. Date | Report Number 63 Competence-Based-Curriculum developed by the REB and MINEDUC establishes specific descriptors for literacy relevant to McGovern-Dole Programme goals.198 Guided by the ESSP, now in its fourth cycle, Rwanda has invested significant resources toward improving the quality and coverage of all levels of education, as well as toward implementing policies that aim to achieve universal and equitable access to basic education for all Rwandan children. The provision of universal, compulsory and free nine years of basic education for children aged seven to 15 years has had a significant impact on increasing access, and this is now being expanded to 12 years. The curriculum has undergone a major reform, with a new competence-based curriculum being phased in from January 2016 (United Nations Rwanda. 2017. Rwanda Common Country Analysis [cited in evaluation TOR]). School Health and Nutrition (2013) policy This policy states, “all Rwandan schoolchildren shall achieve their full development potential by studying in a healthy environment in child-friendly schools.” It includes capacity building for teachers and students on school health and nutrition; school health clubs help disseminate knowledge. Social Protection Sector Strategy (2018-2024) The Government has set the ambitious goal of moving from low-income status to upper-income status by year 2050, and through the forthcoming Social Protection Sector Strategy (2018-2024), has made a commitment to provide a life-cycle approach to social protection systems. The economy is growing and income inequality is decreasing, which strengthens the possibility that the Government will indeed move toward its goals and self-finance social protection, including access to education and food insecurity safety nets for all. National Gender Policy 2010 This policy sets forth guidelines for integrating gender equality and equity into government planning and programmes for social and economic development. This policy is relevant to the goal of equal access to education for girls. Girls’ Education Policy 2008 The purpose of this policy is to establish a legislative and institutional framework to promote gender equality in education, to ensure that gender issues are part of planning at all levels, and to eliminate gender discrimination in education. This policy is relevant to ensuring that girls are able to attend school, and receive non-discriminatory treatment in learning environments. 198 (1) Read a variety of texts accurately and fast; (2) Express ideas, messages and events through writing legible texts in good handwriting with correctly spelt words; (3) Communicate ideas effectively through speaking using correct phonetics of words; and (4) Listen carefully for understanding and seeking clarification when necessary. REB/MINEDUC. 2015. Competence-Based Curriculum: Summary of Curriculum Framework Pre-primary to Upper Secondary 2015. Date | Report Number 64 Annex 4: Map (see next page) Date | Report Number 65 Figure 3: McGovern-Dole Programme coverage in Rwanda Date | Report Number 66 Annex 5: Results Framework Source: WFP Rwanda. 2015. Results Framework. PowerPoint. (“Foundational Results” diagram). (see next three pages) Date | Report Number 67 Date | Report Number 68 Date | Report Number 69 Date | Report Number 70 Annex 6: Outcome and Outputs Tables MGD SO1: Improved Literacy of School-Aged Children Outcome: Number of individuals benefiting directly from USDA-funded interventions Table 14: Number of individuals benefiting directly from USDA-funded interventions (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target New 133,328 0 96,362 72.3% 110,779 83,1% 124,565 93,4% 138,123 103,6% 150,255 112.7% Continuing 77,503 0 0 - 74,294 95,9% 71,627 92,4% 64,000 82,6% 80,887 104.4% Female 61,240 0 48,689 79.5% 55,753 91% 62,508 102,1% 69,151 112,9% 75,096 122.6% Male 69,088 0 47,673 69% 55,026 79,6% 62,057 89,8% 68,972 99,8% 75,159 108.8% Total 133,328 0 96,362 72.3% 110,779 83,1% 124,565 93,4% 138,123 103,6% 150,255 112.7% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Exceeded targets may be attributed to higher numbers of 'new' students in some years and a much higher number of teachers and parents trained/supported than what was initially planned. Outcome: Number of individuals benefiting indirectly from USDA-funded interventions Table 15: Number of individuals benefiting indirectly from USDA-funded interventions (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 175,379 0 285,345 162,7% 254,976 145,4% 243,750 139% 235,230 134.1% 238,872 136.2% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The overall cumulative number of students supported over the life of the project was higher than planned and therefore the indirect beneficiaries, calculated as a multiplier of 3, is also higher than planned. Outcome: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Table 16: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (cumulative figures) Final target FY16 FY2017 FY2018* FY2019 FY2020* FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 77% 0 43.3% 56.2% - - 62.9% 81.7% - - 85% 110.4% Male 62% 0 39.8% 64.2% - - 56.8% 91.6% - - 70.2% 113.2% Total 69% 0 41.5% 60.1% - - 59% 85.5% - - 77.7% 112.6% Source: Midterm and endline EGRA survey. * Data not collected this year. Date | Report Number 71 MGD 1.1: Improved Quality of Literacy Instruction Outcome: Number of teachers in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance Table 17: Number of teachers in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 984 0 874 88.8% 949 96.4% 1,240 126% 1,240 126% 1,240 126% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Both grade 1-3 teachers and grade 4-6 teachers were trained. Targets were exceeded because all teachers who teach grade 1 to 3 were trained (not only Kinyarwanda teachers, but also teachers of different subjects). Furthermore, World Vision also trained upper grade primary English teachers (teachers of grades 4 - 6) in all 104 schools. These were not originally included in the plan and were therefore also not included in the target. MGD 1.1.1: More Consistent Teacher Attendance Outcome: Number of teachers in target schools who attend and teach school at least 90 percent of scheduled school days per school year Table 18: Number of teachers in target schools who attend and teach school at least 90 percent of scheduled school days per school year (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 653 0 1,354 207.4% 1,423 217.9% 1,435 219.8% 1,558 238.6% 1,680 257.3% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: It is difficult to report and interpret this indicator as a percentage. Below are the raw data, using # of teachers attending/teaching for at least 90 percent of scheduled days as the numerator and # of teachers as the denominator, and the corresponding percentage, by year: 1 October 2020 – 31 March 2021: 1680/1706 (98.5 percent) 1 October 2020 – 31 March 2020: not assessed; schools closed due to COVID-19 1 April 2019 - 30 September 2019: 1435/1529 (93.9 percent) 1 October 2018 - 31 March 2019: 1409/1463 (96.3 percent) 1 October 2017 - 30 September 2018: 1423/1448 (98.3 percent) Note also that both grade 1-3 teachers and grade 4-6 teachers were trained. Targets were exceeded because all teachers who teach grade 1 to 3 were trained (not only Kinyarwanda teachers, but also teachers of different subjects). Furthermore, World Vision also trained upper grade primary English teachers (teachers of grades 4 - 6) in all 104 schools. These were not originally included in the plan and were therefore also not included in the target. Date | Report Number 72 Output: Promote teacher attendance and recognition Table 19: Number of teachers receiving awards (annual figures) FY Actual Target Performance (actual vs target) 2016 0 104 - 2017 0 298 - 2018 237 298 80% 2019 431 298 145% 2020 0 104 0% 2021 0 104 0% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 1.1.2: Better Access to School Supplies & Materials Outcome: Number of textbooks and other teaching and learning materials provided as a result of USDA assistance Table 20: Number of textbooks and other teaching and learning materials provided as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 132,990 0 59,653 44.9% 224,621 168.9% 258,005 194% 258,005 194% 269,237 202.4% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Exceeded targets are attributed to the project’s response to the demand for new storybooks and the substantial need for reading materials, especially in the community. Outputs: Distribute school supplies and material Table 21: Number of textbooks and other teaching and learning materials (literacy starter kits) provided as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 26 - 2017 70 26 269% 2018 0 26 - 2019 0 26 - 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 73 Table 22: Number of students benefiting from literacy starter kits provided as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 20,750 - 2017 65,669 20,750 316% 2018 96,041 20,750 463% 2019 87,011 20,750 419% 2020 41,877 20,750 202% 2021 44,648 20,750 215% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data MGD 1.1.3: Improved Literacy Instructional Materials Outcome: Number of target schools with supplemental reading materials available to students as result of USDA assistance Table 23: Number of target schools with supplemental reading materials available to students as result of USDA assistance (cumulative figures) Final target FY16 FY2017-21 Actual Actual Actual vs. target 104 0 104 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Outputs: Establish Libraries and Produce Books and Supplementary Reading Materials Table 24: Number of teaching and reading materials procured as a result of USDA assistance to be used in schools (annual figures) FY Actual Target Performance (actual vs target) 2016 0 6,120 - 2017 10,933 6,300 174% 2018 2,080 6,300 33% 2019 21,528 0 - 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data Table 25: Number of books distributed in communities to establish school and community libraries (annual figures) FY Actual Target Performance (actual vs target) 2016 0 28,560 - 2017 10,080 29,400 34% 2018 164,968 29,400 561% 2019 11,856 0 - 2020 0 0 - 2021 11,322 0 11.322% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 74 Table 26: Number of community-generated reading materials (annual figures) FY Actual Target Performance (actual vs target) 2016 0 4,680 - 2017 290 7,722 4% 2018 92,304 7,254 1,272% 2019 11,856 7,254 163% 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 27: Number of students benefiting from libraries and new reading materials provided as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 20,700 - 2017 84,876 41,500 205% 2018 85,139 83,106 102% 2019 81,250 83,106 98% 2020 75,973 83,106 91% 2021 79,624 83,106 96 Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 1.1.4: Increased Skills and Knowledge of Teachers Outcome: Increased Skills and Knowledge of Teachers Table 28: Increased skills and knowledge of teachers (cumulative figures) Final target FY16 FY2017 FY2018 FY2019-21 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target # who use new techniques or tools 653 0 874 133.8% 949 145.3% 1,240 189.9% # trained/certified 816 0 1 105 135.4% 1,427 174.9% 1,509 184.9% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * Number of teacher/educators/teaching assistants in target schools who demonstrate use of new techniques or tools as a result of USDA assistance ** Number of teacher/educators/teaching assistants trained or certified as result of USDA assistance Note: The total 1,509 teachers trained is comprised of 1,072 grade 1 to grade 3 teachers and 437 grade 4 to grade 6. Achievement was over target because all teachers who teach grade 1 to 3 were trained – not only Kinyarwanda teachers, as there was a challenge regarding teachers changing the subjects they teach). Furthermore, World Vision also trained upper￾grade primary English teachers (grades 4 to 6 teachers) in all 104 schools. These were not originally included in the plan and were therefore also not included in the target. Date | Report Number 75 Output: Training teachers Table 29: Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 168 - 2017 874 272 321% 2018 949 272 349% 2019 1,240 272 456% 2020 1,195 0 - 2021 818 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Both grade 1-3 teachers and grade 4-6 teachers were trained. Targets were exceeded because all teachers who teach grade 1 to 3 were trained (not only Kinyarwanda teachers, but also teachers of different subjects). Furthermore, World Vision also trained upper grade primary English teachers (teachers of grades 4 - 6) in all 104 schools. These were not originally included in the plan and were therefore also not included in the target. MGD 1.1.5: Increased Skills and Knowledge of Administrators Outcome: Increased Skills and Knowledge of School Administrators Table 30: Increased skills and knowledge of school administrators (cumulative figures) Final target FY16-17 FY2018-19 FY2020-21 Actual Actual Actual vs. target Actual Actual vs. target # who use new techniques or tools 93 0 252 271% 252 271% # trained/certified 139 0 252 181.3% 262 188.5% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * Number of school administrators and officials in target schools who demonstrate use of new techniques or tools as a result of USDA ** Number of school administrators and officials trained or certified as result of USDA assistance Note: Overachievement for this indicator was due to the fact that mentors were not included in the initial target. However, to ensure alignment with the teacher coaching model, World Vision decided to include mentors in the teacher coaching activities. Date | Report Number 76 Output: Training school administrators Table 31: Number of school administrators and officials in target schools who demonstrate use of new techniques and tools as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 43 - 2017 0 32 - 2018 181 32 566% 2019 237 32 741% 2020 239 0 - 2021 243 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 1.2: Improved Attentiveness MGD 1.2.1: Reduced Short-Term Hunger / MGD 1.2.1.1: Increased Access to Food (School Feeding) Outcome: Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance Table 32: Number of daily school meals (breakfast, snack, lunch) provided to school￾age children as a result of USDA assistance (cumulative figures) FY Final Target Actual Actual vs. target FY2016 74,808,333 0 0 FY2017 17,631,534 23.6% FY2018 31,698,108 42.4% FY2019 46,381,463 62.% FY2020 52,262,213 69.9% FY2021 60,168,995 80.4 % Source: WFP. Semi-annual donor reports 2016-2020 and consultation with WFP for FY2021 data. Note: In the last reporting period (1 October 2020 – 31 March 2021), 4,906,782 meals were provided. The number is lower than previous semi-annual periods because of school closures: students were not in school in October, P5-P6 returned in early November, P4 later in November, and P1-P3 in January. Date | Report Number 77 Outcome: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance Table 33: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 66,378 0 47,717 71.9% 54,714 82.4% 61,166 92.1% 67,809 102.2% 73,403 110.6% Male 61,272 0 47,398 77.4% 54,681 89.2% 61,396 100.2% 68,311 111.5% 74,395 121.4% New 127,650 0 95,115 74.5% 109,395 85.7% 122,562 96% 136,120 106.6% 147,798 115.8% Continuing 71,970 0 0 - 73,174 101.7% 68,083 94.6% 64,852 90.1% 67,946 94.4% Total 127,650 0 95,115 74.5% 109,395 85.7% 122,562 96% 136,120 106.6% 147,798 115.8% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The >100 percent achievement in FY2021 is explained by the addition of 11,678 new students to the programme (5,594 females and 6,084 males). Outcome: Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance Table 34: Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 66,378 0 47,717 71.9% 54,714 82.4% 61,166 92.1% 67,809 102.2% 73,403 110.6% Male 61,272 0 47,398 77.4% 54,681 89.2% 61,396 100.2% 68,311 111.5% 74,395 121.4% New 127,650 0 94,572 74.5% 109,395 85.7% 122,562 96% 136,120 106.6% 147,798 115.8% Continuing 71,970 0 0 - 73,174 101.7% 68,083 94.6% 64,852 90.1% 67,946 94.4% Total 127,650 0 95,115 74.5% 109,395 85.7% 122,562 96% 136,120 106.6% 147,798 115.8% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The >100 percent achievement in FY2021 is explained by the addition of 11,678 new students to the programme (5,594 females and 6,084 males). Outputs: Provide school meals Table 35: Number of daily school meals provided to school age children as a result USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 14,962,000 - 2017 14,807,052 14,962,000 99% 2018 14,066,574 14,962,000 94% 2019 14,683,355 14,962,000 98% 2020 5,880,750 14,962,000 39% 2021 4,906,782 14,962,000 33% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 78 Table 36: Number of school-aged children receiving daily school meals as a result of USDA assistance (annual figures) Female Male New Continuing Total FY Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 43,215 - 0 39,890 - 0 83,106 - 0 71,970 - 0 83,106 - 2017 47,460 43,215 110% 47,112 39,890 118% 95,115 11,136 854% 73,174 71,970 102% 94,572 83,106 114% 2018 41,696 43,215 96% 43,296 39,890 109% 14,280 14,280 100% 71,327 71,970 99% 84,992 83,106 102% 2019 39,683 43,215 92% 41,567 39,890 104% 13,167 11,136 118% 68,083 71,970 95% 81,250 83,106 98% 2020 38,261 43,215 89% 40,149 39,890 101% 13,558 11,136 122% 64,852 71,970 90% 78,410 83,106 94% 2021 38,995 43,215 90% 40,629 39,890 102% 11,678 11,136 105% 67,946 71,970 94% 79,624 83,106 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 37: Number of students enrolled in schools receiving USDA assistance (annual figures) Female Male Total New Continuing FY Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 43,215 - 0 39,890 - 0 83,106 - 0 88,784 - 0 0 - 2017 47,460 43,215 110% 47112 39,890 118% 94,572 83,106 114% 96,362 11,136 865% 0 77,648 - 2018 41,696 43,215 96% 43,296 39,890 109% 84,992 83,106 102% 14,417 11,136 129% 74,294 77,648 96% 2019 39,683 43,215 92% 41,567 39,890 104% 81,250 83,106 98% 13,786 11,136 124% 71,627 77,648 92% 2020 38,261 43,215 89% 40,149 39,890 101% 78,410 83,106 94% 13,558 11,136 122% 64,000 77,648 82% 2021 38,995 43,215 90% 40,629 39,890 102% 79,624 83,106 96% 11,678 11,136 105% 67,946 77,648 88% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 38: Number of individuals benefiting directly from USDA funded interventions (annual figures) Female Male Total FY Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 46,167 - 0 42,626 - 0 88,784 - 2017 48,689 46,167 105% 47,673 42,626 112% 96,362 88,784 109% 2018 43,076 42,626 101% 44,931 46,167 97% 88,007 88,793 99% 2019 41,479 42,626 97% 43,934 46,167 95% 85,413 88,793 96% 2020 39,754 42,626 93% 41,975 46,167 91% 81,729 88,784 92% 2021 45,614 42,626 107% 47,405 46,167 103% 93,019 88,784 105% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 79 Table 39: Number of individuals benefiting indirectly from USDA funded interventions (annual figures) FY Actual Target Performance (actual vs target) 2016 0 175,379 - 2017 283,716 175,379 162% 2018 254,976 249,318 102% 2019 243,750 175,379 139% 2020 235,230 175,379 134% 2021 238,872 175,379 136% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 40: Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (annual figures) Female Male New Continuing Total FY Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 43,215 - 0 39,890 - 0 83,106 - 0 0 - 0 83,106 - 2017 47,460 43,215 110% 47,112 39,890 118% 95,115 11,136 854% 0 71,970 - 94,572 83,106 114% 2018 41,696 43,215 96% 43,296 39,890 109% 14,280 11,136 128% 73,174 71,970 102% 84,992 83,106 102% 2019 39,683 43,215 92% 41,567 39,890 104% 13,167 11,136 118% 68,083 71,970 95% 81,250 83,106 98% 2020 38,261 43,215 89% 40,149 39,890 101% 13,558 11,136 122% 64,852 71,970 90% 78,410 83,106 94% 2021 38,995 43,215 90% 40,629 39,890 102% 11,678 11,136 105% 67,946 71,970 94% 79,624 83,106 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 80 Table 41: Number of students regularly (80%) attending USDA supported classrooms/schools (annual figures) Female Male Total FY Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 43,215 - 0 39,890 - 0 83,106 - 2017 39,937 43,215 92% 41,470 39,890 104% 81,406 83,106 98% 2018 39,863 43,215 92% 40,957 39,890 103% 80,819 83,106 97% 2019 38,438 43,215 89% 39,628 39,890 99% 78,066 83,106 94% 2020 0 43,215 - 0 39,890 - 0 83,106 - 2021 37,439 43,215 87% 35,894 39,890 90% 73,333 83,106 88% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: No reporting for FY2020 due to COVID-related school closures. Table 42: Percent of school-age children receiving a minimum acceptable diet (male) (annual figures) Female Male Total FY Actual Target A/T Actual Target A/T Actual Target A/T Not collected anymore (globally) Outputs: Develop Partnerships with Farmer Groups to Supply Food to Schools Table 43: Number of public-partnerships formed as a result of USDA assistance (other) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 16 - 2017 15 8 188% 2018 0 7 - 2019 0 0 - 2020 16 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 44: Number of smallholder farmers benefiting from partnerships with schools (annual figures) FY Actual Target Performance (actual vs target) 2016 0 5,935 0% 2017 2,918 8,903 33% 2018 3,922 11,500 34% 2019 3,922 11,500 34% 2020 9,771 11,500 85% 2021 9,771 11,500 85% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 81 Table 45: Value of public and private sector investments leveraged as a result of USDA assistance (private) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 0 - 2018 0 0 - 2019 0 0 - 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Outputs: Establish and Maintain School Gardens Table 46: Number of school gardens established and maintained (annual figures) FY Actual Target Performance (actual vs target) 2016 0 104 - 2017 0 104 - 2018 104 104 100% 2019 104 104 100% 2020 104 104 100% 2021 106 104 102% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 47: Number of students benefiting from the establishment and maintenance of school gardens (annual figures) FY Actual Target Performance (actual vs target) 2016 0 83,106 - 2017 0 83,106 - 2018 84,992 83,106 102% 2019 81,250 83,106 98% 2020 75,973 83,106 91% 2021 79,624 83,106 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 82 MGD 1.3: Improved Student Attendance Outcome: Number of students regularly (80%) attending USDA supported classroom/school Table 48: Number of students regularly (80%) attending USDA supported classroom/school (annual figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 66,378 0 42,226 63.6% 48,943 73.7% 55,137 83.1% 55,137 83.1% 60,507 91.2% Male 61,272 0 43,409 70.8% 50,327 82.1% 56,707 92.5% 56,707 92.5% 62,079 101.3% Total 127,650 0 85,635 67.1% 99,201 77.7% 111,841 87.6% 111,841 87.6% 122,586 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: While the actual/target percentage for FY2021 for females is 91.2 percent (close to but not meeting the target), the total number of female students regularly (80%) attending USDA-supported classrooms (37,439) is 96 percent of female students enrolled. MGD 1.3.1: Increased Economic and Cultural Incentives / 1.3.1.1 Increased Access to Food (School Feeding) Output: Increased Access to Food (School Feeding) - Reported under MGD 1.2.1.1 / Activity reported under Output 1.2.1.1 MGD 1.3.2: Reduced Health-Related Absences Outcome: Percent of students who miss more than 10 school days per year due to illness Table 49: Percent of students who miss more than 10 school days per year due to illness (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 5% 0 1.8% 103.4% 1.4% 103.8% 0.8% 104.4% 0.8% 104.4% 1.72% 103.5% Male 5% 0 1.6% 103.6% 1.3% 103.9% 0.6% 104.6% 0.6% 104.6% 2.38% 102.8% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 1.3.3: Improved School Infrastructure Outcome: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) Table 50: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 104 0 55 52.9% 80 76.9% 80 76.9% 93 89.4% 104* 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * At the end of June 2021, this number will be 107 as new infrastructure has been built in the three new schools added to the programme. Outcome: Number of educational facilities (i.e. school buildings, classroom, and latrines) rehabilitated/constructed as result of USDA assistance (latrines) Table 51: Number of educational facilities (i.e. school buildings, classroom, and latrines) rehabilitated/constructed as result of USDA assistance (latrines) (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020-21 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Stances 364 0 266 73.1% 364 100% 392 107.7% 476 130.8% Latrines 26 0 19 73.1% 26 100% 28 107.7% 34 130.8% Date | Report Number 83 Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Based on additional needs identified in target schools, World Vision mobilized additional funding to build five latrine blocks beyond what the project had originally targeted. The project target of 26 has thus been exceeded. Outputs: Provide fuel-efficient stoves Table 52: Number of fuel-efficient stoves provided and rehabilitated (annual figures) FY Actual Target Performance (actual vs target) 2016 0 104 - 2017 109 0 - 2018 50 30 167% 2019 0 0 - 2020 52 0 - 2021 44 0 44% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 53: Number of individuals directly benefiting from the provision and rehabilitation of fuel-efficient stoves (annual figures) FY Actual Target Performance (actual vs target) 2016 0 83,626 - 2017 40,224 83,626 48% 2018 22,065 83,626 26% 2019 0 83,626 - 2020 0 83,626 - 2021 77,352 83,626 92% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 84 MGD 1.3.4: Increased Student Enrolment Outcome: Number of students enrolled in school receiving USDA assistance Table 54: Number of students enrolled in school receiving USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 66,378 0 47,717 71.9% 54,714 82.4% 61,166 92.1% 67,809 102.2% 73,403 110.6% Male 61,272 0 47,398 77.4% 54,681 89.2% 61,396 100.2% 68,311 111.5% 74,395 121.4% Total 127,650 0 95,115 74.5% 109,395 85.7% 122,562 96% 136,120 106.6% 147,798 115.8% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The final cumulative target (FY2021) is higher than target because over time, a higher number of new students enrolled in certain years compared to what was planned. MGD 1.3.5: Increased Community Understanding of Benefits of Education Outcome: Percent of parents in target communities who can name at least three benefits of primary education Table 55: Percent of parents in target communities who can name at least three benefits of primary education (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 90% - 69% 76.7% 69% 76,7% 69.8% 77.6% 69.8% 77.6% 74.3% 82.6% Source: Midterm and endline head teacher survey. Outputs: Raising awareness on the importance of education Table 56: Percent of parents in target communities who can name at least three benefits of primary education (annual figures) FY Actual Target Performance (actual vs target) 2016 0% 60% - 2017 1% 70% 1% 2018 69% 80% 86.3% 2019 69.8% 80% 87.3% 2020 69.8% 90% 77.6% 2021 74.3 90% 82.6% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 85 Table 57: Number of parents trained on the importance of literacy (annual figures) FY Actual Target Performance (actual vs target) 2016 0 1,456 0% 2017 3,897 1,456 268% 2018 14,164 1,456 973% 2019 1,231 1,456 85% 2020 4,038 1,456 277% 2021 19,134 1,456 1,314% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Outputs: Establish activities to promote literacy Table 58: Number of reading clubs established as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 102 - 2017 168 105 160% 2018 42 105 40% 2019 0 0 - 2020 0 0 - 2021 0 0 - Total: 312 Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: In their review of the draft evaluation report, both World Vision and WFP Rwanda affirmed that the total number of reading clubs established is 312, though there were some nuanced differences in the data noted in their comments. World Vision stated that since 2016, the project established 312 reading clubs: 102 in 2016 and 210 in 2017 (which are different annual numbers from the data in this table). WFP Rwanda stated that the correct number is 312 reading clubs established (three clubs per catchment area. The evaluation team accepts 312 as the correct total and considers this the most relevant data point. Table 59: Number of student writing competitions facilitated as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 8 0% 2018 4 8 50% 2019 8 8 100% 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 86 Table 60: Number of students benefiting from the development of reading clubs (annual figures) FY Actual Target Performance (actual vs target) 2016 0 4,080 0% 2017 24,164 8,280 292% 2018 24,107 12,480 193% 2019 21,356 12,480 171% 2020 26,166 12,480 210% 2021 33,624 12,480 269% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 61: Percent of students who, by the end of two grades primary schooling, demonstrate that they can read and understand the meaning of grade level text (annual figures) Female Male Total FY Actual Target A/T Actual Target A/T Actual Target A/T 2016* - - - - - - - - - 2017 43.3% 77% 56.2% 39.8% 62% 64.2% 41.5% 69% 60.1 2018* - - - - - - - - - 2019 62.9% 77% 81.7% 56.8% 62% 91.6% 59% 69% 85.5% 2020* - - - - - - - - - 2021 85% 77% 110,4% 70,2% 62% 113,2% 77.7% 69% 112,6% Source: Midterm and endline EGRA survey. * Data not collected this year. Outputs: Training (SGACs) Table 62: Number of SGACs or similar school governance structures supported as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 104 - 2017 104 104 100% 2018 79 104 76% 2019 104 104 100% 2020 104 104 100% 2021 104 104 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 63: Number of parents trained as part of SGACs (annual figures) FY Actual Target Performance (actual vs target) 2016 0 520 - 2017 1,287 520 248% 2018 327 520 63% 2019 365 520 70% 2020 1,024 520 197% 2021 0 520 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 87 MGD SO2: Increased Use of Health and Dietary Practices MGD 2.1: Improved Knowledge of Health and Hygiene Practices Outcome: Percent of students who can identify at least three key health and hygiene practices Table 64: Percent of students who can identify at least three key health and hygiene practices (cumulative figures) Final target FY2016 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 80% 0 47% 58.8% 47% 58.8% 49.2% 61.5% 49.2% 61.5% 60.9% 76.1% Male 80% 0 48% 60% 48% 60% 48.6% 60.8% 48.6% 60.8% 44.7% 55.9% Source: Midterm and endline EGRA survey. Output: Training (SGACs) – Reported under MGD 1.3.5 Outputs: Training and awareness on good health and hygiene practices Table 65: Number of students reached with health and hygiene messages as a result of USDA assistance (annual figures) FY Actual Target Performance (actual vs target) 2016 0 83,106 - 2017 85,398 83,106 103% 2018 84,992 83,106 102% 2019 81,250 83,106 98% 2020 75,973 83,106 91% 2021 78,538 83,106 95% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 66: Number of individuals trained in child health and nutrition as a result of USDA assistance (annual figures) Female Male Total FY Actual Target A/T Actual Target A/T Actual Target A/T 2016 0 145 0% 0 63 0% 0 208 0% 2017 723 145 499% 623 63 989% 1,346 208 647% 2018 9,741 145 6,718% 8,617 63 1,3678% 18,358 208 8,826% 2019 1,290 145 890% 2,462 63 3,908% 5,242 208 2,520% 2020 1,881 145 1,297% 1,315 63 2,087% 3,196 208 1,537% 2021 0 145 - 0 63 - 0 208 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Targets were exceeded because the project trained all parents on good health practices (as children are more likely to contract hygiene-related diseases at home); the data reflect the number of parents who have been trained on health and hygiene using the CBHPP model. Date | Report Number 88 MGD 2.2: Increased Knowledge of Safe Food Prep and Storage Practices Outcome: Percent of cooks and storekeepers who can identify at least three safe food preparation and storage practices Table 67: Percent of cooks and storekeepers who can identify at least three safe food preparation and storage practices (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 95% 0 92% 96.8% 92% 96.8% 85% 89.5% 85% 89.5% 40% 42.1% Source: Midterm and endline head teacher survey. Note: The low percentage of achievement at the end of the project may be due to turnover and lack of practice due to the COVID-19 school closures in 2020. As most trainings could not happen during the closures, cooks spent nearly a year with no refresher, plus with high turnover, new cooks were not trained for nearly 12 months. Output: Training food preparation and storage practices Table 68: Number of cooks and storekeepers trained on food preparation and storage practices (annual figures) FY Actual Target Performance (actual vs target) 2016 0 520 - 2017 330 520 63% 2018 469 520 90% 2019 459 520 88% 2020 164 520 32% 2021 309 520 59% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The low percentage of achievement at the end of the project may be due to turnover and lack of practice due to the COVID-19 school closures in 2020. As most trainings could not happen during the closures, cooks spent nearly a year with no refresher, plus with high turnover, new cooks were not trained for nearly 12 months. Date | Report Number 89 MGD 2.3: Increased Knowledge of Nutrition Outcome: Number of individuals trained in child health and nutrition as a result of USDA assistance Table 69: Number of individuals trained in child health and nutrition as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 145 0 723 498.6% 5,888 4,061% 5,888 4,061% 7,769 5,358% 7,769 5,357.9% Male 63 0 623 988.9% 4,816 7,644% 4,816 7,644% 6,131 9,732% 6,131 9,731.7% Total 208 0 1,346 647.1% 10,704 5,146% 10,704 5,146% 13,900 6,683% 13,900 6,682.7% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: Targets were exceeded because the project trained all parents on good health practices (as children are more likely to contract hygiene￾related diseases at home); the data reflect the number of parents who have been trained on health and hygiene using the CBHPP model. Output: Establish and maintain school gardens – Reported under MGD 1.2.1 (1.2.1.1) MGD 2.4: Increased Access to Clean Water and Sanitation Services Outcome: Number of schools using an improved water source and sanitation facilities Table 70: Number of schools using an improved water source and sanitation facilities (cumulative figures) Schools Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Water source 104 0 97 93.3% 104 100% 104 100% 104 100% 104 100% Sanitation facilities 104 0 102 98.1% 104 100% 104 100% 104 100% 104 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Output: Building/rehabilitating latrines: For number of latrines/stances, see: Table 51 Table 71: Number of students benefiting from newly constructed/rehabilitated latrines (annual figures) FY Actual Target Performance (actual vs target) 2016 0 9,581 - 2017 13,025 5,593 233% 2018 5,064 5,593 91% 2019 7,455 5,593 133% 2020 4,475 5,593 80% 2021 0 5,593 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 72: Number of schools with improved sanitary facilities (annual figures) FY Actual Target Performance (actual vs target) 2016 0 54 - 2017 102 65 157% 2018 104 85 122% 2019 104 104 100% 2020 104 104 100% 2021 104 104 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 90 Output: Building/rehabilitating water collecting systems Table 73: Number of rainwater catchment systems constructed and/or enhanced (annual figures) FY Actual Target Performance (actual vs target) 2016 0 20 - 2017 46 20 230% 2018 15 15 100% 2019 8 9 89% 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 74: Number of students benefiting from newly constructed and/or enhanced rainwater catchment systems (annual figures) FY Actual Target Performance (actual vs target) 2016 0 15,960 - 2017 24,535 15,960 154% 2018 8,993 11,970 75% 2019 13,887 7,182 193% 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 75: Number of schools using an improved water sources (annual figures) FY Actual Target Performance (actual vs target) 2016 0 54 - 2017 97 65 149% 2018 104 85 122% 2019 104 104 100% 2020 104 104 100% 2021 104 104 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 91 MGD 2.5: Increased Access to Preventative Health Interventions Outcome: Number of students receiving deworming medication(s) Table 76: Number of students receiving deworming medication(s) (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 127,650 0 8,963a 68.9% 102,243 80.1% 115,410 90.4% 128,968 101% 140,646 110.2% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: The number of students enrolled over the life of the project exceeded the original target for deworming, which accounts for end-of-project achievement higher than 100 percent. a Per consultation with the country office, lower achievement rate in FY2017 (and discrepancy with the figure reported for the corresponding output indicator in Table 77) is likely due to defining and reporting this indicator differently in FY2017 compared to FY2018 and beyond. Counting may have been different in the first year, e.g. by including or not including deworming treatments for students who were programme versus non-programme. Output: Distribute deworming medication Table 77: Number of students receiving deworming medications (annual figures) FY Actual Target Performance (actual vs target) 2016 0 83,106 0% 2017 85,398 83,106 103% 2018 83,590 83,106 101% 2019 81,250 83,106 98% 2020 78,410 83,106 94% 2021 79,624 83,106 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 2.6: Increased Access to Requisite Food Prep and Storage Tools and Equipment Outcome: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) Table 78: Number of target schools with access to improved food preparation and storage equipment (kitchens, cook area, storerooms, stoves and kitchen utensils) (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 104 0 55 52.9% 80 76.9% 80 76.9% 93 89.4% 104* 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * At the end of June 2021, this number will be 107 as new infrastructure has been built in the three new schools added to the programme. Date | Report Number 92 Output: Building/rehabilitating kitchen storerooms Table 79: Number of educational facilities (i.e school buildings, classrooms and latrines) rehabilitated/constructed as a result of USDA assistance (kitchens, cook areas) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 52 - 2017 55 52 106% 2018 25 10 250% 2019 0 5 - 2020 13 5 260% 2021 11 5 220% Total 104* Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * Of the total 104 facilities, 55 were kitchens and stoves rehabilitated/constructed by ADRA. An additional four kitchens are being constructed in the four new schools that have been added to the program; the country office reports that these will be finished before the end of the programme. Table 80: Number of educational facilities (i.e school buildings, classrooms and latrines) rehabilitated/constructed as a result of USDA assistance (other school grounds or school buildings, i.e. storeroom) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 52 - 2017 89 52 171% 2018 25 10 250% 2019 0 5 - 2020 13 5 260% 2021 11 5 220% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 81: Number of students benefiting from kitchens, cook areas and storerooms built or rehabilitated (annual figures) FY Actual Target Performance (actual vs target) 2016 0 41,553 - 2017 61,066 83,106 73% 2018 22,065 83,106 27% 2019 0 83,106 - 2020 0 83,106 - 2021 79,624 83,106 96% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 93 Foundational Results MGD 1.4.1/2.7.1: Increased Capacity of Government Institutions Outcome: Number of government staff trained or certified as a result of USDA assistance Table 82: Number of government staff trained or certified as a result of USDA assistance (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Female 58 0 8 13.8% 12 20.7% 56 96.6% 117 201.7% 121 208.6% Male 86 0 6 7% 10 11.6% 114 132.6% 227 264% 231 268.6% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Note: It is likely that only national government staff were contemplated in the original target. District government staff were not included in reporting until FY2019, which would explain the high achievement rates starting in FY2019. Outputs: Capacity building: Local, regional, national Table 83: Number of government staff trained at national level (annual figures) FY Actual Target Performance (actual vs target) 2016 0 13 0% 2017 6 13 46% 2018 11 13 85% 2019 15 13 115% 2020 35 13 269% 2021 3 13 23% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 84: Number of government staff trained at district level (annual figures) FY Actual Target Performance (actual vs target) 2016 0 60 0% 2017 8 60 13% 2018 216 60 360% 2019 151 60 252% 2020 144 60 240% 2021 114 60 190% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 94 MGD 1.4.2/2.7.2: Improved Policy and Regulatory Framework Outcome: Improved Policy and Regulatory Framework Table 85: Improved policy and regulatory framework (cumulative figures) Schools Final target FY2016-FY2020 FY2021 Actual Actual Actual vs. target Number of Child health and nutrition policies, regulations or administrative procedures in the following stages of development as result of USDA assistance 1 0 0 - Number of educational policies, regulations, or administrative procedures in each of the following stages of development as a result of USDA assistance 1 0 1 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Output: Support implementation of the National HGSF program Table 86: Number of HGSF Steering Committee and Technical Committee meetings held (annual figures) FY Actual Target Performance (actual vs target) 2016 0 6 - 2017 13 6 217% 2018 28 6 467% 2019 10 6 167% 2020 8 6 133% 2021 5 6 83% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 87: Number of government financing strategies developed (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 1 - 2018 0 0 - 2019 0 0 - 2020 0 0 - 2021 0 0 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 88: Number of government monitoring and evaluation systems established (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 1 - 2018 0 0 - 2019 0 0 - 2020 1 0 - 2021 1 1 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 95 Table 89: Number of district and national-level Ministry of Education (MINEDUC) and Ministry of Agriculture (MINAGRI) staff supported by new HGSF policies (annual figures) FY Actual Target Performance (actual vs target) 2016 0 73 - 2017 0 73 - 2018 0 73 - 2019 4 73 5% 2020 4 73 5% 2021 4 73 5% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 90: Number of child health and nutrition policies, regulations or administrative procedures in the following stages of development as a result of USDA assistance (stage 5) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 0 - 2018 0 0 - 2019 0 0 - 2020 0 1 - 2021 0 1 - Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Table 91: Number of educational policies regulation, or administrative procedures in the following stages of development as a result of USDA assistance (stage 5) (annual figures) FY Actual Target Performance (actual vs target) 2016 0 0 - 2017 0 0 - 2018 0 0 - 2019 0 0 - 2020 1 1 100% 2021 1 1 100% Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. MGD 1.4.3/2.7.3: Increased Government Support Outcome: Increased Government Support Table 92: Increased government support (annual figures) Schools Final target FY2016-FY2021 Actual Value of public-private sector investments leveraged as a result of USDA assistance (host government) 0 0 Percent increase in budget allocated by the Government of Rwanda to Home-Grown School Feeding Program 0 0 Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. Date | Report Number 96 MGD 1.4.4/2.7.4: Increased Engagement of Local Organizations and Community Groups Outcome: Increased Engagement of Local Organizations and Community Groups Table 93: Increased engagement of local organizations and community groups (cumulative figures) Final target FY16 FY2017 FY2018 FY2019 FY2020 FY2021 Actual Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target SGACs supported * 104 0 104 100% 104 100% 104 100% 104 100% 104 100% Public-private partnerships formed ** 31 0 24 77.4% 24 77.4% 19 61.3% 35 112.9% 35 112.9% Value of public￾private sector investments *** 0 0 0 0 0 0 0 0 0 0 0 0 Source: WFP. Semi-annual donor reports 2016-2020 and evaluation team consultation with WFP for FY2021 data. * Number of SGACs or similar "school" governance structures supported as a result of USDA assistance ** Number of public-private partnerships formed as a result of USDA assistance (other). Variances in achievement of target are due to two factors: i) the subtraction and addition of cooperatives over the course of the project; some dropped out due to disinterest and others previously supported under LRP were shifted under the support umbrella of McGovern-Dole; and ii) the project added cooperatives supported by the two-year USDA LRP programme, which was not foreseen at the start of the project. *** Value of public-private sector investments leveraged as a result of USDA assistance (private) Date | Report Number 97 Annex 7: Terms of Reference Terms of Reference MIDTERM AND ENDLINE EVALUATIONS of WFP Rwanda’s Home Grown School Feeding Programme 2016-2020 (USDA McGovern Dole Grant FFE￾696-2015/007-00) in 2018 and 2020-21 for WFP Rwanda Table of Contents 1. Introduction 2. Reasons for the evaluations 2.1. Rationale 2.2. Objectives 2.3. Stakeholders and users 3. Context and subject of the evaluations 3.1. Context 3.2. Subject of the evaluations 4. Evaluation approach 4.1. Scope 4.2. Evaluation criteria and questions 4.3. Data availability 4.4. Methodology 4.5. Quality assurance and assessment 5. Phases and deliverables 5.1. Midterm evaluation 5.2. Endline evaluation 6. Organization of the evaluation 6.1. Evaluation conduct 6.2. Team composition and competencies 6.3. Security considerations 7. Roles and responsibilities of stakeholders 8. Communication and budget 8.1. Communication 8.2. Budget Annex 1 Map Annex 2 Tentative Evaluation Schedule Annex 3 EC TOR Annex 4 ERG TOR Annex 5 Results frame Date | Report Number 98 Table of Acronyms ACR Annual Country Report CBEHPP Community‐based Environmental Health Promotion Program CFSVA Comprehensive Food Security and Vulnerability Analysis CHAI Clinton Health Access Initiative CHW Community Health Workers CO Country Office CSR Country Strategic Review of Food Security and Nutrition DDP District Development Plans DEO District Education Officers DEQAS Decentralized Evaluation Quality Assurance System DFID Department for International Development EB Executive Board EC Evaluation Committee ERG Evaluation Reference Group ESSP Education Sector Strategic Plan EQAS Evaluation Quality Assurance System GEWE Gender Equality and Women’s Empowerment HGSF Home Grown School Feeding MDG Millenium Development Goal(s) MGD McGovern-Dole MINAGRI Ministry of Agriculture and Animal Resources MINALOC Ministry of Local Affairs MINEDUC Ministry of Education MINESANTE Ministry of Health MININFRA Ministry of Infrastructure NST National Strategy for Transformation OEV WFP Office of Evaluation PTA Parent-Teacher Association RB(N) Regional Bureau (Nairobi) REB Rwanda Education Board SDG Sustainable Development Goal(s) SEO Sector Education Officers SGBV Sexual and Gender-based Violence SMC School Management Committees SPR Standard Project Report WFP World Food Programme UN CCA UN Common Country Analysis UNDAP UN Development Assistance Plan UNDSS UN Department of Safety and Security UNHDI UN Human Development Index USAID United States Agency for International Development USDA United Stated Department of Agriculture WASAC Water and Sanitation Corporation Date | Report Number 99 1. Introduction 1. This Terms of Reference (TOR) is for the midterm and endline evaluations of WFP Rwanda’s Home-Grown School Feeding (HGSF) Programme 2016-2020199 (USDA McGovern Dole (MGD) Grant FFE-696-2015/007- 00) in Rwanda. These two activity evaluations are commissioned by WFP Rwanda and will take place from June/July to December 2018 and December 2020 to May 2021, respectively. 2. The TOR was prepared by WFP Rwanda based upon an initial document review and consultation with stakeholders and following a standard template. The purpose of the TOR is twofold. Firstly, it provides key information to the evaluation team and helps guide them throughout the evaluation process. Secondly, it provides key information to stakeholders about the proposed evaluations. 3. The HGSF Programme supports around 79,000 primary students annually across 107200 schools in four of Rwanda’s poorest and most food insecure districts – Nyaruguru and Nyamagabe in the south and Rutsiro and Karongi in the west (see map in Annex 1). Children in the south receive a daily hot meal whereas students in the western province are provided a porridge meal. As a contribution to the project, some schools occasionally provide locally grown vegetables to enrich the meals. The programme also undertakes activities to improve student literacy outcomes, increased use of health and dietary practices, including WASH, setting up school gardens, providing deworming medication and supporting the strengthening of government staff capacities. 2. Reasons for the evaluations 2.1 Rationale 4. The midterm evaluation was commissioned to provide an evidence-based, independent assessment of the project’s performance so that WFP Rwanda and its project partners, World Vision and Gardens for Health International, can adjust course as necessary for the remainder of the project term. 5. The endline evaluation is being commissioned to provide an evidence-based, independent assessment of the project to evaluate its success, ensure accountability, and generate lessons learned. The endline evaluation will be designed to inform the baseline evaluation of WFP Rwanda’s Home-Grown School Feeding (HGSF) Programme 2020-2025 phase II to maximise efficiency and avoid stakeholder fatigue during data collection considering the geographic overlap and continuity of some activities. It is important to create sufficient opportunity to consolidate and apply the endline learning to the baseline design and even to inform Phase II detailed implementation decisions, to the extent possible. 199 The HGSF programme was later granted a no-cost extension into FY2021 with a new end date of June 30, 2021. 200 From 2016-2019, the programme supported 104 schools; in 2020, WFP began providing meals and select activities to 3 new schools in Karongi making the total project number 107. Date | Report Number 100 2.2 Objectives 6. Evaluations in WFP serve the dual and mutually reinforcing objectives of accountability and learning. • Accountability – The midterm and endline evaluations will assess and report on the performance and results of WFP Rwanda’s Home Grown School Feeding (HGSF) Programme 2016-2020. • Learning – The evaluations will determine the reasons why certain results occurred, to draw lessons and derive good practices for learning. It will also provide evidence-based findings to inform operational and strategic decision-making. Findings will be actively disseminated, and lessons will be incorporated into relevant information-sharing systems. 7. Specifically, the midterm evaluation allowed to i) review the project’s relevance, effectiveness, efficiency, impact, and sustainability, ii) collect performance indicator data for strategic objectives and higher-level results, iii) assess whether the project is on track to meet the results and targets, iv) review the results frameworks and theory of change, and v) identify any necessary mid-course corrections. Hence, more weight was given to learning as can be expected for a midterm evaluation. 8. Specifically, the endline evaluation will i) review the project’s relevance, effectiveness and efficiency, impact, and sustainability, ii) collect performance indicator data for strategic objectives and higher-level results, iii) assess whether the project has succeeded in achieving McGovern Dole’s two strategic objectives, iv) investigate the project’s overall impact, and v) identify meaningful lessons learned that WFP, USDA, and other stakeholders can apply to future programming. Hence, about equal weight is given to learning and accountability. 2.3 Stakeholders and users 9. Several stakeholders both inside and outside of WFP have interests in the results of the evaluations and some of them will be asked to play a role in the evaluation processes. Table 1 below provides a preliminary stakeholder analysis, which should be deepened by the evaluation team as part of the inception phase. 10. Accountability to affected populations is tied to WFP’s commitments to include beneficiaries as key stakeholders in WFP’s work. As such, WFP is committed to ensuring gender equality and women’s empowerment (GEWE) in the evaluation process, with participation and consultation in the evaluation by women, men, boys and girls from different groups. Table 1: Preliminary stakeholders’ analysis Stakeholders Interest in the evaluations and likely use of evaluation reports for this stakeholder WFP STAKEHOLDERS Country Office (CO) Rwanda Responsible for the country level planning and operations implementation, WFP Rwanda has a direct stake in the evaluations and an interest in learning from experience to inform decision-making. It is also called upon to account internally as well as to its beneficiaries and partners for performance and results of its operation. Regional Bureau (RB) Nairobi Responsible for both oversight of COs and technical guidance and support, the RB management has an interest in an independent/impartial account of the operational performance as well as in learning from the evaluation findings to apply this learning to other country offices. The Regional Evaluation Officers supports CO/RB management to ensure quality, credible and useful decentralized evaluations. Office of Evaluation (OEV) OEV has a stake in ensuring that decentralized evaluations deliver quality, credible and useful evaluations respecting provisions for impartiality as well as roles and accountabilities of various decentralised evaluation stakeholders as identified in the evaluation policy. Date | Report Number 101 WFP Executive Board (EB) The WFP governing body has an interest in being informed about the effectiveness of WFP programmes. These evaluations will not be presented to the Board but their findings may feed into thematic and/or regional syntheses and corporate learning processes. OTHER STAKEHOLDERS Beneficiaries As the ultimate recipients of food assistance, beneficiaries have a stake in WFP determining whether its assistance is appropriate and effective. Consequently, students, teachers, and Parent-Teacher Associations (PTAs)201 are considered key stakeholders. The level of participation of women and men, boys and girls in the evaluations through interviews, surveys and focus￾group discussions in the evaluations and their perspectives will be sought. Available data will be disaggregated by sex and age when relevant. Government As WFP is implementing the HGSF Programme to support the government in setting up a national school feeding programme, the Ministry of Education (MINEDUC), the Ministry of Agriculture and Animal Resources (MINAGRI), the Ministry of Infrastructure (MININFRA), Ministry of Local Affairs (MINALOC) and the Ministry of Gender and Family Promotion (MIGEPROF) as well as the four implementing districts have a direct interest in knowing whether activities are aligned with its priorities, harmonised with the actions of other partners, and meet the expected results. Issues related to capacity development, handover and sustainability will be of interest. Main donor USDA USDA funds WFP’s HGSF Programme through a McGovern Dole Grant (FFE￾696-2015/007-00) and so has a strong interest in knowing whether their funds have been spent efficiently and if WFP’s work has been effective and contributed to their own strategies and programmes. Other donors Additionally, MasterCard funds some of the food commodities used in the programme and contributed to overall implementation of complementary activities. UN Country Team The UNDAP 2018-23 contributes to the national goal of “developing Rwandans into a capable and skilled people with quality standards of living and a stable and secure society” through Outcome 3 which reads as follows: By 2023 people in Rwanda, particularly the most vulnerable, enjoy increased and equitable access to quality education, health, nutrition and WASH services. The UNCT therefore has a shared interest in the evaluation findings, particularly UNICEF, UNESCO, WHO, FAO, UNFPA and UNHCR whose work in this area is interconnected with that of WFP. Other partners World Vision, Gardens for Health International, and Rwanda Biomedical Centre NGOs are WFP’s partners for the implementation of some activities while at the same time having their own interventions. The results of the evaluation might affect future implementation modalities, strategic orientations and partnerships. World Vision is a sub-grantee focusing on literacy and health. Gardens for Health International and Rwanda Biomedical Centre are key implementing partners. 11. The primary users of the midterm evaluation were: 201 PTAs have officially been renamed School General Assembly Committees (SGAC). Date | Report Number 102 • WFP Rwanda and its partners World Vision, Gardens for Health International, and Rwanda Biomedical Centre to adjust course as necessary for the remainder of the project term; • MINEDUC and MINAGRI to learn whether the programme is performing well and is aligned with its priorities, particularly in terms of capacity development, handover and sustainability; • USDA as the primary funder of the HGSF Programme to learn whether the programme is performing well. USDA may use findings and lessons learned to inform McGovern Dole programme funding, design, and implementation decisions; • MasterCard as a funder of the HGSF programme may use the findings to inform its decision on the best models of school feeding as well as to target its funding; • Given the core functions of the Regional Bureau Nairobi (RBN), it is expected to use the evaluation findings to provide strategic guidance, programme support, and oversight; • WFP HQ may use the evaluation for wider organizational learning and accountability; • OEV may use the evaluation findings, as appropriate, to feed into evaluation syntheses as well as for annual reporting to the Executive Board. 12. The primary users of the endline evaluation will be: • WFP Rwanda and its partners World Vision, Gardens for Health International, and Rwanda Biomedical Centre to learn from programme implementation; • MINEDUC and MINAGRI to learn whether the programme performed well and aligned with its priorities, particularly in terms of capacity development, handover and sustainability; • USDA as the primary funder of the HGSF Programme to learn whether the programme is performing well. USDA may use findings and lessons learned to inform McGovern Dole programme funding, design, and implementation decisions; • MasterCard as a funder of the HGSF programme may use the findings to inform its decision on the best models of school feeding as well as to target its funding; • Given the core functions of the Regional Bureau Nairobi (RBN), it is expected to use the evaluation findings to provide strategic guidance, programme support, and oversight; • WFP HQ may use the evaluation for wider organizational learning and accountability; • OEV may use the evaluation findings, as appropriate, to feed into evaluation syntheses as well as for annual reporting to the Executive Board. 3. Context and subject of the evaluations 3.1 Context 13. Rwanda is a densely populated, low-income country with a total population of about 12 million people, ranked 157 of 189 countries in human development (2019 UNHDI). Since the 1994 genocide, the Government of Rwanda has consistently and rather successfully pursued development objectives. Rwanda made significant progress in implementing the MDGs and is well placed to continue within the SDG framework (UNDAP 2013-18). An annual GDP growth rate of 7.2 percent since 2010 has been accompanied by decreasing income inequality, although still among the highest in Africa (UN CCA 2017). 14. Moreover, 41 percent of Rwandans are undernourished, and 38.2 percent of the population continues to live below the poverty line and almost one fifth is food insecure (SDG 2.1.1) (CFSVA 2018). Levels of stunting among young children remain very high (38 percent in the 2019 Global Nutrition Report for Rwanda). Stunting is more common among children with mothers who are young, did not complete secondary education, or are stunted themselves and have an unbalanced dietary intake. 12.6 percent of children under five are underweight (low weight for age) and 2 percent are acutely malnourished (low weight for height) (CFSVA 2018). Date | Report Number 103 15. Micronutrient deficiencies are also a public health concern; 37 percent of children under five and 19 percent of women of reproductive age are anemic. The most common causes of anemia are lack of iron in the diet, and intestinal worms, preventing the absorption of micronutrients and minerals such as iron. Worm infections affect 65 percent of the population in Rwanda, and school-aged children are particularly affected. Main drivers for malnutrition are poor access to quality water, health services and sanitation (WASH) as well as poor care practices such as not receiving antenatal care, even among those who can access a nutritional, balanced diet (UN CCA 2017). 16. At 98.5 percent, Rwanda’s primary enrolment rate is among the highest in sub-Saharan Africa. Girls and boys show very similar numbers, but equitable access is an issue among vulnerable populations such as children with disabilities. Worryingly, primary school completion has considerably dropped from 73 percent in 2012 to 65 percent (boys 59 percent, girls 71 percent) in 2016202, indicating a high dropout rate. The student-to-teacher ratio is high at 57:1, leaving little time for teachers to interact with students. The extent of limited delivery of quality education is evidenced by low competencies in literacy and numeracy among primary-school-going children. 17. The average primary school has one toilet for every 52 students. The national target is 40:1 for boys and 30:1 for girls. MINEDUC estimates that menstrual management alone accounts for an average of 50 days/girl/year in absences. While some steps have been taken to improve the situation, menstrual management continues to negatively affect girls, especially girls in the poorest districts (UN CCA 2017). In addition, only 54.1 percent of schools in Rwanda have access to piped tap water. 18. National literacy assessments in Rwanda have revealed generally poor reading skills among primary school students. In 2012, a Rwanda National Standards Committee defined third-grade Kinyarwanda reading fluency as 33-47 words correct per minute, yet, according to an Early Grade Reading Assessment (EGRA) conducted that year, 13 percent of students in primary grade 4 (P4), could not read a single word of a P2/3 level text in Kinyarwanda. Another 13 percent were reading fewer than 15 correct words per minute, far below what is necessary for comprehension. In 2014, a reading assessment conducted by the USAID-funded Literacy, Language, and Learning project found that 60 percent of P1 students, 33 percent of P2 students, and 21 percent of P3 students were unable to read a single word of grade level text. By 2017, research showed that 54.9 percent of primary grade 3 students and 56.4 percent of primary grade 6 students met standards (In Rwanda, 61 percent of adult women are literate whereas the figure is 39 percent for men.) (MINEDUC Annual Statsbook, 2019). 19. The social protection system has evolved in the last decade, moving from a host of fragmented, uncoordinated and often underfunded programmes to increasingly coordinated, government-owned programmes operating at scale. Thus, great strides were made regarding poverty reduction and vulnerability, mainly with households being the primary targeting unit (UN CCA 2017). The Social Protection Sector Strategy 2018-24 is taking this further by adopting a life-cycle approach and promoting universal access to social security and protection for all Rwandans, whether poor or not. Quite a radical shift, and appropriate given the country’s vision to reach upper income status by 2050. 20. Rwanda has made commendable progress in ensuring gender equality however glaring challenges still exist between males and females. While Rwanda has the highest percentage of women in parliament in the world (62 percent) and female representation in high also in other positions of power (55 percent of the cabinet, 57 percent of Supreme Court Justices), gender balance in local government leadership has not yet been fully addressed as most positions continue to be dominated by men, raising concerns around effective implementation of all gender equality related programmes. Similar observations and arguments suggest that there is a critical gap in the quality of overall participation and specifically women’s participation in these platforms as there is limited evidence to suggest otherwise. 21. The HGSF Programme supports students in four of Rwanda’s poorest and most food insecure districts: Nyaruguru and Nyamagabe in the southern province and Rutsiro and Karongi in the western province (see map in Annex 1). Nyaruguru and Nyamagabe districts have particularly high numbers of households led by women and people with disabilities, compounding vulnerability. The 107 programme schools were selected from sectors with the highest poverty levels per the government’s 202 From 2017 onward, MINEDUC has not reported on primary school completion rates. Date | Report Number 104 household poverty classification (UBUDEHE), also considering each school’s capacity and willingness to implement activities (WFP Rwanda 2015 MGD proposal). 22. WFP is implementing the HGSF Programme to support the government in setting up a national school feeding programme, with an initial view to build capacity and complete a full handover of activities by 2021. As the programme has evolved, WFP’s plans to handover the programme have shifted into building the capacity of schools further, to serve as model school feeding schools to eventually transition into the national school feeding programme during the next project phase. Since its establishment in 2017, MINEDUC and WFP co-chair the HGSF National Steering Committee which is bringing together key stakeholders to coordinate the programme and ensure sustainability. In 2020, MINEDUC also established the National School Feeding Steering Committee, to oversee all school feeding operations in the country. WFP will co-chair the Committee with MINEDUC. 23. The Government of Rwanda’s mid- to long-term outlook is guided by the national development plan Vision 2050, which envisions Rwanda transforming from an agrarian to a knowledge-based economy, attaining upper middle-income country status by 2035 and high-income status by 2050. To help achieve this, the country’s key poverty reduction strategy, the National Strategy for Transformation (NST), focuses on three pillars, of which the social transformation pillar has a priority area to ensure quality of education for all, aiming at building a knowledge-based economy (UN CCA 2017). 24. As the lead of the education sector, MINEDUC heads policy formulation, planning, coordination, regulation, monitoring and evaluation of the entire education sector (UN CCA 2017). The ministry works closely with the semi-autonomous Rwanda Education Board (REB) which provides national oversight for coordinating and implementing education activities at pre-primary, primary and secondary level. 25. Under the Local Government Act (2013) District Administrations have responsibility for the delivery of education services. The extent to which MINEDUC and REB have influence at the district level is determined by the level of interest and priority afforded to education by the District Executives. District Development Plans (DDPs) determine district priorities and where resources are allocated. District Education Officers (DEOs) are employed by the Ministry of Local Government (MINALOC) and managed by their District Administrations. DEOs are actively involved in the planning, delivery and monitoring of education in their districts. Below the DEOs are sector education officers (SEOs) responsible for overseeing delivery of education services and running of schools. 26. Guided by the Education Sector Strategic Plan III (ESSP), now in its third203 cycle, Rwanda has invested significant resources and made considerable progress over the past 25 years, towards improving the quality and coverage of all levels of education, as well as towards implementing policies that aim to achieve universal and equitable access to basic education for all Rwandan children. In 2012 the GoR committed to a full twelve-year cycle of free and compulsory basic education. Today, primary level enrolment is the highest in East Africa, at 98.3 percent, and having surpassed its ESSP 2014 enrolment target of 97 percent, Rwanda is close to achieving its goal of universal primary education. Gender parity in enrolment is nearly achieved and stable across pre-primary and primary levels. 27. The Soma Umenye Project (2016-2021), implemented by Chemonics, is a national early grade reading intervention with the aim to improve Kinyarwanda reading skills for primary grades 1-3 (1million children) in public and GoR-aided schools. Activities focus on classroom instruction techniques, teacher manuals, and pre-service teacher training programmes. USAID’s Mureke Dusome Project (2015-2020), implemented by Save the Children, is a nationwide early grade literacy project which aims to foster partnerships between schools and the broader community in a bid to improve Kinyarwanda literacy among primary students through community mobilization, and reading clubs. These USAID investments in child literacy are reinforced by the UK Department for International Development￾funded Building Learning Foundations project, implemented by Voluntary Service Overseas, which seeks to improve learning outcomes by enhancing quality of English and Math teaching in primary grades 1-3 in all public and GoR-aided primary schools through teacher development, leadership and systems strengthening. These interventions respond to Soma Umenye 2017 EGRA findings showing 203 The current ESSP III runs from 2018/2019-2023/2024 Date | Report Number 105 that without stronger foundations in literacy and numeracy, and expanded access to age-appropriate reading materials, learners are unlikely to see improved education outcomes. 28. In support of school health, the Ministry of Health (MINESANTE) has a ‘12+ Programme’, supported by the Nike Foundation and PSI, targeting 114,500 girls between the ages of 10‐12 years in primary schools across Rwanda, funded by DFID, with the objective of reducing drop‐out of girls by creating safe spaces in schools for girls. World Vision also has a cost‐sharing partnership with Rwanda’s Water and Sanitation Corporation (WASAC) through its Ubuzima WASH project partnering with MINESANTE, training Community Health Workers (CHWs) and school‐based volunteers through the Community‐ based Environmental Health Promotion Programme (CBEHPP). CBEHPP is MINESANTE’s approved methodology of working with communities to help them identify and solve their own health and hygiene issues. World Vision is Rwanda’s largest implementer of CBEHPP and outside of the MGD project is supporting MINESANTE in training 45,000 CHWs in 15,000 villages to reduce hygiene‐related diseases in communities and within schools (WFP Rwanda 2015 MGD proposal). 204 29. UNICEF has supported modelling and scaling‐up Child‐Friendly School standards, which were adopted as the national quality guidelines for school infrastructure and software inputs. The Learning Achievement in Rwandan Schools (LARS) Assessment was supported by UNICEF to improve the quality of education and measure learning outcomes in literacy and numeracy. The joint UN Development Assistance Programme (UNDAP) 2018‐2023 is focusing on increased and equitable access to quality education, health, nutrition and WASH services. 3.2. Subject of the evaluations 30. This is an activity evaluation of a USD 25 million grant to improve literacy and increase the use of health and dietary practices. The midterm evaluation was conducted in 2018, while the endline will be completed in 2020-21. The midterm evaluation covered the 2016-2018 (second quarter included) timeframe, while the endline evaluation will cover the entire implementation period (2016-2020205). 31. The HGSF Programme supports students across 107 schools in four of Rwanda’s poorest and most food insecure districts – Nyaruguru and Nyamagabe in the south and Rutsiro and Karongi in the west (see map in Annex 1). Children in the south receive a daily hot meal whereas students in the western province are provided a porridge meal. The programme started in October 2015 and will conclude in March 2021. 32. WFP has planned to reach 83,000 students annually in grades 1-6, and reached 99 percent in 2016 and 103 percent in 2017, with similar numbers of boys and girls. In 2018, WFP reached 101 percent of students, while in 2019 and 2020, student enrolment reduced206 to 98 percent and 94 percent respectively, resulting mostly from the establishment of new satellite schools to reduce overcrowding. It is important to highlight that in a country like Rwanda where national net enrolment measures 98.3 percent, increased enrolment is not the main focus of the programme but rather improved quality of education, nutrition, health and attendance. Over the programme cycle, WFP plans to use 4,657 metric tonnes of SuperCereal and 540 metric tonnes of vegetable oil. Additionally, maize meal, beans, salt and sugar are non-USDA commodities and as such purchased from other mobilized funds (WFP Rwanda 2015 MGD proposal). 33. The HGSF Programme aims to achieve the McGovern Dole strategic goals of improved literacy of school-age children (MGD SO1) and increased use of health and dietary practices (MGD SO2) in the targeted areas through a set of interconnected activities, with a view to supporting the government in establishing a national school feeding programme, including building capacity at national, district and 204 These programmes have since finished and WFP requests that the evaluation firm conducts a mapping of new/current WASH programmes in schools. 205 Implementation extended to March 31, 2021 and project agreement to June 31, 2021 to allow sufficient time to complete the endline evaluation. 206 Causes of the reduction as identified during the March 2020 remote data collection exercise include: i) the creation of satellite schools to reduce the number of students travelling long distances to reach school; ii) a shortage of classrooms for students following the Government’s transition of upper primary students from double to single shift days, resulting in the Government limiting grade 1 enrolment only to students who are 7 years old; iii) the expansion of some primary schools to include secondary students (groupe scolaire) influenced lower student enrolment numbers as more classes had to be dedicated to secondary students. Date | Report Number 106 school levels to facilitate a future handover (see results frame in annex). It is important to note that following the approval of the national school feeding policy in 2020, and the resulting increased budget allocation to scale up the government’s school feeding programmes to all pre-, primary and secondary students in the 2020/21 academic year, the previous focus on a handover of the programme to government has now shifted to an aim to transition the HGSF Programme into the National School Feeding Programme. 34. WFP, together with its implementing government partners MINEDUC and MINAGRI as well as key implementing NGO partners, is carrying out activities to achieve MGD SO1 by: promoting teacher attendance and recognition; distributing school supplies and materials; improving literacy instruction materials; increasing the skills and knowledge of teachers and administrators; providing school meals; developing partnerships with farmer groups to supply food to schools; establishing and maintaining school gardens; increasing use of health and dietary practices; raising awareness on the importance of education; and reducing health-related absenteeism at schools. 35. WFP and partners carry out activities to achieve MGD SO2 by: raising awareness on good hygiene practices; enhancing food preparation and cooking practices; building and rehabilitating latrines and water collection systems; distributing deworming medication; and building/rehabilitating kitchens, cooking areas and storerooms, and providing fuel-efficient stoves. 36. To ensure the development of a nationally owned, sustainable school feeding programme, WFP and partners carry out activities to achieve MGD foundational results by: increasing the capacity of government institutions (1.4.1/2.7.1), improving the policy and regulatory framework (1.4.2/2.7.2), and increasing government support (1.4.3/2.7.3). To increase engagement of local organizations and community groups (1.4.4/2.7.4), WFP and partners train PTAs, raise awareness on the importance of education, develop partnerships with farmer groups and local cooperatives to supply food to schools, engage parents and communities through the establishment and maintenance of school gardens, and strengthen school health clubs through training and awareness on good health and hygiene practices. 37. The McGovern Dole grant for the five-year programme is USD 25 million. WFP estimates that the total value of contributions mobilized for this project outside of McGovern Dole resources will be around USD 12.1 million. These resources have enabled the full implementation of the programme, including locally procuring maize meal and beans with funding from MasterCard. 38. The HGSF baseline study report from July 2016 focused on indicators that could be measured before project implementation. Some indicators could not be measured because their definition is linked to the implementation of project activities. Baseline values for each indicator measured against its corresponding target, as per the project document, were summarized (see baseline report in annex). Following the baseline study, the report continues, it is essential that WFP reviews and realigns the targets. In preparation for the midterm and endline evaluations, there is also a need for a strong programme monitoring component that collects and compiles data from each of the beneficiary school and related activities on a regular basis. All indicators, including policy-related indicators, require specific project records. 39. The centralized midterm evaluation of WFP Rwanda’s Country Programme (June 2017) noted that although the McGovern Dole grant provided much-needed funding for HGSF, it has also led to some unalignment with WFP as well as national priorities. The MGD intervention saw a return to providing (for part of the beneficiaries) food imported from the U.S. – a condition which was clearly required by the donor. This modality is in contradiction with the Government of Rwanda’s own expressed preference but also with the logic promoted by WFP’s School Feeding Policy of giving priority to helping countries establish and maintain nationally owned programmes linked to local agricultural production. However, WFP has sought to address this by designing the project in such a way that it will support the local production of vegetable oil and CSB+207 and contribute to the development of a national strategy that if successful will sustain the benefits of USDA support beyond the life of the project. The report says that it was too soon to measure sustainability/handover in 2016. 207 In 2020, the Government made it clear that school feeding programmes should move toward a design where all students receive a hot meal. Date | Report Number 107 40. The evaluation report concluded that HGSF activities are “completely” integrated in government planning and monitoring at local level, with officials participating in all steps of the process. The programme promotes the participation of girls, and the baseline allows monitoring against gender indicators as advised by the WFP 2009 Gender Policy (later subsumed by the WFP 2015-20 Gender Policy), the report continues. The programme does not, however, include a specific approach to address Gender Equality and Women’s Empowerment (GEWE) or Sexual and Gender-based Violence (GBV) as foreseen in the 2009 policy. 41. A cost-benefit analysis conducted by WFP in collaboration with MasterCard in November 2017 concluded that every dollar invested in school meals in Rwanda can generate a return of USD 4.8 and 5.6 for home-grown and in-kind modalities, respectively, over a child’s lifetime. Finally, the midterm and endline evaluations will be guided by the WFP Evaluation Policy 2016-2021 and the USDA Monitoring and Evaluation Policy 2019. 4. Evaluation approach 4.1 Scope 42. WFP Rwanda is looking to assign one contract for both a midterm and endline evaluation of the 2016- 2020 McGovern Dole grant. The programme started in late 2015 with the goal of supporting school feeding in 104 schools in four districts: Nyaruguru, Nyamagabe, Rutsiro and Karongi in Rwanda (see map in Annex 1). In 2020, WFP expanded the coverage of meals and a few select activities to 3 additional schools, making the total 107 supported. 43. This is an activity evaluation of a USD 25 million grant to improve literacy and increase the use of health and dietary practices. The programme provides U.S. produced agricultural commodities and financial assistance, and supports capacity development and enhanced monitoring and reporting, with a key emphasis on sustainability and government ownership. In addition to USDA-provided commodities, WFP procures maize meal and beans locally through additional funds raised separately through private donors, such as MasterCard. 44. The midterm evaluation took place in 2018, while the endline will be completed in 2020-21. The midterm evaluation covered the 2016-2018 (second quarter included) timeframe, while the endline evaluation will cover the entire implementation period (2016-2021). 45. The beneficiaries of the programme are 79,000 primary school students per year, grades 1-6. Of these, 49 percent are female. Other stakeholders who get access to capacity building activities are Parent-Teacher Associations (PTAs)208, School Management Committees (SMCs), teachers and head teachers, storekeepers and cooks. 4.2 Evaluation criteria and questions 46. Evaluation Criteria. The evaluations will address all five OECD-DAC criteria (relevance, effectiveness, efficiency, impact and sustainability), as per USDA Monitoring and Evaluation policy. 47. Evaluation Questions. Aligned with the evaluation criteria, the evaluations will address the following key questions, which will be further developed by the evaluation team during the inception phase. Collectively, the evaluation questions aim at highlighting the key lessons and performance of the HGSF Programme, which could inform future strategic and operational decisions. The four key questions that need to be investigated are: a. Have literacy rates of school age children improved over the duration of the programme? If so, how and why? For example, are students able to read grade-level text? Are teachers demonstrating new methods of teaching? 208 PTAs have now been changed to School General Assembly Committees (SGAC) and SMCs are now known as Senior Management Teams, which report to the SGAC. However, important to note that only the SGACs are mentioned in the law. Date | Report Number 108 b. Has the use of health and dietary practices increased? If so, how? Has illness-related absence decreased? Are students washing their hands and are schools and school kitchens clean? How are school gardens being used? c. What is the level of community-level involvement and participation in decision-making in school governance mechanisms (PTAs and SMCs)? Particularly, what is the level of involvement and participation of men? Also, what is the level and sustained continuity of community contributions in cash and in kind? d. What are the key institutions (i.e.. international, national, provincial/district and local stakeholders) and governance structures required to effectively deliver, implement, and sustain school meal interventions? What relationship structures among these institutions yield the most successful and effective school meal programmes? Is WFP’s capacity support to smallholder farmers and key line ministries appropriate/sufficient to effectively facilitate national ownership? Has the provided capacity support increased the government’s capacity to own and sustain a national school meals programme? 48. The evaluation questions will be reviewed within the course of the inception period, and there will be some differentiation between midterm and endline questions. For example, the midterm also included achievement of outputs rather than only outcomes and objectives, whereas the endline will focus more on impact. 49. Gender equality and empowerment of women should be mainstreamed throughout the evaluation, including disaggregation of all data and considering whether gender has been integrated in design, planning, implementation and results. Reflecting UNDAP concern on gender equality “Ensure women’s full and effective participation and equal opportunities for leadership at all levels of decision making in political, economic and public life”, the evaluators are specifically required to investigate gender aspects in question c. above. 50. Table 2 below presents key evaluation criteria and corresponding questions: Table 2: Criteria and evaluation questions Criteria Evaluation Questions Relevance To what extent is the programme in line with the needs of beneficiaries (boys and girls) and partners, including government? To what extent is the activity aligned with community, local government, and national government policies and priorities? To what extent is the intervention based on a sound gender analysis? To what extent is the design and implementation of the intervention gender-sensitive? Are the changes made to activities (design and implementation) due to Covid-19 relevant? Effectiveness To what extent are the outcomes or objectives of the intervention likely to be achieved? What are the major factors influencing progress in achievement or non-achievement of the outcomes/objectives of the intervention? To what extent does the intervention deliver results for boys and girls? Efficiency Is the programme implemented in a timely way? Are the activities cost-efficient? Is the programme implemented in the most efficient way compared to alternatives? Were the project strategies efficient in terms of financial and human resource inputs as compared to outputs? Date | Report Number 109 Does the monitoring system efficiently meet the needs and requirements of the project? Impact What are the medium-term effects on beneficiaries’ lives? What are the gender-specific medium term impacts? Did the intervention influence the gender context? To what extent did COVID-19 affect project implementation and performance? Sustainability To what extent is the government taking ownership of the programme (e.g., demonstrated commitment and contributions)? What is the demonstrated capacity at central and sub-national levels to manage the programme? Are local communities (PTAs, farmers’ groups, etc.) fully involved in and contributing toward school feeding and education activities? Has the policy framework supporting the HGSF been strengthened within the project period? What are the major factors influencing the achievement or non-achievement of sustainability of the program? 4.3 Data availability 51. The evaluations will entail qualitative and quantitative primary data collection that the evaluation team will be responsible for. In addition, the following is a list of background data and/or information available for the evaluation team. It is expected that the team will expand this at inception phase. • Baseline and midline reports for WFP’s USDA McGovern Dole HGSF Programme 2016-2020, including data collection tools; • School feeding handbook; • WFP School feeding policy; • 2016, 2017, and 2018 Standard Project Reports (SPRs); • 2019 ACR Report • WFP HGSF semi-annual reports to USDA; • USDA commitment letter for Agreement; • Evaluation Plan; • USDA Monitoring and Evaluation Policy; • USDA McGovern-Dole Indicators and Definitions Handbook; • Other government education data/policies as applicable. 52. The evaluation team responsible for the baseline assessment warned about the availability and usability of certain data sets. Following the recommendations of the baseline team, WFP and its partners have improved data collection tools to provide the level of granularity required by the donor and to answer most of the evaluation questions. For instance, school records now provide attendance information per individual child and teacher, records are revised monthly and are subject to random checks. However, during the inception phase, the evaluation team will be responsible for controlling the quality and reliability of data sets and formulate alternative strategies to fill potential data gaps. 53. The evaluation team is expected to explore key questions c. (gender) and d. (institutional preparedness for hand-over) largely through qualitative data (although some quantitative data on gender parity is also expected). Key question d. will require an analysis of similar experiences in other countries and a comparison with the situation in Rwanda. Date | Report Number 110 54. Even though, at this point, WFP does not envision the use of such data collection tools, the evaluation team should also bear in mind that the Government of Rwanda requires formal approval of household surveys three months before the field phase takes place. 55. Concerning data and information, the evaluation team should: • Assess availability, validity and reliability as part of the inception phase of the midterm evaluation expanding on the information provided in section 4. This assessment will inform the data collection; • Systematically check accuracy, consistency and validity of collected data and information and acknowledge any limitations/caveats in drawing conclusions using the data. 4.4 Methodology 56. The independent evaluation team is responsible for developing the full methodology during the inception phase. In this stage, the ET should validate the methodological approach followed during the baseline assessment and propose improvements where required. 57. Question a. (literacy) will be answered using data collected through the EGRA standard test. To conduct the EGRA and adapt it to the local context, the selected evaluator should reference the EGRA toolkit209. 58. Question b. (health and dietary practices), is likely to be based on data collected through school and student surveys, direct observation and key informant interviews. 59. Key questions c. and d. were not explicitly included in the baseline assessment and will require the team to develop an appropriate methodological approach at this stage. 60. Midterm findings on key question d.informed the development of a hand-over strategy from WFP to the Government of Rwanda. It is expected that the evaluation team formulates clear recommendations that could help a smooth transition to a country-owned home-grown school feeding programme. At endline, the evaluation team should concentrate on assessing the progress made in handing over the activities. 61. The endline design will replicate the full mid-term design with (1) appropriate adaptations/supplements to areas of inquiry to account for hypothesized impacts of COVID-19, and (2) context-specific data collection protocols. 62. The team should identify potential risks of the approach and mitigation measures. The following should be considered and included by the evaluation team: • Firstly, confirm and define specific evaluation questions that are answered, and record them in the WFP Evaluation Matrix; • Include description of sample categories and identify appropriate sample sizes (margin of error 5 percent, confidence level 95 percent); • Design credible data collection instruments; • Use mixed methods in the evaluation design and data collection (including quantitative and qualitative) to ensure a comprehensive design, and the reasons for the changes in indicators can be explained. This can include triangulation of information through a variety of means, or different evaluation questions being answered through different methods and types of data. The use of mixed methods should be documented in the inception report; • To the extent possible, ensure that data collection tools are consistent with baseline tools to ensure comparability; • WFP anticipates that the consultants will recommend a methodology that will likely include carrying out key informant interviews and focus group discussions (list of interviews to be agreed upon at inception phase). The qualitative data collection will gather information on gender equality, capacity strengthening and changes in the institutional context. However, bidding companies should also propose a wider variety of methods (including, but not limited to most significant change, outcome harvesting, etc.) whenever they feel these could be useful in enriching the evaluation products; 209 EGRA Toolkit, Second Edition. https://globalreadingnetwork.net/resources/early-grade-reading-assessment-egra￾toolkit-second-edition Date | Report Number 111 • Ensure the evaluation design considers ways to ensure that the voices of women, girls, men and boys are heard and documented; • Ensure the methodology and evaluation implementation are ethical and conform to the UNEG Ethical Guidelines for Evaluation. • In light of Covid-19, international consultants may face travel restrictions and quarantine measures to enter Rwanda. In addition, the evaluation team may be unable to conduct face-to-face data collection. To avoid any evaluation delays, national team members may need to lead on the primary data collection, supported by international team members remotely • COVID-19 influence on impact pathways and establishing attribution/contribution to measurable programme results will need to be addressed by a strong evaluation design., i.e., undertaking the EGRA that accounts for a dynamic education context and ensuring appropriate school, community and government engagement protocols across all areas of inquiry. 63. The following mechanisms for independence and impartiality are employed: • Appointment of an Evaluation Manager with no previous involvement with the HGSF programme (Daniel Svanlund, WFP M&E Officer); • Establishment of an Evaluation Committee; • Establishment of an Evaluation Reference Group. 64. The main risk identified that could affect the methodology proposed for the midterm and endline is the weakness of the baseline study (the sample size of students was small; some data sets had to be reconstructed due to the unavailability of reliable data, e.g., teachers’ attendance). As a mitigating measure, the evaluation team should highlight the strength of the evidence underlying the findings in the midterm and endline. 65. In addition, the CO will continuously assess whether the COVID-19 pandemic will affect the implementation of the evaluation. Explicit and systematic consideration of risks and benefits throughout the evaluation process is required. WFP has guidance and options available to support Country Offices. The set of norms and standards (including ethical) that guide the practice of evaluation in the UN, including Independency and Impartiality, will continue to guide the evaluation. If evaluations can no longer rely on field missions and face-to-face interviews, affecting methodology and data collection options, remote data collection will be considered. Decisions about evaluation data collection timing and approaches will aim to minimize exposure to risk (including of contracting COVID-19) for individuals (women, men, girls and boys), communities, WFP and partners’ employees who would be involved in the evaluation as interviewees/informants as well as national and international evaluators. 4.5 Quality assurance and assessment 66. WFP’s Decentralized Evaluation Quality Assurance System (DEQAS) defines the quality standards expected from this evaluation and sets out processes with in-built steps for Quality Assurance, Templates for evaluation products and Checklists for their review. DEQAS is closely aligned to the WFP’s evaluation quality assurance system (EQAS) and is based on the UNEG norms and standards and good practice of the international evaluation community and aims to ensure that the evaluation process and products conform to best practice. 67. DEQAS will be systematically applied to these evaluations. The WFP Evaluation Manager will be responsible for ensuring that the evaluations progress as per the DEQAS Process Guide and for conducting a rigorous quality control of the evaluations’ products ahead of their finalization. 68. WFP has developed a set of Quality Assurance Checklists for its decentralized evaluations. This includes Checklists for feedback on quality for each of the evaluation products. The relevant Checklist will be applied at each stage, to ensure the quality of the evaluation process and outputs. 69. To enhance the quality and credibility of these evaluations, an outsourced quality support (QS) service directly managed by WFP’s Office of Evaluation in Headquarter provides review of the draft inception and evaluation report (in addition to the same provided on draft TOR), and provide: Date | Report Number 112 a. systematic feedback from an evaluation perspective, on the quality of the draft inception/midterm/endline evaluation report; b. recommendations on how to improve the quality of the final inception/midterm/endline evaluation report. 70. The evaluation manager will review the feedback and recommendations from QS and share with the team leader, who is expected to use them to finalise the inception/ evaluation report. To ensure transparency and credibility of the process in line with the UNEG norms and standards[1], a rationale should be provided for any recommendations that the team does not consider when finalising the report. 71. This quality assurance process as outlined above does not interfere with the views and independence of the evaluation team, but ensures the report provides the necessary evidence in a clear and convincing way and draws its conclusions on that basis. 72. The evaluation team will be required to ensure the quality of data (validity, consistency and accuracy) throughout the analytical and reporting phases. The evaluation team should be assured of the accessibility of all relevant documentation within the provisions of the directive on disclosure of information. This is available in WFP’s Directive CP2010/001 on Information Disclosure. 73. All final evaluation reports will be subjected to a post hoc quality assessment by an independent entity through a process that is managed by OEV. The overall rating category of the reports will be made public alongside the evaluation reports. 5. Phases and deliverables 74. The evaluation will be conducted in two stages: a midterm evaluation was conducted between June/July and December 2018, and an endline evaluation that will take place between December 2020 and May 2021. Although the two phases are interconnected steps of the same evaluative exercise, their objectives are slightly different as outlined in the following sections. 75. The deliverables and deadlines for each phase are as follows: Figure 1: Summary Process Map 5.1 Midterm evaluation 76. The objective of the midterm evaluation was to provide an evidence-based, independent assessment of performance of the project so that WFP and its project partners can adjust course as necessary for the remainder of the project term. Specifically, the midterm evaluation allowed to (1) review the project’s relevance, effectiveness, efficiency, impact, and sustainability, (2) collect performance indicator data for strategic objectives and higher-level results, (3) assess whether the project is on track to meet the results and targets, (4) review the results frameworks and theory of change, and (5) identify any necessary mid-course corrections. The evaluation will rely on the Baseline Study for baseline data and critical context necessary to evaluate the project at interim. [1] UNEG Norm #7 states “that transparency is an essential element that establishes trust and builds confidence, enhances stakeholder ownership and increases public accountability” 1. Prepare 2. Inception •Inception Report 3. Midterm data collection • Aide memoire / debriefing PPT 4. Analyze data and Report • Midterm Evaluation Report 5. Endline data collection • Aide memoire /debriefing PPT 6. Analyze Data and Report • Final report 7. Disseminate and follow￾up Date | Report Number 113 77. The evaluation firm selected for this assignment developed the methodological approach following the indications provided in 4.2 Evaluation Questions and 4.4 Methodology. The evaluators also validated and revised the assumptions and risk analysis underlying the project design. 78. The main deliverables of the midterm evaluation were the following: • Inception report. It was written following WFP recommended template. The evaluators validated the methodology utilized in the baseline phase and/or proposed alternative methods to measure the same indicators. This meant setting out a full study design including what data was being collected and for what purpose, how sampling was done determined by the evaluation team), how the data was being analysed and triangulated. The inception report also included a data quality assurance plan, and how the evaluators managed and safeguarded ethics during the life of the evaluation. Annexed to the inception report, the evaluation team included a detailed work plan including, timeline and activities. • Midterm report, including a first draft, where the final approach, methodology and data collection tools were clearly recorded, including their limitations and mitigations measures. The report recorded all standard and custom indicator baseline and midterm values. • Clean data sets, including quantitative data sets in Excel, statistical software code, and transcripts and/or notes from focus group discussions and key informant interviews. • A PowerPoint presentation of main findings and conclusions to debrief WFP Rwanda management and other relevant staff were prepared and presented • 2-page brief containing findings, conclusions and recommendations. Table 3: Midterm evaluation deliverables Dates Deliverables June/July – August 2018 • Desk review of key project documents • Create a data quality assurance plan • Review baseline methodology; confirm and finalise evaluation questions and evaluation design and methodology (including sampling strategy), and draft an inception report for agreement • Finalize inception report with the inclusion of Evaluation Reference Group’s comments • Data collection instruments • Arrange field visits September – October 2018 • Conduct field visits • Conduct key stakeholder focus groups and key informant interviews and collect data with other suggested methods • Enter, clean, and analyse data October – early December 2018 • Draft midterm report • Seek Evaluation Reference Group’s comments on the draft midterm report • Finalize midterm report • Prepare and present a PowerPoint presentation of main findings and conclusions to debrief WFP Rwanda management and other relevant staff • Prepare and share a 2-page brief with key stakeholders 5.2 Endline evaluation 79. The objective of the final evaluation is to provide an evidence-based, independent assessment of the performance of the project to evaluate the project’s success, ensure accountability, and generate lessons learned. Specifically, the final evaluation will: (1) use the same methodology developed for the midterm to measure key indicators, (2) analyses data to compare results before and after the intervention, and (3) identify meaningful lessons learned that WFP, USDA, and other relevant stakeholders can apply to future Date | Report Number 114 programming. The endline evaluation will also allow to unpack the COVID-19 influence on impact pathways on programme results. 80. WFP will carry out the final evaluation during the final year of the USDA-MGD grant between December 2020 and May 2021. 81. The main deliverables of the endline are the following: • Inception report. It must be written following WFP recommended template. The evaluators must validate the methodology utilized in the baseline and midterm phases and/or propose alternative methods to measure the same indicators. This means setting out a full study design including what data is being collected and for what purpose, how sampling is done (to be determined by the evaluation team), how the data is being analysed and triangulated. The inception report must also include a data quality assurance plan, and how the evaluators will manage and safeguard ethics during the life of the evaluation. Annexed to the inception report, the evaluation team should include a detailed work plan including, timeline and activities. • Endline report, including a first draft, using WFP recommended template. It must set out a detailed methodology section, study design, and any limitations or where the study design was compromised. Should detail how data was collected, validated and analysed, and how conclusions were drawn. How different types of methods were brought together in the analysis. Annexes to the final report include but are not limited to a copy of the final ToR, bibliography, detailed sampling methodology, maps, a list of all meetings and participants, final survey instruments, table of all standard and custom indicator with baseline, midterm and endline values, list of supported schools. • Clean data sets, including quantitative data sets in Excel, statistical software code, and transcripts and/or notes from focus group discussions and key informant interviews. • PowerPoint presentation of main findings and conclusions for de-briefing and purposes. • 2-4 page brief containing findings, conclusions and recommendations written for a nontechnical audience that includes photos and graphs or charts. • Conduct a 1-day workshop to share evaluation findings with key stakeholders. Date | Report Number 115 Table 4: Endline evaluation deliverables Dates Deliverables December 2020 • Finalize updated evaluation questions and evaluation design and methodology (including sampling strategy), and draft an inception report for agreement • Finalize inception report with the inclusion of Evaluation Reference Group’s comments • Update data collection instruments • Arrange field visits January – February 2021 • Conduct field visits • Conduct key stakeholder focus groups and key informant interviews and collect data with other suggested methods • Enter, clean, and analyse data March – early April 2021 • Draft endline report • Seek Evaluation Reference Group’s comments on the draft endline report • Finalize endline report • Prepare and present a PowerPoint presentation of main findings and conclusions to debrief WFP Rwanda management and other relevant staff • Prepare and share a 2-4 page brief with key stakeholders • Conduct a 1-day workshop to share evaluation findings with key stakeholders 6. Organization of the evaluation 6.1 Evaluation conduct 82. The evaluation team will conduct the evaluation under the direction of its team leader and in close communication with the WFP Evaluation Manager and School Feeding Programme Manager. The team will be hired following agreement with WFP on its composition. All communication between the evaluation team and stakeholders should go through or include the evaluation manager and School Feeding Programme Manager. 83. The evaluation team will draw its own conclusions free from political influence or organization pressure. The evaluation team will not have been involved in the design or implementation of the subject of evaluation or have any other conflicts of interest. Further, they will act impartially and respect the code of conduct of the evaluation profession. The Evaluation Manager has not been part of the project’s implementation. 84. Specifically, evaluators shall respect and protect the rights and welfare of human subjects and communities, in accordance with the UN Universal Declaration of Human Rights and other human rights conventions. Evaluators shall respect differences in culture, local customs, religious beliefs and practices, personal interaction, gender roles, disability, age and ethnicity, while using evaluation instruments appropriate to the cultural setting. Evaluators shall ensure prospective participants are treated as autonomous agents, free to choose whether to participate in the evaluation, while ensuring that the relatively powerless are represented. Evaluators shall make themselves aware of and comply with legal codes (whether international or national) governing, for example, interviewing children and young people. 85. The evaluation team should also guarantee the right to provide information in confidence and make participants aware of the scope and limits of confidentiality, while ensuring that sensitive information cannot be traced to its source (right to confidentiality). 86. Evaluators shall act to minimise risks and harms to, and burdens on, those participating in the evaluation, without compromising the integrity of the evaluation findings (avoidance of harm). 6.2 Team composition and competencies 87. The Team Leader should be a senior evaluator with at least 15 years of experience in research and/or evaluation with demonstrated expertise in managing multidisciplinary and mixed quantitative and qualitative Date | Report Number 116 method evaluations, and additional significant experience in other development and management positions. In addition, the team leader should also have prior experience evaluating school meals programmes, ideally USDA-funded McGovern-Dole grants. 88. The Team leader will also have expertise in designing methodology and data collection tools. She/he will also have leadership and communication skills, including a track record of excellent writing and presentation skills. Her/his primary responsibilities will be: i) design the approach and methodology; ii) guiding and managing the team; iii) leading the field missions and representing the baseline team; iv) drafting and revising, as required, the inception report, the end of field work i.e. (exit) debriefing presentation midterm and endline reports. 89. The team must include strong demonstrated knowledge of qualitative and quantitative data and statistical analysis will be required. It should include both women and men and at least one team member should be familiar with WFP’s operations (preferably school feeding). 90. The team will be multi-disciplinary and include members who together include an appropriate balance of expertise and practical knowledge in the following areas: • School Feeding; • WASH; • Primary Education (with a strong knowledge of early primary school reading process); • Food security; • Gender expertise; • Some expertise in smallholder farmer support is desirable; • All team members should have strong analytical and communication skills, evaluation experience and some familiarity with Rwanda; • The team should have knowledge of English and Kinyarwanda. The required language of both the midterm and endline reports is English. 91. Team members will bring together a complementary combination of the technical expertise required and have a track record of written work on similar assignments. 92. Team members will: i) contribute to the methodology in their area of expertise based on a document review; ii) conduct field work; iii) participate in team meetings and meetings with stakeholders; iv) contribute to the drafting and revision of the evaluation products in their technical area(s). 93. Considering the COVID-19 travel restrictions, it is strongly suggested to prioritize strong national team members supported by international team members remotely. 6.3 Security considerations 94. Security clearance where required is to be obtained. • As an ‘independent supplier’ of evaluation services to WFP, the evaluation company is responsible for ensuring the security of all persons contracted, including adequate arrangements for evacuation for medical or situational reasons. The consultants contracted by the evaluation company do not fall under the UN Department of Safety & Security (UNDSS) system for UN personnel. • Consultants hired independently are covered by the UN Department of Safety & Security (UNDSS) system for UN personnel which cover WFP staff and consultants contracted directly by WFP. Independent consultants must obtain UNDSS security clearance for travelling to be obtained from designated duty station and complete the UN system’s Basic and Advance Security in the Field courses in advance, print out their certificates and take them with them.210 95. However, to avoid any security incidents, the WFP Rwanda Evaluation Manager is requested to facilitate that: • The WFP CO registers the team members with the Security Officer on arrival in country and arranges a security briefing for them to gain an understanding of the security situation on the ground. 210 Field Courses: Basic; Advanced Date | Report Number 117 • The team members observe applicable UN security rules and regulations – e.g., curfews, child protection protocols. 7. Roles and responsibilities of stakeholders a- WFP Rwanda: The WFP Rwanda Deputy Director will take responsibility to: • Assign an Evaluation Manager for the evaluation: Sarah Cruz, M&E Officer. • Compose the internal Evaluation Committee and the Evaluation Reference Group (see below). • Approve the final TOR, inception and evaluation reports. • Ensure the independence and impartiality of the evaluation at all stages, including establishment of an Evaluation Committee and of a Reference Group (see below and TN on Independence and Impartiality). • Participate in discussions on the evaluation design and subject, its performance and results with the Evaluation Manager and the evaluation team. • Participate in debriefing(s). • Oversee dissemination and follow-up processes, including the preparation of a Management Response to the evaluation recommendations. The Evaluation Manager: • Manages the evaluation process through all phases including drafting this TOR. • Ensures quality assurance mechanisms are operational. • Consolidates and shares comments on draft TOR, inception and evaluation reports with the evaluation team. • Ensures expected use of quality assurance mechanisms (checklists, quality support). • Ensures that the team has access to all documentation and information necessary to the evaluations; facilitates the team’s contacts with local stakeholders; sets up meetings, field visits; and provides logistic support during the fieldwork. • Organises security briefings for the evaluation team and provides any materials as required. b- An internal Evaluation Committee has been formed as part of ensuring the independence and impartiality of the evaluation (see EC TOR in annex). c- USDA, as the main funder of the programme, will be given the opportunity to review and comment on the key evaluation products. USDA will also participate in the evaluation as a key informant prior to the start of in-country fieldwork. d- The Regional Bureau: When not the Commissioning Office, the RB will take responsibility to: • Advise the Evaluation Manager and provide support to the evaluation process where appropriate. • Participate in discussions with the evaluation team on the evaluation design and on the evaluation subject as relevant, as required. • Provide comments on the draft TOR, Inception and Evaluation reports • Support the Management Response to the evaluation and track the implementation of the recommendations. • While the Regional Evaluation Officer Nikki Zimmerman will perform most of the above responsibilities, other RB relevant technical staff may participate in the Evaluation Reference Group and/or comment on evaluation products as appropriate. e- Relevant WFP Headquarters divisions will take responsibility to: • Discuss WFP strategies, policies or systems in their area of responsibility and subject of evaluation. • Comment on the evaluation TOR, inception and evaluation reports, as required. f- Other stakeholders (Government, NGOs) will review and comment on key evaluation deliverables as experts in an advisory capacity (see ERG TOR in annex). g- The Office of Evaluation (OEV). OEV, through the Regional Evaluation Officer, will advise the Evaluation Manager and provide support to the evaluation process when required. It is responsible for providing access Date | Report Number 118 to the outsourced quality support service reviewing draft ToR, inception and evaluation reports from an evaluation perspective. It also ensures a help desk function upon request. 8. Communication and budget 8.1 Communication 96. To ensure a smooth and efficient process and enhance the learning from these evaluations, the evaluation team should place emphasis on transparent and open communication with key stakeholders. This will be achieved by ensuring a clear agreement on channels and frequency of communication with and between key stakeholders. All communication between the evaluation team and stakeholders should go through or include the evaluation manager and School Feeding Programme Manager. 97. As part of the international standards for evaluation, WFP requires that all evaluations are made publicly available. Following the final approval of the evaluation reports, findings and recommendations shall be shared in various ways, including through discussions with WFP senior management and staff (primarily to enhance strategic and operational aspects) as well as with key partners including USDA, MINEDUC and MINAGRI, as well as World Vision, Gardens for Health International and relevant UN agencies. WFP will publish both the reports and the management responses. The published versions of the evaluation reports must be free from proprietary and personal identifying information. 98. The evaluation team is requested to prepare and present PowerPoint presentations (to debrief WFP management to inform strategic and operational decision-making) and 2-page briefs on the midterm and endline reports, both of which will be published on the WFP website and shared with national stakeholders and other WFP country offices currently receiving USDA grants. The evaluation team will also organize a one￾day workshop at WFP Rwanda premises to share the findings and recommendations from the endline evaluation with key stakeholders. WFP and the evaluation team may discuss further the detailed communication/dissemination plan. 8.2 Budget 99. For this evaluation, the budget will: • Be based on procurement through Long-term Agreements. Rates are guided by pre-agreed rates. • As detailed in the project’s Evaluation Plan, the total combined budget for this contract will not exceed USD 300,000. • Travel/subsistence/other direct expenses should be accounted for as instructed in the WFP budget template. Please send any queries to the Evaluation Manager, Sarah Cruz, M&E Officer, WFP Rwanda at sarah.cruz@wfp.org. Date | Report Number 119 Annex 8: Data Collection Tools As described in Annex 13 H, the data collection tools presented here were accompanied by corresponding excel data entry and topline analysis matrices WFP COUNTRY OFFICE Illustrative list of interviewees: McGovern-Dole Programme Team, M&E Team, Health/ Nutrition Officer, Admin/Finance Officer, Logistics Team, Gender Focal Point, Deputy Country Director 1. How has MINEDUC collaborated with the McGovern-Dole Program? 2. To what extent are HGSF activities aligned with national education policy? Has the WFP HGSF work fed into national policy development? 3. How was MINEDUC involved in the design of HGSF activities? Is the design relevant and realistic? 4. How have gender issues been taken into account in the program? Is this approach appropriate and effective? 5. What is your assessment of the performance of the WFP Rwanda country office and sub-national offices? 6. What are the communication and information-sharing processes between WFP and MINEDUC? How effective are these processes for coordination and decision-making? 7. Extent to which WFP has considered and addressed national and local capacity opportunities and constraints (probe: design phase, implementation, capacity development/handover plan)? 8. What analysis has been done regarding the cost-efficiency and cost-effectiveness of the program? What are your impressions of cost-efficiency and cost-effectiveness? 9. What capacity development support has WFP provided? Relevance and quality of support? Examples of changes as a result of that support? 10. Strengths of the McGovern-Dole Programme to date? What has worked well? 11. Constraints/ challenges of the McGovern-Dole Programme to date? How have these been addressed? 12. What factors have influenced achievement/ non-achievement of results? 13. Are the resources and expertise WFP has mobilized, adequate to implement HGSF? 14. Do you see any further opportunities for cooperation with Governmental and non-Governmental partners (e.g., in health, education, gender equity)? 15. To what extent has WFP supported partnerships with the private sector to address sustainability? 16. What are MINEDUC’s needs and concerns around sustainability and full Government ownership of the McGovern-Dole Program? 17. What should be the priorities for the remainder of the programme period? Suggestions on how to improve the program? MINISTRY OF EDUCATION – NATIONAL LEVEL Topical outline may also be used for donor interviews 1. What do you see as the purpose of the McGovern-Dole Program? 2. What changes have you noted in the schools or students since the project started? Differences between boys and girls? 3. Describe any positive or negative impact of the programme on: a. the school? Date | Report Number 120 b. wider community? c. the people who prepare the meals? d. local farmers/producers? 4. How/to what extent does the McGovern-Dole Programme complement other educational and social protection initiatives? 5. What capacity development support has WFP provided? Relevance and quality of support? Examples of changes as a result of that support? 6. What are the communication and information-sharing processes between WFP and MINEDUC at sub-national level? How effective are these processes for coordination and decision-making? 7. Strengths of the McGovern-Dole Programme to date? What has worked well? 8. Constraints/ challenges of the McGovern-Dole Programme to date? (probe: logistics, human resources, capacity)? How have these been addressed? 9. What factors have influenced achievement/ non-achievement of results? 10. How have gender issues been taken into account in the program? Is this approach appropriate and effective? 11. What should be the priorities for the remainder of the programme period? Suggestions on how to improve the program? MINISTRY OF EDUCATION – SUB-NATIONAL LEVEL General Questions 1. How has your institution collaborated with the McGovern-Dole Program? 2. How did the design of HGSF activities take place? What was your institution’s involvement? Is the design relevant and realistic? 3. To what extent are HGSF activities aligned with the policies and priorities of your institution? 4. What is your assessment of the performance of the WFP Rwanda country office and sub-national offices? 5. What are the communication and information-sharing processes between WFP and your institution? How effective are these processes for coordination and decision-making? 6. What are your institution’s needs and concerns around sustainability and full Government ownership of the McGovern-Dole Program? 7. Are activities by other partners or other agencies sufficient to complement the McGovern-Dole Programme to enhance sustainability? What additional partnerships could be explored? 8. Strengths of the McGovern-Dole Programme to date? What has worked well? 9. Constraints/ challenges of the McGovern-Dole Programme to date? How have these been addressed? 10. What factors have influenced achievement/ non-achievement of results? 11. What should be the priorities for the remainder of the programme period? Suggestions on how to improve the program? Date | Report Number 121 Ministry-specific Topics Ministry Additional/Specific Line of Inquiry Ministry of Agriculture (MINAGRI) --Alignment with PSTA4 --Role the ministry plays with smallholder farmers Ministry of Infrastructure (MININFRA) Ministry of Local Affairs (MINALOC) Seek MINALOC perspective on how the implementation of the programme is involving the local authorities as key stakeholders Ministry of Gender and Family Promotion (MIGEPROF) --Alignment of HGSF with nutrition guideline/ policy/ strategy -- How have gender issues been taken into account in the program? Is this approach appropriate and effective? IMPLEMENTING PARTNERS, UNICEF Interviewees: World Vision, Gardens for Health International, Rwanda Biomedical Centre, UNICEF211 1. How has your organization collaborated with the McGovern-Dole Program? 2. How was your organization involved in the design of HGSF activities? Is the design relevant and realistic? 3. What are the communication and information-sharing processes between WFP and your institution? How effective are these processes for coordination and decision-making? 4. What factors have influenced the performance of the collaboration during this period (positively or negatively)? 5. What additional opportunities exist for collaboration/synergies with your own organization? 6. Do you see any further opportunities for cooperation with Governmental and non-Governmental partners (e.g., in health, education, gender equity)? 7. What analysis has been done regarding the cost-efficiency and cost-effectiveness of the program? What are your impressions of cost-efficiency and cost-effectiveness? 8. Strengths of the McGovern-Dole Programme to date? What has worked well? 9. Constraints/ challenges of the McGovern-Dole Programme to date? How have these been addressed? 10. How have gender issues been taken into account in the program? Is this approach appropriate and effective? 11. What should be the priorities for the remainder of the programme period? Suggestions on how to improve the program? SCHOOL ADMINISTRATION AND TEACHERS Illustrative list of interviewees: Administrator, head teachers/ headmaster, teachers 1. What do you see as the purpose of the McGovern-Dole Program? 2. What records are kept on (1) administration of meals and (2) student attendance? (ask to see records; check if gender-disaggregated) 3. What changes have you noted in the school or students since the project started? Differences between boys and girls? 4. Describe any positive or negative impact of the programme on: 211 UNICEF supported modelling and scaling up Child‐Friendly School standards, which were adopted as the national quality guidelines for school infrastructure and software inputs. UNICEF also supported the Learning Achievement in Rwandan Schools (LARS) Assessment to improve the quality of education and measure learning outcomes in literacy and numeracy. Date | Report Number 122 a. the school? b. wider community? c. the people who prepare the meals? d. local farmers/producers? 5. Strengths of the McGovern-Dole Programme to date? What has worked well? (probe: logistics, relationship with the community) 6. Constraints/ challenges of the McGovern-Dole Programme to date? How have these been addressed? 7. What factors have influenced achievement/ non-achievement of results? 8. Have deliveries during the last year been regular and complete (all items received in the requested amounts)? 9. Has the school had to interrupt feeding for any extended periods during the last year? Why? 10. Does the school follow the official ration scale and number and timing of meals per day? If not, why not? 11. Are there any other health activities in the school (deworming, malaria prevention)? Who implements them? Quality and effectiveness? 12. Suggestions on how to improve the program? SCHOOL GENERAL ASSEMBLY COMMITTEES (SGACS) AND SCHOOL MANAGEMENT COMMITTEES (SMCS) 1. What do you see as the purpose of the McGovern-Dole Program? 2. What is the role of the [SGAC/SMC] in the program? 3. Do parents contribute to the meals (probe: fresh vegetables, condiments, fuel; level of participation, difficulties) 4. What changes have you seen in the students since the project started? Differences between boys and girls? 5. Describe any positive or negative impact of the programme on: a. the school? b. wider community? c. the people who prepare the meals? d. local farmers/producers? 6. What has worked well? 7. Constraints and challenges? How have these been addressed? 8. Suggestions on how to improve the program? COOKS 1. What do you see as the purpose of the McGovern-Dole Program? 2. What is your role in the program? 3. What changes have you noted in the school or students since the project started? Differences between boys and girls? 4. Is the food sufficient? 5. Do parents contribute to the meals (fresh vegetables, condiments, fuel, work)? 6. Who decides on the composition of the meals? Date | Report Number 123 7. Have deliveries during the last year been regular and complete (all items received in the requested amounts)? If not, why not? 8. Has the school had to interrupt feeding for any extended periods during the last year? Why? 9. Does the school follow the official ration scale and number and timing of meals per day? If not, why not? 10. Has the initiative had an impact on your lives? Has it affected how you are seen/ treated by the school or by the community? Describe. 11. Suggestions on how to improve the program? STUDENTS (GRADE 5 AND HIGHER) 1. What do you like most about the school meals? 2. What do you like least about the school meals? 3. Has the McGovern-Dole Programme changed anything for you? (probe: concentration, attendance, one extra meal a day, one less meal at home per day, extra burden of in-kind or cash contribution) 4. On the days that there are no meals, do you still come to school? Do other children come to school? 5. If you could change something about the school meals, what would that be? FARMER GROUPS, LOCAL COOPERATIVES, STOREKEEPERS 1. What do you see as the purpose of the McGovern-Dole Program? 2. What is your role in the program? 3. Benefits/advantages to your of participating in the programme (probe: benefits to group/cooperative; individual/personal benefits)? What has worked well? 4. Constraints/challenges of participating in the program? How have these been addressed? 5. What training or support have you received from WFP (or partners)? Relevance and quality of training/support? Examples of changes you’ve made as a result of that support? 6. Do you plan to continue participating in the McGovern-Dole Program? Why or why not? 7. Suggestions on how to improve the program? Date | Report Number 124 Annex 9: Evaluation Matrix Note: As noted in Section 1.4, the evaluation matrix has been reorganized since the inception phase of this endline evaluation to more clearly harmonize the interrelationships of the OECD-DAC criteria and the key evaluation questions. Where questions have been relocated, the reference to the question number in the inception report (IR) is noted in parentheses at the end of the question. OECD-DAC CRITERIA 1: RELEVANCE Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria To what extent are the objectives, targeting, activities and transfer modalities … OECD 1.1 …coherent with/relevant to the needs of the food-insecure population, and have they remained relevant? (Consider the distinct needs of women, men, boys and girls from different groups and geographical areas.) … (1.1 in IR matrix) OECD 1.2 …in line with the needs of beneficiaries (boys and girls)? … (1.1 in IR matrix) OECD 1.3 …based on a sound gender analysis? … (1.1 in IR matrix) OECD 1.4 To what extent is the design and implementation of the intervention gender-sensitive? (1.1 in IR matrix) OECD 1.5 *(new in endline) Are the changes made to activities (design and implementation) due to COVID-19 relevant to beneficiaries, global, country, 1) Level of access to education and retention at programme design stage and over time 2) Prevalence of poverty and food insecurity by district 3) Indications of the role that school feeding can play in improving access to education 4) Extent to which beneficiary and geographical targeting are in line with the spatial pattern of food insecurity and address inclusion / exclusion error 5) Coherence with recommendations from past assessments, analyses, evaluations 6) Extent to which problem analysis and previous assessments were used to guide programme design 7) Extent that analysis was done to identify (1) differentiated needs of targeted groups and (2) objectives and components designed to respond to those needs 8) Extent to which local stakeholders were consulted and involved in programme design 9) Level of ownership and responsibility accorded to stakeholders at programme design stage Assessments, analyses of the target population (e.g., baseline, midterm quant survey, outside assessments) Perspectives of target population (via FGDs, KIs) KIs with gov’t ministries (national and sub-national levels) and partners National policies/strategies UN and WFP corporate policies Documentation of gov’t, donor, and partner strategies and programmes Compare programme design and results framework against recommendations of past assessments, analyses, evaluations Compare perspectives among KIs (gov’t nat’l and sub-nat’l level], NGO partners, WFP staff, UN partners, Relevance Sustainability Date | Report Number 125 and partner/institution needs, policies, and priorities? … (1.1 in IR matrix) To what extent are the objectives, targeting, activities and transfer modalities… OECD 1.6 … coherent with/relevant to national policies and strategies, including education, food security, nutrition, and gender? (1.2 in IR matrix) OECD 1.7 …complementary to the policies, priorities, and interventions of donors, relevant government entities, and NGO partners? (1.3 in IR matrix) OECD 1.8 …in line with the needs of partners, including Government? (1.3 in IR matrix) OECD 1.9 …coherent with/relevant to relevant WFP and UN -wide system strategies, policies and normative guidance? (1.4 in IR matrix) OECD 1.10 To what extent is the overall approach (combining direct assistance and capacity development) to support the transition toward full gov’t management and implementation of a national school feeding programme, appropriate? (1.5 in IR matrix) 10) Analysis of programme results framework 11) Alignment of objectives, targeting, activities and transfer modalities with relevant national policies/strategies 12) Extent to which the handover strategy is in line with the conditions/constraints in terms of human and financial capacity at the moment that the project was conceived 13) Extent to which lessons from other school feeding handover situations were taken into account in programme design 14) Coherence of the objectives and activities with those of partners; synergies with other projects and with related sectors (e.g., health) 15) Degree of consultation between WFP and other actors in relevant areas 16) Degree of understanding of the project demonstrated by other actors who work in the same area 17) Existence of memoranda of understanding and their relevance to programme objectives/activities 18) Coherence of the objectives and activities with WFP and UN -wide strategies, policies, and normative guidance, e.g., regarding education/school feeding, food security, nutrition, capacity development, gender 19) Appropriateness and extent to which WFP has supported the gov’t in selecting a school feeding model that best fits with Government institutions, processes and policy frameworks structures and policies 20) Quality of the capacity analysis that took place at the time of project design and degree to which the needs of different Programme documents, incl. results framework Budget documents Date | Report Number 126 institutions/key actors was taken into account 21) Coherence between the identified needs, budgeting in the project document, and actual expenditure in areas related to capacity development and handover 22) Degree to which issues around financial sustainability were taken into account in the design and implementation of the project 23) Extent to which a cost analysis approach was employed at the moment when decisions were taken about project transitioning and the extent to which this has continued to be considered 24) Extent to which the technical capacity needs of WFP were taken into account to adequately support a transition process, and were reviewed/adjusted over time Date | Report Number 127 OECD-DAC CRITERIA 2: EFFECTIVENESS Key EQ 1: Have literacy rates of school-age children improved over the duration of the program? Elaboration per TOR: If so, how and why? For example, are students able to read grade-level text? Are teachers demonstrating new methods of teaching? Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 2/ KEQ 1.1 What is the level of attainment of planned results and performance? (gender-disaggregated, as data permit) (2.1 in IR matrix) OECD 2/ KEQ 1.2 Have literacy rates of school-age children improved over the duration of the program? If so, how and why? (2.1 in IR matrix) OECD 2/ KEQ 1.3 Are students able to read grade-level text? (2.1 in original matrix) OECD 2/ KEQ 1.4 Are teachers demonstrating new methods of teaching? (2.1 in IR matrix) 25) Comparison of endline survey data against baseline or midline, as appropriate; focus on EGRA indicators 26) Attention to gender-disaggregated data, where available Baseline, midterm and endline data (EGRA and school survey) Observation Interviews with informants for survey modules Follow-up all self-reported school survey data with in-person interviews to elaborate data and conduct spot checks against school-level documentation Effectiveness Impact Key EQ 2: Has the use of health and dietary practices increased? Elaboration per TOR: If so, how? Has illness-related absence decreased? Are students washing their hands? Are schools and school kitchens clean? How are school gardens being used? Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 2/ KEQ 2.1 What is the level of attainment of planned results and performance? (gender￾disaggregated, as data permit)? (2.1 in IR matrix) OECD 2/ KEQ 2.2 Has the use of health and dietary practices increased? If so, how? (2.1 in IR matrix) OECD 2/ KEQ 2.3 Has illness-related absence decreased? (2.1 in IR matrix) 27) Comparison of endline survey data against baseline or midterm, as appropriate; focus on McGovern￾Dole indicators 28) Attention to gender-disaggregated data, where available Baseline data (EGRA and school survey) Midterm and endline data (EGRA and school survey) (incl. data on health and dietary practices) Observation Follow-up all self-reported endline school survey data with in-person interviews to elaborate data and conduct spot checks against school￾level documentation Effectiveness Impact Date | Report Number 128 OECD 2/ KEQ 2.4 Are students washing their hands? (2.1 in IR matrix) OECD 2/ KEQ 2.5 Are schools and school kitchens clean? (2.1 in IR matrix) OECD 2/ KEQ 2.6 How are school gardens being used? (2.1 in IR matrix) Interviews with informants for survey modules Key EQ 3: What is the level of community-level involvement and participation in decision-making in school governance mechanisms Elaboration per TOR: Particularly, what is the level of involvement and participation of women? Also, what is the level and sustained continuity of community contributions in cash and in kind? Note: School governance mechanisms include School General Assembly Committees (SGACs) and School Management Committees (SMCs). Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 2/ KEQ 3.1 What is the level of community￾level involvement and participation in decision￾making in SGACs and SMCs? (2.2 in IR matrix) OECD 2/ KEQ 3.2 Particularly, what is the level of involvement and participation of women? (2.2 in IR matrix) OECD 2/ KEQ 3.3 What is the capacity and willingness for community/parent contributions in cash and in kind? (2.2 in IR matrix) 29) Qualitative analysis of SGAC and SMC structure, composition, and processes 30) Qualitative analysis of the influence of SGACs and SMCs on school feeding policies and practices 31) Extent of involvement and participation of women and men; qualitative analysis of any gender dynamics that affect representation and decision-making Focus groups with SGACs and SMCs KIs with school-level actors and other stakeholders who have first-hand knowledge of this issue Observation Compare KII findings, FGD findings, and observation Effectiveness Impact Sustainability OECD 2/ KEQ 3.4 To what extent has the programme addressed gender equality and protection issues? (2.3 in IR matrix) OECD 2/ KEQ 3.5 What are the gender-specific medium-term impacts? Did the intervention influence the gender context? (2.3 in IR matrix) OECD 2/ KEQ 3.6 To what extent has the programme influenced gender equality and protection? (2.3 in IR matrix) 32) Qualitative analysis of WFP, gov’t, UNICEF, and partner perspectives on how the programme has addressed gender and protection 33) Focus groups with students, SGACs, SMCs on gender and protection issues that affect them and how the programme has handled them 34) KIIs with stakeholders with knowledge of how the programme has addressed gender and protection Baseline, midterm and endline data (EGRA and school survey) KIIs and FGDs Observation Compare WFP perspectives on gender and protection with those of programme participants and non-WFP stakeholders Effectiveness Impact Sustainability Date | Report Number 129 Key EQ 4: What are the key institutions and governance structures required to effectively deliver, implement, and sustain school meal interventions? Elaboration per TOR: What relationship structures among these institutions yield the most successful and effective school meal programmes? Is WFP’s capacity support to smallholder farmers and key line ministries appropriate/sufficient to effectively facilitate national ownership? Has the provided capacity support increased the Government’s capacity to own and sustain a national school meals program? Question and Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 2/ KEQ 4.1 What are the key institutions (i.e., international, national, provincial/district and local) and governance structures required to effectively deliver, implement, and sustain school meal interventions? (3.2 in IR matrix) OECD 2/ KEQ 4.2 What relationship structures among these institutions yield the most successful and effective school meal programmes? (3.2 in IR matrix) OECD 2/ KEQ 4.3 Has the provided capacity support increased the Government’s capacity to own and sustain a national school meals program? (3.2 in IR matrix) 35) Quality and effectiveness of external context, relationships/partnerships, and processes, including but not limited to: o Communication, information￾sharing, and coordination with partners o Are the right partners on board – is anyone excluded o Extent of systematization of cooperative arrangements 36) Capacity strengths and weaknesses of institutional partners 37) Financial resource opportunities and constraints of institutional partners 38) Country/regional context: policy framework, politics, governance, civil infrastructure, business, markets, natural hazards 39) Factors inherent to serving remote communities 40) GEEW Programme documents Country/regional context information KIs with WFP staff and external stakeholders Observation Comparison of assessment of contributing factors by internal and external KIs Comparison of subjective and objective information Effectiveness Efficiency Sustainability Impact Date | Report Number 130 OECD-DAC CRITERIA 3: EFFICIENCY Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 3.1 How efficient is the programme in achieving results to date? (2.4 in IR matrix) OECD 3.2 Is the programme implemented in a timely way? (2.4 in IR matrix) OECD 3.3 Are the activities cost-efficient? (2.4 in IR matrix) OECD 3.4 Is the programme implemented in the most efficient way compared to alternatives? (2.4 in IR matrix) OECD 3.5 Were the project strategies efficient in terms of financial and human resource inputs as compared to outputs? (2.4 in IR matrix) OECD 3.6 Does the monitoring system efficiently meet the needs and requirements of the project? (2.4 in IR matrix) 41) Timing and timeliness of implementation; adherence to implementation schedule 42) Cost-efficiency of activities relative to alternatives 43) Efficiency of project strategies in terms of financial and human resource inputs vs outputs 44) Adequacy of monitoring system relative to programme needs and requirements Programme documentation, incl. budget documentation Endline data (EGRA and school survey) KIIs with WFP management, financial and M&E staff Compare assessments by KIs with performance data Efficiency Date | Report Number 131 OECD-DAC CRITERIA 4+5: IMPACT AND SUSTAINABILITY Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria OECD 4/5-1 Has the programme yielded any unintended results, positive or negative? (3.3 in IR matrix) 45) Existence/identification of positive results not anticipated in the results framework; exploration of the extent to which these results can be attributed to the program 46) Existence/identification of unintended negative results; exploration of the extent to which these results can be attributed to the program 47) GEEW Focus groups KIs Programme documents, incl. results framework Observation Comparison of data from all sources; exploration with the country office Impact Effectiveness OECD 4/5-2 How effective are efforts to date to build government capacity to fully hand over school feeding? (4.1 in IR matrix) OECD 4/5-3 To what extent is the Government taking ownership of the programme (e.g. demonstrated commitment and contributions)? (4.1 in IR matrix) OECD 4/5-4 What is the demonstrated capacity at central and sub-national levels to manage the program? (4.1 in IR matrix) OECD 4/5-5 Are local communities (SGACs, farmers’ groups, etc.) fully involved in and contributing toward school feeding and education activities? (4.1 in IR matrix) OECD 4/5-6 Has the policy framework supporting the HGSF been strengthened within the project period? (4.1 in IR matrix) 48) Analysis of programme documentation regarding steps taken and steps planned toward capacity development of smallholders and gov’t 49) Assessment of the existence, rationale and quality of capacity development plans 50) WFP perspectives (national and sub￾national) on gov’t readiness for handover 51) Gov’t perspectives (national and sub￾national) on gov’t readiness for handover 52) Perceptions of staff and stakeholders on efficiency (cost, systems, staff, alternatives, etc.). 53) Extent to which resources (human, physical, financial, organizational and functional) were optimally used in project implementation Programme documents KIIs – both national and sub-national level Compare perspectives of WFP and gov’t Relevance Effectiveness Sustainability Impact Date | Report Number 132 OECD 4/5-7 What are the major factors influencing the achievement or non-achievement of sustainability of the program? (4.1 in IR matrix) OECD 4/5-8 Is WFP’s capacity support to smallholder farmers and key line ministries appropriate/sufficient to effectively facilitate national ownership? (key Q from TOR) (4.1 in IR matrix) OECD 4/5-9 How WFP can work more strategically in its capacity development efforts with Government? (4.1 in IR matrix) OECD 4/5-10 What is the progress and status of capacity development? (4.1 in IR matrix) OECD 4/5-11 What capacity development approaches/ activities are working/not working? (4.1 in IR matrix) OECD 4/5-12 To what extent is the programme on track for handover readiness and Government ownership? (4.1 in IR matrix) 54) Quality of processes (e.g., supply chain management), relevance of the division of responsibilities in the management of the operation at different levels 55) Existence and quality of coordination that has been put in place with partners to optimize resources Date | Report Number 133 FACTORS AFFECTING THE RESULTS Sub-questions Analysis/ Indicators Data Sources Triangulation Evaluation Criteria Factors 1 What are the key WFP-internal factors contributing to achievement or non-achievement of desired results? (3.1 in IR matrix) Factors 2 What are the key external factors contributing to achievement or non-achievement of desired results? (3.2 in IR matrix) Factors 3 *(new in endline) To what extent did COVID-19 affect project implementation and performance? (3.1 and 3.2 in IR matrix) Note: Factors 1 and 2 replace What are the major factors influencing progress in achievement or non￾achievement of the outcomes/objectives of the intervention? (3.1 and 3.2 in IR matrix) 56) Quality and effectiveness of WFP￾internal context and processes, including but not limited to: o Staff level o Staff capacity o Financial resources o Supply chain/ pipeline management o Quality, timeliness and use of monitoring data o Quality, timeliness, and processes for internal communication and decision making (HQ-RB-CO-field) o Technical backstopping from HQ￾RB o Knowledge management o GEEW Programme documents KIs with WFP staff and external stakeholders Observation Comparison of assessment of contributing factors by internal and external KIs Effectiveness Efficiency Sustainability Impact Date | Report Number 134 Annex 10: Methodology This annex provides further details regarding the evaluation methodology. The evaluation is a performance evaluation and as noted in Section 1.1 of the main report, has the dual objectives of accountability and learning. Specifically, the endline evaluation i) reviews the project’s relevance, effectiveness and efficiency, impact, and sustainability, ii) collects performance indicator data for strategic objectives and higher-level results, iii) assesses whether the project has succeeded in achieving McGovern Dole’s two strategic objectives, iv) investigates the project’s overall impact, and v) identifies meaningful lessons that stakeholders can apply to future programming. A. DATA COLLECTION METHODS AND TOOLS The table below summarizes the data collection tools. Overview of main data collection tools Data collection method(s) Type of data to be collected Comments School administrator /headmaster survey (ODK) McGovern-Dole indicators -- ET to conduct in-person interviews. --Additional qualitative data to be collected from KIs and/or focus groups1 with headmasters/ head teachers, teachers, students, cooks, storekeepers, SGACs and SMCs. EGRA tool McGovern-Dole indicators ET will administer the EGRA (literacy + WASH questions) to 3rd graders. Topical outlines Qualitative data on all evaluation questions and to validate and help interpret indicator data Topical outlines for: --WFP Kigali and field staff --Government ministries --District Government (District Education Officials) --Project-supported District Coordinators --Implementing Partners (World Vision, Gardens for Health International, Rwanda Biomedical Centre) --Donors (USDA, MasterCard) --United Nations Agency Partners (UNICEF, FAO) --Schools (Head teachers/ headmasters, teachers, SGAC, School Feeding Committee) --Plus observation of WASH facilities, school meal distribution, kitchen conditions, commodity storage 1 Focus groups to be disaggregated by gender where feasible. 229.The baseline did not establish a value for the McGovern-Dole indicator, Percent of parents in target communities who can name at least three benefits of primary education. The report noted that this indicator could not be collected via a household survey, as a household survey would require approval by the National Institute of Statistics in Rwanda (typically a 3-month process).212 The endline evaluation assessed parent knowledge, attitudes and practices (KAPs) and perspectives via focus groups with SGACs, and – for triangulation – also included a question in the school headmaster survey to ask their perception of parent knowledge of the benefits of primary education. 212 A parent survey had been planned for 2016 but was not carried out. The midterm evaluation faced the same limitation to conducting a parent survey but added a question on this indicator in the head teacher survey to report progress based on head teachers’ estimates. Date | Report Number 135 230.See the evaluation matrix (Annex 9) for a detailed description of data sources and data collection tools and how these link to evaluation sub-questions. The evaluation questions were translated into data collection tools: topical outlines for interviews and focus groups, the EGRA tool, and a school administrator survey. Topical outlines were reviewed and updated where necessary for the field work once current programme documents were reviewed; the updated outlines were shared with WFP and USDA for review. The EGRA and school headmaster tools were pre-tested before commencing data collection. All tools were adjusted based on reviewer comments and pre-test results before they were finalized and deployed. 231.Quantitative data (EGRA results and school administrator survey data) were collected on Android tablets using Open Data Kit (ODK) and Tangerine (RTI) data collection software. Final versions of the tools were coded for application onto the Android devices and translated into Kinyarwanda. At any time, the tool questions and responses could be viewed on the tablets in Kinyarwanda or in English for quality and content review. 232.At TANGO’s request, the World Vision literacy team updated the Kinyarwanda reading content in the EGRA tool at endline to ensure that the students would be tested on fresh content. The endline tool reflects the same standard as in previous evaluations and was administered to 3rd graders to ensure comparability of results. World Vision’s literacy team worked in-country with the local ET to refresh and update their knowledge of the survey programming and orient them to the new content. In addition, the EGRA tool was augmented with the few questions needed to collect data for required WASH indicators, which would also map new and current WASH programmes in the surveyed schools. 233.TANGO worked with World Vision on the design, programming and training on tablets and software. National team members participated in all training activities, and field tested and administered the tools. 234.Quality control included researchers providing daily updates to TANGO. Data was uploaded to TANGO secure servers twice per week at minimum (see table below). TANGO reviewed the data and provided feedback on data quality and survey progress and highlighted specific data issues to be discussed with researchers. Data protection Data Protection: TANGO maintains daily backup copies of all qualitative and quantitative data in a secure physical location, on site at TANGO headquarters as well as in separate secure locations on secure cloud servers that are only accessible to TANGO data managers. TANGO assignments that employ tablets for data collection (e.g., mixed-methods baseline, midterm and final evaluations) use CAPI software. Data are uploaded daily from the field to secure cloud servers in an encrypted format. Data on the servers are only accessible to authorized TANGO data managers. The downloadable ODK software TANGO uses does not have any mechanisms that might allow ODK to access or control TANGO’s devices or systems. TANGO contracts with an IT specialist who follows a protocol to ensure that TANGO IT systems (hardware and software) are equipped with current anti￾virus, malware, and other relevant tools to ensure the maintenance and security of the data and information that TANGO collects and produces in the course of business. 235.TANGO concluded the field mission with a debrief or workshop that served as an informal validation of preliminary findings. This was followed by discussions with the country office and ERG to formulate actionable and complete recommendations to include in the draft report. B. SAMPLE SELECTION 236.Two kinds of quantitative data were collected from all students in the sample: (1) data from applying the Early Grade Reading Assessment (EGRA), to assess any changes in reading levels, and (2) data to inform the indicator, Percent of students who can identify at least three key health and hygiene practices, to assess any changes in health and nutrition practices. 237.The endline sampling methodology replicated the methodology used at midterm. The ET applied the formula below to arrive at the minimum sample size of students to assure statistical accuracy in comparisons across subsamples or survey rounds: Date | Report Number 136 n = Deff* [(Zα + Zβ)2 * (P1*(1-P1) + P2*(1-P2) / (P2 - P1)2]*NR where: Variable Assumed value Description n = Deff = 2 Design effect for complex sample design (assumed to be = 2) Zα = 1.282 Z value associated with desired significance level for confidence (90%, one-tailed) Zβ = 0.842 Z value associated with desired significance level for power (80%, one￾tailed) P1 = 50.0% estimated level of an indicator measured as a proportion at the time of the first survey or within a comparison group P2 = 65.0% expected level of the indicator either at a later survey round or different comparison group. (P2 - P1) is the magnitude of change or difference across subgroups that the sample is powered to detect (in this case, a difference of 30%, or 15 percentage points). NR 10.0% Non-response rate 238.This formula computed a minimum required sample size of 210 students to enable statistically accurate comparisons for a single stratum. The McGovern-Dole indicators are program-wide, i.e., based on students in all four districts where HGSF is implemented. While it was possible to stratify the sample by grade and achieve statistical accuracy, this would have required a sample of 210 students in each grade of interest (i.e., 210 students in 1st grade, 210 students in 2nd grade, etc.). Likewise, stratifying by district would have require samples of 210 students per district. Given that the McGovern-Dole indicators are whole-programme and that significant additional resources would have been required to make statistically robust comparisons across grades or districts, the ET used only two strata, stratifying by gender only. Such disaggregation was important to make statistically valid comparisons of results for boys and girls. As statistically representative gender-disaggregated data were not available for the baseline, stratifying at midterm allowed the ET to establish statistically representative data points for boys and girls and thus allowed meaningful comparisons at endline. This required a minimum sample size of 210 male students and 210 female students; the ET rounded up the sample size to 220 males and 220 females for logistical ease – where 11 male and 11 female P3 students were interviewed in 20 schools. 239.The ET noted the country office’s interest in disaggregating results to enable exploring the possible reasons for any differences. The ET therefore proposed to (1) disaggregate results by district – with the understanding that these results are indicative, not statistically representative; and (2) use the preliminary disaggregated results to inform qualitative lines of inquiry, to better understand why some districts may be achieving different results than others. C. STUDENT SAMPLING FRAME 240.All 220 male students and all 220 female students were administered the EGRA and the health/WASH questions added to the end of EGRA tool. The sampling frame was 3rd grade students, to maintain comparability with the baseline. The process for defining the student sampling frame mirrored the systematic random sampling method used in the baseline. The first stage of the sample selected 20 schools from the 104 schools that were receiving support at the time of the midterm evaluation, using probability-proportional-to-size (PPS) systematic random sampling. The ET acknowledged that the programme has since added three schools, increasing the number of total schools from 104 to 107; this was a very small percentage increase in the sample frame and thus did not require a new calculation. Moreover, the three new schools only received a meal and a few selected activities, thus other areas of the programme that were measured at midterm could not be assessed such as reading comprehension, WASH etc. Using the same 20 schools at midterm and endline allowed a panel study at the school level, whereby the same schools were visited in both rounds and assessed with respect to the same activities. Date | Report Number 137 241.The second-stage sample selection and sampling frame was developed and completed upon arrival in the school using the attendance register for that day (this presumed that the Covid context/protocols allowed for an in-school EGRA). In each 3rd grade class one sampling frame was developed for girls and one for boys. Then 11 students were randomly selected from each sampling frame. The ET notes that resources did not allow for collecting data from a control group/counterfactual for outcome monitoring. D. SCHOOL SELECTION AND SCHOOL-LEVEL SURVEY 242.As discussed above, 220 boys and 220 girls were interviewed for two sampling strata, totaling 440 students. The number of schools to be visited to administer the school survey and EGRA tool and reach 440 students was a function of how many EGRAs and school surveys could be completed in a day. The EGRA/school survey team consisted of four national evaluators. The ET estimated that in one day, one team could administer the EGRA to 11 students and conduct the survey in 2-4 schools. The ET had budgeted 15 school days (i.e., excluding weekends) for this activity, which allowed the team to reach the sample size of 440. 243.As noted in the previous section, the endline used the same sample of schools used at midterm, which was a two-stage clustered sample of 20 schools drawn using the PPS method. The ET used a country￾office-provided list of all project schools showing how many girls and boys in each school’s 3rd grade class. While the margin of error corresponding to the EGRA and WASH data from the two student samples (girls and boys) remained 10 percent, the margin of error for the data from the multi-module school surveys was 17 percent. We found the latter margin of error reasonable given the survey budget. A lower margin of error would have required a larger sample for the school surveys, and this was not achievable with the current budget. 244.The students in each selected school were administered the EGRA+WASH tool. In addition, the EGRA/school survey team administered a school-level quantitative survey to school administrators to assess progress on school-level indicators. 245.There were a few considerations regarding the timing and manner of administration of the school survey tool. The draft Word Vision version from the midterm survey was adjusted slightly for the endline, and sent to WFP to review the changes, then returned to TANGO for finalization in Word and ODK. The EGRA/school survey team pre-tested the tool with age-appropriate children after the EGRA workshop and before the start of fieldwork. Revisions were made to integrate any needed adjustments indicated by the pre-test to finalize the tool for fieldwork. 246.For the headmaster portion of the survey, the team collected indicator data for the school directly from the headmaster. While two team members administered the EGRA to students, the third team member interviewed the headmaster to complete that portion of the survey on the tablet. E. ADMINISTRATION OF EGRA 247.The EGRA was administered by the national ET members made up of Rwandan consultants (the EGRA/school survey team), who worked in teams of two. 248.As in the baseline, the ET sought to minimize interruption of school learning activities by coordinating closely with school headmasters and teachers on student sampling, warm-up and assessment. The national team members consulted with WFP staff, literacy partner (World Vision), and school staff in advance to determine the timeliest window for conducting the assessment in each school. F. SITE SELECTION FOR QUALITATIVE TEAM FIELD VISITS 249.As international travel restrictions related to Covid-19 prevented the international evaluators from being present in Rwanda, the national team also assisted with the collection of primary qualitative data with the guidance of the international evaluators. This constituted a “deep dive” inquiry at school, sector and district level, covering all districts and geographically focused on a subset of 10 of the 20 schools in the EGRA/school survey sample. As noted, data from EGRA/school survey were statistically analyzed by TANGO in tranches on a rolling basis, in order to inform and fine-tune both the sampling and the lines of inquiry of the qualitative work. Date | Report Number 138 250.The national team visited the same 10 schools and/or communities (depending on access under Covid￾19 guidelines) visited during the midterm. These schools were purposively selected based on (1) schools where quantitative data analysis has been completed; (2) trends and questions emerging from the initial quantitative data analysis (3) location, to collect input from schools in varied settings and contexts (4) school size and (5) logistical feasibility. The national team interviewed headmasters, teachers, SGAC members, cooks and made observations on WASH facilities, kitchen conditions, commodity storage facilities, and where possible observed school meal distributions. G. SITE SELECTION FOR AGRICULTURE COOPERATIVES AND FARMER GROUPS 251.Selection of agriculture cooperatives and farmer groups was undertaken in close consultation with WFP staff and included cooperatives visited at midline to assess their development, as well as cooperatives that were most relevant to the procurement options at endline. Number of cooperatives visited by the ET was determined in consultation with WFP staff and on convenience based on the school field tour. H. DATA ANALYSIS 252.Quantitative analysis was conducted using STATA version 15.0 software. The ET conducted descriptive analysis, such as cross-tabulations, means, and significance tests to provide additional context to the underlying components of key indicators. This analysis was conducted per module of the school administrator survey tool. The student and school samples drawn at endline were self-weighted, and therefore no sampling weights was calculated or applied in the estimation of statistics. The data collection tools were applied using software programmed with internal controls to ensure data are consistent and reduce the likelihood of data entry errors. During the data analysis stage, all data were reviewed for outliers (initially flagged as three standard deviations or more for continuous variables). Discrete variables were reviewed for consistency. When potential data errors were identified, the analyst asked the researcher to consult any notes associated with the interview. If no further information was available, then the data point was recoded to missing. 253.The qualitative analysis used a matrix approach to record, organize and analyze qualitative data and ensure all team members recorded information consistently and in a manner that directly responded to research questions. Capturing qualitative data in matrices enabled identification of important patterns in responses and contextual information that may help to explain quantitative or secondary data. Data matrices also helped triangulate responses from FGDs, KIIs, in-depth interviews and other sources to determine whether information is reliable. 254.During the analysis and reporting phase, the ET built in systematic opportunities for WFP and stakeholder consultation. This included but is not limited to sharing the draft report and soliciting comments, sharing draft recommendations to review and refine via teleconference and/or email communications, and other teleconferences as needed. Date | Report Number 139 I. LIMITATIONS AND RISKS Table 94 replicates the table in the inception report for this evaluation, listing potential methodological limitations to the evaluation and proposed strategies to minimize their impact. It adds a column to report whether the anticipated limitations were encountered in the conduct of the evaluation, and steps taken. Table 94: Anticipated methodological limitations and mitigation strategies; comments post-evaluation Possible Limitation as Identified in Inception Phase Discussion Mitigation Strategy Comments Post-Evaluation Quality and availability of secondary data, including the baseline data set Secondary data sources/ documents typically vary in quality and reliability. Some of the data/ information the ET requests may not be available or may take a significant time investment to acquire. Some of the baseline data may vary in quality or structure and thus limit comparability in subsequent evaluations. Assess the quality of secondary sources; prioritize analysis of research/data deemed high-quality; triangulate data across sources and provide opportunities for diverse stakeholders to validate findings. Consult with the country office and other stakeholders early in the evaluation process to identify data needs and gaps to enable primary data collection to address gaps and quality issues. The presence of non-response data for reading comprehension in the baseline data reduced the comparability from endline to baseline. To address this constraint, baseline values were re￾estimated. Respondent bias Respondent bias is an inherent risk in any evaluation. For example, respondents may wish to report in a way they think will favor them in terms of new or continued programme benefits or positive recognition. The ET will solicit perspectives from a range of stakeholders and take anticipated biases into account during analysis; the team will seek a balance of perspectives. The methodology will rely on a cross-section of information sources (e.g., stakeholder groups, beneficiaries) and using a mixed-methods approach to ensure triangulation of information through a variety of means. Strategy implemented as planned. No issues encountered. Date | Report Number 140 Budget and time constraints The scope of the evaluation and time in the field are determined by resources available. The scope must therefore be clearly defined and delimited. Use the inception phase to determine, per consultation with WFP stakeholders, evaluation priorities and special points of interest (e.g., thematic, geographic, strategic). Ensure clear expectations of all parties on the main evaluation questions and the extent to which these can feasibly be investigated given existing data/ information and the time and resources available for collecting and analyzing this and new information. The evaluation budget and in-country time were adequate. The ET found the timeline between fieldwork and the submission of the draft report to be challenging, compounded by delays in receiving information due to office absences, and the receipt of needed data well into the drafting period Generalizability of findings As noted above, resources and time determine how much ground the ET can cover. This is not necessarily problematic, however when framing findings it is important to identify any limitations on generalizability of findings. In addition, the impact of Covid-19 prevention measures and the closure of schools for much of 2020 will limit the generalizability of findings to the baseline and midterm. Specific limitations of the programme will be noted in the evaluation report and presentations of findings. Again, agreement and understanding of all stakeholders regarding the scope of the evaluation is important to interpreting the findings, analysis, and recommendations in the appropriate context. The ET will assess findings in light of the impact of Covid-19 and its associated prevention measures on the programme. Strategy implemented as planned. No issues encountered. Date | Report Number 141 Turnaround time for approval of evaluation tools Conformance with the proposed schedule is contingent on the timely review and approval of the EGRA/school survey tool and the topical outlines. The schedule is tight, so there is little room for delays from any party. TANGO will ensure regular, frequent communication with the country office so that any adjustments by any party to anticipated timelines/due dates are shared immediately and the calendar can be adjusted to accommodate the requirements of all sides as best as possible. Certain activities must take place in January: the EGRA workshop, updating and programming the EGRA and school survey, and field tour planning; TANGO and the country office must work concurrently to the inception report review to obtain the information needed for these tasks. Strategy implemented as planned. No issues encountered. Mobility The international evaluators are unable to be present in Rwanda due to international travel restrictions. The local team may encounter mobility limitations such as those due to Covid-19 restrictions, weather or terrain. These constraints could affect the sampling and field schedule. At this time, there are no issues to constrain mobility, although Covid-19 restrictions will need to be assessed just prior to fieldwork and are subject to change. The ET will communicate regularly with the country office regarding any changes that would affect the activity/ travel plan. Date | Report Number 142 School closures or delays in opening schools Covid-related protocols in Rwanda affect school opening and hours, and thus the ET’s easy access to students for EGRA testing and to personnel for interviews. At this writing, the re-opening date for lower primary is set for 18 January 2021. TANGO and the country office are communicating regularly to stay abreast of school status and adjust plans as necessary. If lower-grade primary schools are not open or not fully accessible due to Covid-19 restrictions, and the team is not able to start the survey at the schools within the time allocated for field data collection, the EGRA cannot be conducted and will be dropped from the endline. A community-based approach to the EGRA survey is not advisable. As applicable, TANGO will also follow the “Do No Harm” protocols we have modified for working in the Covid context; see Annex 10. Strategy implemented as planned. No issues encountered. Date | Report Number 143 J. ETHICAL SAFEGUARDS 255.All TANGO staff, consultants, and officers associated with this evaluation complied with TANGO’s policies and procedures, including TANGO’s Code of Ethics and Conduct. The ET also received refresher information as needed on ethical research safeguards and child and youth protection, based on UNICEF guidance and WFP policies and standards. 256.The ET ensured appropriate ethical considerations were in place for all interviews by informing all interviewees of the purpose and duration of the interview, how they were identified to participate in the interview, informing interview participants of their rights, providing guarantees that specific interview findings will remain confidential and that all information provided will be used to assess the programme, with no direct attribution to the interviewee. Finally, all interviewees were informed that they may choose not to participate, and gave their verbal consent to participate before the interview moved forward. The ET also obtained verbal consent before taking any photographs. There were neither interview nor photo refusals. Enumerators introduced themselves to the head teacher, teachers and students, and obtained consent before starting activities at the school. They treated administrators, teachers, students and others respectfully and talked to students in a friendly and supportive manner. 257.TANGO has conformed to WFP and UNEG ethical standards in the conduct of this evaluation. We safeguarded and ensured the observation of ethical practices at all stages of the evaluation cycle. This has included ensuring informed consent, protecting privacy, confidentiality and anonymity of participants, ensuring cultural sensitivity, respecting the autonomy of participants, ensuring fair recruitment of participants (including women and socially excluded groups) and ensuring that the evaluation results in no harm to participants or their communities. 258.Table 95 describes ethical issues that were considered in the evaluation preparation/design, data collection, data analysis, reporting and dissemination, and how they were monitored and managed during the implementation of the evaluation. Table 95: Ethical issues and safeguards implemented • Ethical issue • Safeguard/measure • 1. Interviewing child/youth respondents (child protection issue) – for EGRA assessment and student interviews/ focus groups • 1. Ensured that a teacher was present, to monitor the interview and ensure that the child felt safe and comfortable. • 2. Reviewed World Vision child protection protocols and signed statements of compliance. • 3. Consulted with WFP programme staff for any additional guidance. • 2. Reliability and accuracy of school-level data • 2. During school visits, EGRA/school survey team reviewed survey responses and spot-checked supporting documentation to validate data accuracy. Individual student performance on EGRA tests were not shared with teachers. • 3. Compliance with Evaluation Code of Conduct • 3. As contractually required, all consultants read and agreed to conform to the WFP Code of Conduct and Ethical Standards and the UNEG Code of Conduct for Evaluation in the UN System. • 4. Data protection • 4. Tablets were password secure and data were uploaded as soon as possible to a remote secure server. K. QUALITY ASSURANCE 259.A foundation of quality control is effective training on data collection techniques, methodologies, and the technology used to collect data. The entire team is highly experienced in all methodologies employed in this evaluation, so the training elements focused on reviewing specific data collection tools, and data entry and protocols for ODK surveys with the national researchers. The ET members, with the support of HQ-based TANGO analysts and survey specialists, were the primary developers of Date | Report Number 144 both the topical outlines and the school survey tools, and provided orientation and training to national researchers on all tools. The ET, including quantitative and qualitative researchers, was small, with team members engaged in both qualitative and quantitative activities. This increased the reliability of the data – ensuring tools were applied equally across different regions, settings, and schools. 260.Sample size determination and sampling procedures were verified by Mark Langworthy, a TANGO partner and economist with extensive experience in quantitative surveys. A HQ-based TANGO analyst programmed the ODK survey tools into computer tablets and provided in-person training and real-time support to data collection and quality assurance for the EGRA/school survey. Supervision and coordination with the quantitative team were maintained by remote data quality control to identify any problems or anomalies, plus regular voice communication between the enumerators and the team leader. The team uploaded data to the TANGO server daily. TANGO reviewed the data and provided feedback on data quality and survey progress and highlighted specific issues twice per week. Data quality checks confirm strong internal validity for the quantitative tools. Equally, the quantitative results were largely in line with qualitative data – suggesting strong external validity. 261.It is the duty of the team leader to ensure the protection of human subjects and their confidentiality, and training interviewers in obtaining verbal consent from participants. To ensure the security of the data, TANGO follows standard operating practices such as locked files, password secured databases and the handing over of all hard copies (where applicable) to TANGO. Participating individuals were assigned identification numbers, and this number is the only identifier to appear on any data collection tools such as surveys, written notes, transcripts and labels on audio tapes. A single list linking the names of participating individuals and their identification numbers is kept in a protected file that is only accessible to a small number of TANGO senior staff. 262.Quality assurance was maintained in the reporting process by a TANGO Quality Assurance Manager who reviewed content to ensure that it met the criteria laid out in the TOR, that all comments by stakeholders were addressed, and that the report content fairly reflects the findings of the qualitative team and the quantitative data. Qualitative interviews with adults were recorded with the permission of the interviewees, and the recording was used to verify the accuracy of the interviewer notes. 263.Accountability to affected populations links to WFP’s commitments to include beneficiaries as key stakeholders in its work. As such, WFP is committed to ensuring gender equality and women’s empowerment (in the evaluation process, with participation and consultation in the evaluation by women, men, boys and girls from different groups. The final evaluation report’s findings, conclusions and recommendations reflect the ET’s GEEW analysis as appropriate. Date | Report Number 145 Annex 11: List of People Interviewed Table 96 presents summary information regarding interviews conducted to comply with USDA requirements for excluding personally-identifying information in the evaluation report. Table 96: Key informant interviews conducted Title #KIIs WFP Rwanda WFP Rwanda staff 9 KIIs (2M, 7F) School - based School-based staff (teachers, head teachers) and SGAC members 10 KIIs (8M, 2F) District Government District staff/ officials (department heads, mayors/vice-mayors, education officers, inter alia) 15 KIIs (11M, 4F) National Government National government staff/officials 2 KIIs (1M, 1F) Partner NGOs Partner NGO staff 6 KIIs (5M, 1F) USDA USDA staff 2 KIIs (1M, 1F) Table 97: Focus group discussions conducted Focus Group Type #M #F District/sector School/other Date Cooks 3 0 Nyamagabe Primary school 08/02/2021 Storekeeper 1 0 Nyamagabe Primary school 08/02/2021 Students 3 3 Nyamagabe Primary school 08/02/2021 Cooks 2 1 Nyamagabe Primary school 09/02/2021 Literacy teachers 2 6 Nyamagabe Primary school 09/02/2021 SGAC 2 2 Nyamagabe Groupe scolaire 09/02/2021 Parents 2 3 Nyamagabe Groupe scolaire 09/02/2021 Parents 4 3 Nyamagabe Primary school 09/02/2021 Cooks 2 1 Nyamagabe Groupe scolaire 09/02/2021 Students 4 4 Nyamagabe Groupe scolaire 09/02/2021 Students 3 2 Nyamagabe Primary school 09/02/2021 Cooperative 5 3 Nyamagabe Other 10/02/2021 Cooperative 4 3 Nyaruguru Other 11/02/2021 Storekeeper 0 1 Nyaruguru Groupe scolaire 11/02/2021 Cooks 3 1 Nyaruguru Groupe scolaire 11/02/2021 Cooks 3 1 Nyaruguru Primary school 12/02/2021 Literacy teachers 3 3 Nyaruguru Primary school 12/02/2021 SGAC 3 2 Nyaruguru Groupe scolaire 12/02/2021 Parents 3 3 Nyaruguru Primary school 12/02/2021 Students 4 4 Nyaruguru Primary school 12/02/2021 Parents 5 1 Karongi Groupe Scolaire 16/02/2021 Date | Report Number 146 Table 97: Focus group discussions conducted Focus Group Type #M #F District/sector School/other Date Literacy teachers 1 5 Karongi Groupe scolaire 16/02/2021 Cooks 2 1 Karongi Groupe scolaire 16/02/2021 Storekeeper 0 1 Karongi Groupe scolaire 16/02/2021 Students 5 3 Karongi Primary school 16/02/2021 Students 7 8 Karongi Groupe scolaire 16/02/2021 Reading facilitators 1 2 Karongi Primary school 16/02/2021 Cooperative 2 0 Karongi Other 17/02/2021 Cooks 2 1 Nyaruguru Group scolaire 18/02/2021 Storekeeper 0 1 Nyaruguru Groupe scolaire 18/02/2021 Students 3 3 Nyaruguru Primary school 18/02/2021 SGAC 5 0 Rutsiro Groupe scolaire 18/02/2021 Cooperative 2 3 Rutsiro Other 19/02/2021 Cooks 2 1 Rutsiro Primary school 19/02/2021 SGAC 5 0 Rutsiro Primary school 19/02/2021 Students 4 4 Rutsiro Primary school 19/02/2021 Students 4 4 Rutsiro Groupe scolaire 19/02/2021 Reading facilitators 3 3 Karongi Primary school 19/02/2021 Reading facilitators 1 1 Karongi Primary school 19/02/2021 Parents 3 3 Karongi Primary school 22/02/2021 Storekeeper 1 0 Karongi Primary school 22/02/2021 Cooks 2 0 Karongi Primary school 22/02/2021 Storekeeper 2 0 Karongi Primary school 22/02/2021 Students 4 4 Karongi Primary school 22/02/2021 Literacy teachers 4 4 Rutsiro Primary school 22/02/2021 Reading facilitators 3 4 Karongi Primary school 24/02/2021 Literacy teachers 8 0 Karongi Primary school 24/02/2021 SGAC 5 3 Karongi Primary school 24/02/2021 Cooks 1 1 Karongi Primary school 24/02/2021 Storekeeper 1 0 Karongi Primary school 24/02/2021 Students 4 4 Karongi Groupe scolaire 24/02/2021 Students 4 4 Karongi Groupe scolaire 24/02/2021 Students 4 4 Karongi Groupe scolaire 24/02/2021 TOTAL 156 119 (total: 275) Date | Report Number 147 Annex 12: Field Schedule Table 98: Field schedule Day # Date Day Time Region District School name Day 1 04.févr Thur AM Nyaruguru Ngoma Groupe Scolaire Kiyonza PM Nyaruguru Ngoma -- Day 2 05.févr Fri AM Nyaruguru Ruheru Remera Primary School PM Nyaruguru Ngoma Groupe Scolaire Kiyonza Day 3 08.févr Mon AM Nyamagabe Kamegeri Kirehe Primary School PM Nyamagabe Kamegeri Kiraro Protestant Primary School Day 4 09.févr Tue AM Nyamagabe Cyanika Rugogwe Primary School PM Nyamagabe Cyanika Kiyumba Primary School Day 5 10.févr Wed AM Nyamagabe Musange Musange Primary School PM Nyamagabe Musange -- Day 6 11.févr Thur AM Nyarurguru Nyabimata Nyabimata Primary School PM Nyaruguru Nyabimata Gihemvu Primary School Day 7 12.févr Fri AM Nyaruguru Kivu Rugerero Primary School PM Nyarurugu Kivu Day 8 15.févr Mon AM Rutsiro Kivumu Kivumu Groupe Scolaire PM Rutsiro Kivumu Buganamana Primary School Day 9 16.févr Tue AM Karongi Gashari Mwendo Groupe Scolaire PM Karongi Gashari -- Day 10 17.févr Wed AM Rutsiro Ruhango Rwamiko Groupe Scolaire PM Rutsiro Ruhango -- Day 11 18.févr Thur AM Rutsiro Manihira Kabeza Groupe Scolaire PM Rutsiro Manihira Day 12 19.févr Fri AM Rutsiro Manihira Ruhango Primary School PM Rutsiro -- Day 13 22.févr Mon AM Karongi Murundi Nyamabuye Primary School PM Rutsiro Murundi -- Day 14 23.févr Tue AM Karongi Ruganda Nyamugwagwa Primary School PM Karongi Gitesi Kanunga Primary School Day 15 24.févr Wed AM Karongi Murundi Mujyojyo Primary School PM Karongi Murundi -- Date | Report Number 148 Annex 13: Supplementary Tables: EGRA Survey Results Table 99: Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text Final target Baseline Midterm Endline Overall Actual achievement Actual v. target Actual Actual v. target Actual Actual v. target Female 77% 43.3% 56.2% 62.9% 81.7% 85% 110.4% 110.4% Male 62% 39.8% 64.2% 56.8% 91.6% 70.2% 113.2% 113.2% Total 69% 41.5% 60.1% 59% 85.5% 77.7% 112.6% 112.6% Source: Baseline, midterm and endline EGRA surveys Table 100: Reading, listening and comprehension of short story, midterm and endline Midterm Endline Total Male Female Total Male Female Percent of students to read short story1 64.6 60.5 68.8 80.5 73.5 87.3 Percent of students to fully comprehend the short story (regardless if they read or listened) 82.8 82.4 83.2 95.2 94 96.4 n 441 220 221 435 215 220 Source: Midterm and endline EGRA surveys 1 No time restrictions were placed on the students for this exercise, however the 60-second point is tracked in the data. This measure includes students who complete the story (regardless of time taken) with minimal errors. Table 101: Percent of third grade students in words-correct-per-minute (WCPM) range, midterm and endline WCPM Midterm Endline Overall Male Female Overall Male Female 0 18.4 25.0 11.8 0 0 0 1 to 15 14.3 8.6 19.9 0.7 0.5 0.9 16 to 30 32.4 25.5 39.3 3.1 1 5.1 31 to 40 16.1 16.8 15.4 8.5 4.3 12.6 41 to 56 18.8 24.1 13.6 38.6 29.5 47.4 > 56 words n/a n/a n/a 49.2 64.76 34 n 441 220 221 435 215 220 Sources: Midterm and endline EGRA surveys Table 102: Reading and comprehension at endline, by district, midterm and endline Indicator Target Midterm n Endline n Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text 69.0 51.5 441 77,7 435 Karongi district n/a 41.8 110 76,2 105 Nyamagabe district n/a 51.8 110 70,0 110 Nyaruguru district n/a 58.2 110 83,6 110 Rutsiro district n/a 54.1 111 81,0 110 Source: Midterm and endline EGRA surveys Date | Report Number 149 Table 103: Students' home support and time allocation, midterm and endline Indicator Midterm Endline Percent of students to get help reading from parents for homework/reading 83.2 91.5 Male students 87.3 86.9 Female students 78.7 95.9 Percent of students to have enough time to study and do homework 88.0 98.9 Male students 89.1 97.7 Female students 86.9 100.0 n 441 435 Source: Midterm and endline EGRA surveys. No significant difference between male and female students at midterm is observed at a p<0.05 level or lower. Table 104: Percentage of students in classroom identified as attentive by their teachers Final target Baseline Midterm Endline Actual Actual vs. target Actual Actual vs. target Actual Actual vs. target 80% 60% 75% 57.4% 71.8% 74.1% 92.6% Source: Baseline, midterm and endline head teacher surveys Table 105: Student attentiveness, by grade; baseline, midterm, and endline Target Baseline Midterm n a Endline n Percent of students in classrooms identified as attentive by their teachers as attentive 80.0 60.0 57.4 74.1 First grade n/a 66.0 2,116 72.1 10,254 Second grade n/a 58.0 1,804 75.4 6,213 Third grade n/a 59.8 1,844 79.4 5,848 Fourth grade n/a 51.8 1,667 73.3 5,649 Fifth grade n/a 51.4 1,540 73.8 5,938 Sixth grade n/a 50.9 805 70.8 4,448 Weighted total 57.4 57.4 9,776 74.0 38,350 a Reported number of total students in each grade Source: Midterm and endline head teacher surveys Table 106: School administrator use and application of teaching techniques, midterm and endline Midterm Endline % of Head teachers Target # of Head teachersa % of Head teachers Target # of Head teachers School administrators and officials who received trainings or certifications as a result of USDA assistance 95.0 93 99 100.0 104 n 20 104 20 104 a Values extrapolated from sample data Source: Midterm and endline head teacher surveys Date | Report Number 150 Table 107: Sources of food for school meals, midterm and endline Midterm Endline % of Schools # of Schools a % of Schools # of Schools Partnership with local farmer groups 0 0 0 0 Source of food for schoolchildren: NGOs providedb 100 104 90.0 94 School garden 50 52 45.0 47 Local markets 40 42 55.0 57 Parents provided 5 5 0.0 0 n 20 104 20 104 Source: Midterm and endline head teacher surveys a Extrapolated from sample data b The survey question is Where is the food for the children obtained? This response option is worded NGOs provided; there is no response option for WFP. NGOs include WFP, and there are no agencies providing food to the schools. Table 108: Parents' knowledge of education benefit and storekeepers' knowledge of good practices; baseline, midterm and endline Indicator Baseline Target Midterm n Endline n Percent of parents in target communities who can name at least three benefits of primary education n/a 90.0 69.8a 20 74.3 20 Percent of cooks and storekeepers who can identify at least three safe food preparation and storage practices 92 95.0 85.0 20 40.0 20 a A proxy indicator asking school head teachers to estimate this proportion Source: Midterm and endline head teacher surveys Table 109: School SGAC engagement, midterm and endline Midterm Endline Mean number of schools with SGAC (or similar group) per school 1.5 1.55 Mean number of active SGAC members 9.6 9.95 Mean number of SGAC meetings per year 3.8 3.05 Percent of SGACs to have available meeting minutes 100.0 100.0 Percent of SGACs to receive training 100.0 100.0 Percent of SGACs to receive type of training: Nutrition / school feeding 90.0 80.0 School garden 85.0 100.0 Improved school management 80.0 70.0 School governance 65.0 20.0 Health 60.0 15.0 School infrastructure 30.0 0.0 n 20 20 Source: Midterm and endline head teacher surveys Date | Report Number 151 Table 110: Health and hygiene practices at endline, by district, midterm and endline Indicator Midterm Target n Endline Target n Percent of students who can identify at least three key health and hygiene practices 49.2 80 441 52.9 80 435 Karongi district 48.2 n/a 110 41.0 n/a 105 Nyamagabe district 47.3 n/a 110 54.5 n/a 110 Nyaruguru district 60.9 n/a 110 64.5 n/a 110 Rutsiro district 40.5 n/a 111 51.0 n/a 110 Source: Midterm and endline EGRA surveys Table 111: Health and hygiene practice utilization/application, midterm and endline Indicator Midterm Endline Percent of students to regularly practice: Other personal hygiene (i.e., bathing) 91.4 80,7 Handwash with soap after toilet 42.0 44,4 Handwash before eating 36.5 46,7 Drink clean/treated water 15.2 n/a Use of clean water 10.2 n/a Brush teeth 8.4 21,1 Eating a balanced diet 5.0 8,7 Avoid open defecation 4.8 8,0 Eat well-cooked food 4.8 n/a Wash fruits and vegetables 2.0 n/a n 441 435 Source: Midterm and endline EGRA surveys Table 112: School water source and availability, midterm and endline Midterm Endline Target % of Schools # of Schoolsa % of Schools # of Schools Schools using an improved water source 104 95.0 99b 100.0 104 Piped water (improved) n/a 80.0 83 100.0 104 Protected spring (improved) n/a 10.0 10 10.0 10 Rainwater (improved) n/a 5.0 5 5.0 5 Unprotected spring (not improved) n/a 5.0 5 5.0 5 Water is available from primary water source 4+ days per school week n/a 75.0 78 75.0 78 n 20 104 20 104 Source: Midterm and endline head teacher surveys a Values extrapolated from sample data b There is a small discrepancy in the midterm value for this indicator as found in the MTE survey data (value: 99) and the 2018 semi￾annual report (value: 104). The midterm survey finds that 95% of the sample has an improved water source, which was determined by enumerator observation, which suggests that at least one school was observed to lack an improved water source. In an email communication (28 Nov 2018), the country office stated that at baseline, 62 of the 104 schools already had an improved water source (this is consistent with earlier documentation) and that since the baseline, World Vision has supplied 55 schools with rainwater harvesting tanks “meaning that all 104 schools now have an ‘improved water source.’ ” A possible explanation for this discrepancy is a possible difference in how improved water source was defined at baseline and midline and/or interpreted by the respondents. Date | Report Number 152 Table 113: External support to schools, midterm and endline Midterm Endline % of Schools # of Schools a % of Schools # of Schools Sources of Support Receiving external support for: School feeding 100 104 100 104 WFP Training of teachers 100 104 70 73 WFP, World Vision Deworming 95 99 80 83 Government Provision of school materials, textbooks, books 95 99 90 94 MINEDUC, World Vision Renovation/construction of infrastructure in school, e.g., classes, kitchens, stores 95 99 45 47 WFP, MINEDUC Sanitation (water and toilets) 85 88 90 94 Plan International, World Vision, UNICEF School governance 85 88 95 99 WFP, MINEDUC Other nutrition activities 80 83 95 99 GHI Health education 55 57 40 42 MINISANTE n 20 104 20 104 Source: Midterm and endline head teacher survey a Values extrapolated from sample data Date | Report Number 153 Bibliography EGRA Toolkit, Second Edition. https://globalreadingnetwork.net/resources/early-grade-reading-assessment￾egra-toolkit-second-edition. Government of Rwanda. 2004. Rwanda Vision 2020. Government of Rwanda. 2013. Economic Development and Poverty Reduction Strategy 2013 – 2018. National Institute of Statistics Rwanda (NISR). 2015. Rwanda Poverty Profile Report 2013/14. NISR. 2018. Rwanda Poverty Profile Report 2016/17-Results of Integrated Household Living Conditions Survey (EICV5). NISR. 2018. Vision 2020 Umurenge Programme (VUP) - Baseline Survey. REB/MINEDUC. 2015. Competence-Based Curriculum: Summary of Curriculum Framework Pre-primary to Upper Secondary 2015. Republic of Rwanda. Rwanda Demographic and Health Survey 2019-20. Key Indicators Report. National Institute of Statistics of Rwanda, Ministry of Health, and ICF. 2020. Republic of Rwanda, Ministry of Education. 2018. Education Sector Strategic Plan, 2018/19-2023/24. Republic of Rwanda, Ministry of Education. 2016 Education Statistical Yearbook. Republic of Rwanda, Ministry of Education. 2014. National School Health Policy, Kigali 2014. Republic of Rwanda, Ministry of Education. 2008. Girls’ Education Policy, April 2008. Republic of Rwanda, Ministry of Gender and Family Promotion. 2010. National Gender Policy, July 2010. Republic of Rwanda. 2018. National Cooperative Policy in Rwanda Toward Private Cooperative Enterprises and Business Entities for Socio-Economic Transformation. Revised version [1]. Kigali, January 15, 2018. Republic of Rwanda. Ministry of Finance and Economic Planning. 2015. Vision 2050. Republic of Rwanda. 2019. Rwanda Voluntary National Review Report. Sachs, J., et al. 2018. SDG Index and Dashboards Report 2018. SDG Center for Africa and Sustainable Development Solutions Network. 2018. The State of Gender Equality in Rwanda from Transition to Transformation. Gender Monitoring Office, Republic of Rwanda. March 2019. UN Rwanda. 2018. UNDAP 2018-2023 for Rwanda. Signed 31 July 2018. UNDP. 2020. Human Development Report 2020: The Next Frontier Human Development and the Anthropocene. UNHCR. 2018. Burundi situation. October 2018. UNHCR. 2020. DRC situation. October 2020. Date | Report Number 154 United Nations Rwanda. 2017. Rwanda Common Country Analysis. Final report. United Nations Rwanda. 2018. UNDAP 2018-2023 for Rwanda. Signed 31 July 2018. United States Department of Agriculture, Foreign Agricultural Service. NDa. Foreign Agricultural Service Strategic Plan (2019-2022). United States Department of Agriculture, Foreign Agricultural Service. NDb. FY 2019 Food Assistance Proposal Guidance and Request for Applications. USAID. 2018. Rwanda: Nutrition Profile. USAID. History of USAID Rwanda. Accessed at https://www.usaid.gov/history-usaidrwanda. USDA McGovern-Dole Grant FFE-696-2015/007-00. WFP. 2017a. WFP Strategic Plan (2017-2021). WFP. Operation Evaluation: Rwanda, Common Country Programme, 200539, Midterm Evaluation (2013- 2016) WFP Rwanda. 2015. FY 2015 McGovern-Dole proposal. WFP Rwanda. 2020. School Feeding Newsletter. April-September 2020. WFP VAM. 2016. Rwanda 2015: CFSVA (Data collected in April-May 2015). WFP. 2014. Purchase for Progress - P4P Rwanda. WFP. 2015. WFP Rwanda FY 2015 McGovern-Dole proposal. WFP. 2016. Country Programme – Rwanda (2013-2018). SPR 2016. WFP. 2017. Country Programme - Rwanda - (2013-2018) SPR 2017. WFP. 2017d. WFP Rwanda Country Brief. November 2017. WFP. 2018. Draft Country Strategic Plan (2019-2023). WFP. 2018. MGD Semi-Annual Report (April 1, 2018 – September 30, 2018). WFP. 2018. Rwanda: Comprehensive Food Security and Vulnerability Analysis (CFSVA) 2018 (Data collected in March-April 2018). WFP. 2018. Rwanda: Comprehensive Food Security and Vulnerability Analysis (CFSVA). WFP. 2019. MGD Semi-Annual Report (April 1, 2019 – September 30, 2019). WFP. 2019. MGD Semi-Annual Report (October 1, 2018 – March 31, 2019). WFP. 2019. Rwanda Annual Country Report. WFP. 2020. Evaluation of USDA’s Local and Regional Food Aid Procurement Programme (Rwanda 2017-19). Endline. WFP. 2020. MGD Semi-Annual Report (April 1, 2020 – September 30, 2020). Date | Report Number 155 WFP. 2020. MGD Semi-Annual Report (October 1, 2019 – March 31, 2020). World Bank Group. 2021. Rwanda Overview. World Bank. Rwanda Economic Update. Protect and Promote Human Capital in a Post-COVID-19 World. Edition No. 16, January 2021. World Food Programme. 2019. Rwanda Annual Country Report 2020. Country Strategic Plan 2019-2023. World Vision. 2016. Rwanda Annual Report 2016. Date | Report Number 156 Acronyms ADRA Adventist Development and Relief Association ARI Acute Respiratory Infection CFSVA Comprehensive Food Security and Vulnerability Analysis CHAI Clinton Health Access Initiative CP4P Common P4P CRRF Comprehensive Refugee Response Framework CU5 Children under five years of age CVA Citizen Voice and Action DEO District Education Officer DEQAS Decentralized Evaluation Quality Assurance System EGRA Early Grade Reading Assessment ESWG Education Sector Working Group ET Evaluation Team FY Fiscal Year GAP Good Agricultural Practices GDP Gross Domestic Product GEWE Gender Equity and Women’s Empowerment GHI Gardens for Health International GII Gender Inequality Index HGSF Home Grown School Feeding HQ Headquarters IHW Integrated Health Week IR Inception report LEGRA Local Early Grade Reading Assessment LRP Local Regional Procurement Project MDG Millennium Development Goal MINAGRI Ministry of Agriculture and Animal Resources MINALOC Ministry of Local Affairs MINEDUC Ministry of Education MoU Memorandum of Understanding MTE Midterm evaluation NST National Strategy for Transformation NSFP National School Feeding Programme OECD-DAC Organisation for Economic Co-operation and Development - Development Assistance Committee OEV (WFP) Office of Evaluation P4P Purchase for Progress PPS Probability-proportional-to-size RBC Rwanda Biomedical Centre REB Rwanda Education Board SDG Sustainable Development Goal SGAC School General Assembly Committee SMC School Management Committee SO Strategic Objective THR Take-home ration TOR Terms of Reference TWG Technical Working Group UNDAP United Nations Development Assistance Programme USAID United States Agency for International Development USDA United States Department of Agriculture Date | Report Number 157 WASH Water, Sanitation and Hygiene WCPM Words correct per minute WFP World Food Programme Date | Report Number 158 World Food Programme Rwanda http://www1.wfp.org/countries/rwanda [Place, Month b6/download/ https://docs.wfp.org/api/documents/cdcbe4be57914c65b0dbcbd201196a and Year, Report number]