KYRGYZ REPUBLIC McGovern-Dole International Food for Education and Child Nutrition 2017 – 2021 Project Final Evaluation 1/5/2022 1 Final Evaluation Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-116-2017/024-00 Funding Year: Fiscal Year 2017 Project Duration: 2017-2021 Implemented by: MERCY CORPS KYRGYZSTAN Evaluation Authored by: ETHAN WILENSKY-LANFORD, PHD (CAUSAL DESIGN, INC.) SOPHIE TURNBULL (CAUSAL DESIGN, INC.) MATTHEW KLICK, PHD (CAUSAL DESIGN, INC.) ALEX PORTER (CAUSAL DESIGN, INC.) DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or of the United States Government. 2 Table of Contents List of Acronyms............................................................................................................................................ 5 Executive Summary....................................................................................................................................... 6 Project Background and Purpose.............................................................................................................. 6 Evaluation Questions, Design, Methods and Limitations......................................................................... 6 Key Findings and Conclusions................................................................................................................... 7 Recommendations.................................................................................................................................. 10 Lessons Learned...................................................................................................................................... 12 1. Introduction and Purpose ................................................................................................................... 14 1.1. Project Context .............................................................................................................................. 14 1.2. Project Description......................................................................................................................... 15 1.3. Results Framework......................................................................................................................... 15 1.4. Purpose of the Evaluation.............................................................................................................. 18 2. Evaluation Design and Methodology.................................................................................................. 18 2.1. Evaluation Questions ..................................................................................................................... 18 2.2. Evaluation Design........................................................................................................................... 19 2.3. Sampling Methods ......................................................................................................................... 20 2.4. Data Analysis Methods................................................................................................................... 26 2.5. Evaluation Limitations.................................................................................................................... 27 3. Findings............................................................................................................................................... 28 Relevance................................................................................................................................................ 29 Effectiveness........................................................................................................................................... 34 Impact ..................................................................................................................................................... 50 Efficiency ................................................................................................................................................. 61 Sustainability........................................................................................................................................... 71 5. Conclusions............................................................................................................................................. 81 6. Recommendations.................................................................................................................................. 86 Annexes....................................................................................................................................................... 95 Annex A: Indicators................................................................................................................................. 95 Annex B: Data Collection Instruments.................................................................................................. 102 Annex C: Terms of Reference................................................................................................................ 132 Annex D: Results Framework................................................................................................................ 140 Annex E: Program Schools Distribution by Region ............................................................................... 142 Annex F: Suggested Evaluation Methodology ...................................................................................... 142 Annex G: Suggested Roles and Responsibilities.................................................................................... 145 Annex H: Evaluation Criteria ................................................................................................................. 146 Annex I: Literacy Results....................................................................................................................... 148 Annex J: Observation Assessment ........................................................................................................ 157 Bibliography .............................................................................................................................................. 166 3 List of Figures Figure 1. Intersection of McGovern-Dole Legislation and Results Framework ..........................................17 Figure 2. Kitchen Infrastructure and Equipment ........................................................................................ 37 Figure 3. Total Number of Meals per Year by Province.............................................................................. 38 Figure 4. Installation and Functionality of MGD Infrastructure Components (collected during the final evaluation for a sample of schools)............................................................................................................ 44 Figure 5. Reading Status in the Language of Instruction (Baseline n=768, Endline n=718) .......................51 Figure 6. Child Age by Gender................................................................................................................... 150 Figure 7. Child’s Language of Instruction.................................................................................................. 150 Figure 8. Subtask Average Percentage Correct (n=718) ........................................................................... 151 Figure 9. Reading Status in the Language of Instruction (Baseline n=768, Endline n=718) .....................152 Figure 10. Reading Comprehension in Class Language (n=718) ...............................................................152 Figure 11. Number of MC-Provided Books in the Reading Corner...........................................................165 List of Tables Table 1. Intended Sample Selection Ratios by Province and Cohort.......................................................... 22 Table 2. Actual Count of Sampled Schools.................................................................................................. 22 Table 3. Literacy Test School Sample.......................................................................................................... 23 Table 4. Summary of Interviews and FGDs Conducted............................................................................... 24 Table 5. Performance Categories................................................................................................................ 28 Table 6. Monitoring Data - School Feeding Program (SFP)......................................................................... 35 Table 7. Daily school meals and school-age children receiving meals as a result of USDA assistance.......38 Table 8. Monitoring Data - Social and Behavior Change (SBC)................................................................... 40 Table 9. Monitoring Data - Winterization and WASH................................................................................. 43 Table 10. Monitoring Data - Literacy and Learning Promotion .................................................................. 44 Table 11. Midterm Recommendations and Current Status........................................................................ 64 Table 12. USDA Kyrgyz Republic MGD Findings, Conclusions, and Recommendations (FCR) Matrix ........86 Table 13. MC IPTT Indicator Results ........................................................................................................... 95 Table 14. Literacy Boost Literacy Subtasks............................................................................................... 148 Table 15. Literacy Boost Background Questions....................................................................................... 149 Table 16. Geographic Distribution of Sample ........................................................................................... 149 Table 17. Literacy Test Results by Gender................................................................................................ 153 Table 18. Literacy Accuracy and Reading Comprehension by Gender – Kyrgyz.......................................153 Table 19. Literacy Accuracy and Reading Comprehension by Gender – Russian .....................................153 Table 20. Reading Materials by Gender.................................................................................................... 154 Table 21. Literacy Test Results by Language of School Instruction ..........................................................154 Table 22. Literacy Accuracy and Reading Comprehension in Language and Language of School Instruction ..................................................................................................................................... 154 Table 23. Reading Materials by Language of School Instruction ..............................................................155 Table 24. Average Literacy Test Results by Province................................................................................ 155 Table 25. Literacy Accuracy and Reading Comprehension by province – Kyrgyz.....................................156 Table 26. Literary Accuracy and Reading Comprehension by Province – Russian....................................156 4 Table 27. Average Number of Reading Materials by province ................................................................. 156 Table 28. Sample Regions......................................................................................................................... 157 Table 29. What Mercy Corps Interventions Took Place at the School?....................................................157 Table 30. Schools that Received an Infrastructure Grant from Mercy Corps and Type of Grant.............158 Table 31. Are all windows in working order at the time of the visit?.......................................................158 Table 32. Were the doors installed (via grant from Mercy Corps) correctly? ..........................................158 Table 33. Are all doors in working order at the time of the visit? ............................................................159 Table 34. If the heating system was installed or repaired with a grant from Mercy Corps, is it in working order at the time of the visit?................................................................................................................... 159 Table 35. If the roof was repaired with a grant from Mercy Corps, have there been any issues with it since? ............................................................................................................................................. 159 Table 36. If a kitchen ventilation system was installed with a grant from Mercy Corps, have there been any issues with it?............................................................................................................................................ 159 Table 37. School Storage Room Conditions.............................................................................................. 159 Table 38. Equipment Present at Sampled Schools ................................................................................... 160 Table 39. Are there enough chairs for all students in each group that eats at the same time? ..............160 Table 40. Are Mercy Corps visibility and program materials present?.....................................................161 Table 41.Handwashing Facilities in Dining Hall......................................................................................... 161 Table 42. Does all new donated equipment have inventory numbers to indicate that it was registered? ................................................................................................................................................ 161 Table 43. Does the school provide full hot meals five days per week? ....................................................161 Table 44. Kitchen Maintenance: Ventilation, Water Supply, and Cleaning Schedule ..............................162 Table 45. Kitchen Hygiene Behaviors........................................................................................................ 162 Table 46. Types of Latrines Used by Students.......................................................................................... 162 Table 47. Student Latrine Conditions........................................................................................................ 163 Table 48. Conditions in Latrines Repaired or Built by Mercy Corps (at the time of observation)............163 Table 49. Handwashing Facility Conditions .............................................................................................. 163 Table 50. Water Filtration......................................................................................................................... 164 Table 51. Are additional books given to students to borrow and take home? ........................................165 5 List of Acronyms Acronym Full Term AO Ayil Okmotu (local self-government executive body) CARM Community Accountability and Response Mechanism COI Conflict of Interest DDoE District-level Department of Education of the Kyrgyz Republic FCR Findings, Conclusions, and Recommendations Matrix FFE Food for Education FGD Focused Group Discussion GoK Government of Kyrgyzstan IDI In-Depth Interview KGS Kyrgyz som (currency) KII Key Informant Interview MC Mercy Corps MGD/MGD17 McGovern-Dole Food for Education and Child Nutrition Program 2017 - 2021 MoES Ministry of Education and Science of the Kyrgyz Republic MoH Ministry of Health of the Kyrgyz Republic MTE Mid-Term Evaluation OECD-DAC Organization for Economic Co-Operation and Development’s Development Assistance Committee PLWD People Living with Disabilities PMP Performance Monitoring Plan PTA Parent Teacher Association SA School Administrator SBC Social Behavioral Change SFP School Feeding Program TOR Terms of Reference TTR Time to Read Program TVET Technical, Vocational Education and Training USAID United States Agency for International Development USDA United States Department of Agriculture WFP World Food Program 6 Executive Summary Project Background and Purpose The McGovern-Dole International Food for Education and Child Nutrition Program 2017 (FFE17 or FFE) helps support education, child development, and food security in low-income, food-deficit countries around the globe. The program utilizes the donation of U.S. agricultural commodities, as well as financial and technical assistance, to support school feeding and maternal and child nutrition projects. In the Kyrgyz Republic specifically, MGD is designed to support the National School Feeding Program developed by the Ministry of Education and Science (MoES), while the 2017 support includes additional components targeting literacy and learning promotion. Thus, in Kyrgyzstan, MGD consists of a suite of overlapping programs that address issues of access, school and learning environment, and community involvement. Interventions, more specifically, included the provision of prepared meals in schools, new books, teacher training in Positive Discipline, and critical facility upgrades. The result is a nutrition and literacy program that is bolstered by a phalanx of stakeholders working in concert to build a positive learning environment for students. Evaluation Questions, Design, Methods, and Limitations This final evaluation is designed around a non-experimental performance evaluation model. Extensive qualitative inquiry, with the variety of stakeholders mentioned above and with national government and donor respondents and parents across all seven provinces, constitutes the backbone of this evaluation approach. Carefully designed tools guided the qualitative inquiry and the coding of responses. Coded responses, which were cross-tabulated by cohort, region, and actor-type, were ultimately distilled to answer evaluation questions. Evaluation questions largely map the Organization for Economic Cooperation and Development, Development Assistance Committee (OECD-DAC) evaluation criteria: Relevance, Effectiveness, Efficiency, Impact, and Sustainability. "Coherence," which is a relatively recent addition to these core criteria, was not explicitly examined, but findings related to this topic are included throughout the report where appropriate. Additional evaluation questions explore the effects of COVID-19 on the program. Concurrent to the extensive qualitative inquiry, a string of quantitative measurements was examined, including a literacy test results. The same test was conducted at baseline, and, where possible, differences between baseline and endline are reported. Results do not constitute a statistically valid impact evaluation, though this was never anticipated in the final evaluation. Instead, the literacy test results provide a snapshot of current student performance. Monitoring data was also tabulated and reported in the index in detail. Finally, the results of an on-site observation checklist, designed to gauge the quality of facility upgrades, maintenance, adherence to hygiene, and other best practices, was also included. These results are also reported in the Annex J and provide a summary snapshot of the state of participating schools. Limitations center mostly on access to an indicator map. In the transition between implementers, data sources, definitions, and formatting have changed to a degree that made assessing monitoring data onerous, with the possibility that some inaccuracies may still be present. For example, data on provision of support to parents during the first year was not available as it was not shared between partners and could not be traced and verified by Causal Design. In other instances, source data was not clearly formatted or labelled to produce the necessary indicators. 7 Other limitations, which are discussed in the body of the report, are mostly within the bounds of operating and conducting field work in a challenging context, including rural, mountainous regions, and schools spanning all seven provinces. Despite COVID-19 and some delays in data collection, this component was concluded successfully. Key Findings and Conclusions Relevance Conclusion Rating Observations The data emerging from the final evaluation reinforce the midterm evaluation findings, which resoundingly highlight the relevance and importance of FFE17. In short, the program is appropriately designed to meet its core objectives and is positively viewed by stakeholders at all levels. Satisfactory • The school feeding component of McGovern-Dole 2017 remains highly relevant and in line with the GoK policies and strategic aims. • The Kyrgyz School Feeding Policy was passed in December 2019. • The project emphasizes teaching teachers positive discipline techniques and increasing access to reading materials and literacy. SBC activities are highly appreciated elements of the MGD design. • As stated during the midline, the fact that the program does not teach improved literacy techniques to teachers and that TTR is working in the same schools on literacy, definitive attribution for the learning outcomes is impossible unless there is a comparative quasi￾experimental design employed. ● The program design has been adapted and modified from the outset to ensure its appropriateness and relevance to institutional and individual participants. Effectiveness Conclusion Rating Observations The MGD17 program is a complex multi-sectoral program; the data shows highly effective performance across all of the outputs related to the SFP/enabling learning environment component and the second Literacy and learning promotion components. Despite the school closures due to the pandemic, MGD17 was able to meet or surpass all of its pre￾established objectives except those linked to meals served to students. Overall, the program took a comprehensive approach Highly Satisfactory • 108% percent of target schools have received all the requisite school infrastructure and equipment inputs for kitchen rehabilitation. • As it relates to the improved water sources, over 100% of the targets have been reached. • Most (75% or above kitchen equipment and materials were functioning at time of field visit. • In terms of training the school staff for SFP, including safe food preparation, management of commodities, and water systems, the McGovern-Dole Monitoring data highlights that all targets have been surpassed substantially relative to these variables. • All winterization and WASH infrastructure targets were exceeded. • Mercy Corps received a modification from USDA that allowed them to engage firms directly to build the latrines based on international standards latrine construction. 8 to SFP that addressed commodities and training and sought successfully to create the environment required to provide students with nutritious, hygienic, and healthy hot meals five days a week. • MGD17 also made distributions of household rations to the families of participating primary school children. ● MGD17 also surpassed all targets linked to literacy and learning promotion, including extra-curricular activities, positive discipline training for teachers and the creation of reading corners. Impact Conclusion Rating Observations The data highlight an 8- percentage point increase in the number of reads with comprehension, while the non￾readers remain at 5%. While this is a single-digit improvement in the reading comprehension results from the baseline, these results are promising in the backdrop of the ongoing pandemic and school closures. They suggest that the efforts of the GoK and MGD17 are reversing the years of malnutrition that had previously undermined the performance of school-age children in Kyrgyzstan. Satisfactory • Evaluation, monitoring, and independent household study evidence suggest that there has been significant progress in improving health, nutrition, and dietary practices at school and in the home. • The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. • MGD17 monitoring for 2020-2021 shows that 87% of program schools have a diverse school menu (i.e., the same meal is not repeated more than twice per month, and all food groups are included). • MGD17 monitoring for 2020-2021 shows that 78% of program schools that have school feeding (complete meals) five days per week • MGD17 monitoring for 2020-2021 shows that 57% of program schools have regular (monthly) financial contributions from parents. ● The program performance and impact suggest that while the pandemic presented challenges to program implementation, it did not block the MGD17 from achieving its program outcomes related to SFP, literacy promotion, and learning outcomes for primary school children at participating schools. Efficiency Conclusion Rating Observations McGovern-Dole 2017 performed well across all the dimensions of efficiency. The evaluation data shows solid financial and programmatic management performance. The MGD17 team and USDA applied its extraordinarily adaptive and innovative leadership style to Highly Satisfactory • The program made multiple programmatic and management adjustments to ensure program success and efficiency, including everything from early revision to the program design, adjusting the infrastructure approach repeatedly, and adjusting program components to respond to the ongoing pandemic. • MGD17 leadership and USDA maintained program implementation during the pandemic and added household rations to primary school students' families, online training and activities for the SFC, and literacy promotion components. The program also amped up the 9 ensure the program was managed for results and address emerging needs and crises like the pandemic. WASH component to encourage the school's reopening and fight the spread of COVID-19. • The program performed well in terms of financial and personnel management efficiency. • Most stakeholders positively view the program oversight and management processes. ● Lastly, most if not all feedback emerging from the midterm evaluation has been addressed or completed despite delays caused by the pandemic. The midterm recommendations result from the combined feedback and recommendations from stakeholders across the program. Efficiency Conclusion Rating Observations Fundamentally GoK and its partners USDA, MC, and the WFP have put in place the basic structures to support sustainability at the national level, with the passage of the National Law on School Feeding, along with the provision of the 7 or 10 KGS per student per day provide the legal and financial framework for school feeding to continue nationwide. However, the data suggests that in order for schools to provide and sustain a diverse, nutritional daily diet for each student, increased resources are necessary. Moreover, the legal and financial framework for the continuation of the literacy and learning promotion activities are not in place to ensure the continuity of these activities. Moderately Unsatisfactory • Though the National Law on School Feeding bodes well for sustainability, the amount of money allocated under this law (7 to 10 KGS per day per student) is not considered sufficient to provide students with an acceptably diverse diet. Evidence to indicate the long￾term sustainability of government support for improved literacy instruction practices is yet more sparse. The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. • Many school staff expressed intent or desire to continue school feeding but noted they had no or minimal capacity to do so. • Regarding the responsibility for ensuring the sustainability of the hot meal programs at MGD17 schools there seems to be a tension between whether the responsibility falls to the parents or the government. • The data suggests that given the lack of budget and the fact that most of the literacy promotion work with the exception of the positive discipline are considered extra￾curricular, there is no certainty regarding the ability of these stakeholders to prioritize the literacy and learning promotion activities. • The findings illustrate that from the perspective of the school administration staff, there is no consistent transformation of the collaboration and information sharing between the tripartite of actors that is attributable to MGD17. • Because of resource constraints and the importance of behaviour change at the institutional level, all actors, including the school administration, local authorities and 10 communities, and parents, must collaborate to support SFP success. ● This pandemic and the economic crisis resulting in increasing the cost of commodities and diminishing the capacity of parents to subsidize school feeding after the support from Mercy Corps draws to a close, which has the potential to diminish program sustainability in general because even the government budget will not provide as many commodities as it did before the pandemic. Recommendations Relevant Recommendations Timeline and Prioritization Duty Bearer 1. GoK, MC, and USDA should meet to explore and consider the best approach to the literacy component in the future, particularly exploring whether traditional literacy training and positive discipline will be combined or whether the positive discipline approach will be abandoned. Given the recent literacy study results, there may be evidence that a combined approach is ideal. ASAP GoK, MC and USDA 2. USDA, GoK, WFP, and MC should hold a symposium to explore the issue of sustainability of school feeding programs to fully explore what design components are required in order to promote the sustainability and independence of the GoKs SFP programs in graduated schools, especially in light of the current economic issues in the country. ASAP GoK, WFP, MC, and USDA 3. USDA, GoK, and MC should develop a white paper regarding lessons learned from the flexible and adaptable approach to implementation employed during the Implementation of MGD17. ASAP GoK, MC, and USDA Effectiveness Recommendation Timeline and Prioritization Duty Bearer 1. Current MGD17 data highlights that the indicator for meals served was not achieved; however, it is not clear if the household level rations for the families of participating primary school children were considered in this calculation. If not, some consideration for these distributions and the additional activities undertaken should be reflected in the MC PMP and performance assessments to demonstrate the quality of the effectiveness and management demonstrated by the MGD17 management and oversight efforts. ASAP MC & USDA 11 2. If not already done, the program should share with the GoK the infrastructure and needs assessment framework to integrate this into their planning and prioritization standard operating practices should they so choose. End of 2022 MC, USDA & GoK 3. The MGD17 and USDA should develop guidance on delivering effective online training, SBC activities for child and adult learners participating in the development and/or quasi-humanitarian programming. End of 2020 MC, USDA and GoK Impact Recommendations Timeline and Prioritization Duty Bearer 1. To fully explore the impact of the MGD17 program, a quasi￾experimental design is required that compares the performance of MGD 17 schools with WFP and schools that do not have hot meal programs. MC should work with GoK to explore the possibility of conducting this type of study. There should also be control for TTR and related literacy programming to ensure that the true value-add of the MGD17 holistic approach is captured. March 2022 MC, USDA & GoK 2. An internal study of MGD12 & MGD17 graduated schools to explore the continued impact and sustainability would also be beneficial for exploring lessons learned and best practices for ongoing and future programming. There should be some attempt to explore the impact of the ongoing pandemic and how to support the GoK and local communities to find solutions. End of 2022 MC, GoK, and USDA Efficiency Recommendations Timeline and Prioritization Duty Bearer Given MGD17's and USDA's experience and success with program continuation and remote/virtual activities during a pandemic, MC and USDA should develop guidance documents and best practice materials that can serve as guidance and training materials for related programming and staffing to prepare for future shutdowns and similar situations. End of 2022 MC, USDA, and GoK Sustainability Recommendations Timeline and Prioritization Duty Bearer 1. MC, Gok, and USDA need to develop an exit and handover strategy for each graduated school that is specific to their needs, context, and circumstances. End of 2022 MC, USDA, & GoK 2. The exit strategy development should also explore the formalization of the communication and collaboration between schools, AO, and DDoE following the departure of MC. End of 2022 MC, GoK, and USDA 12 3. GoK, MC and USDA should consider working together to develop school-level sustainability strategies that extend beyond parent contribution and subsidies. June 2022 MC, GoK, and USDA 4. The literacy promotion components, particularly positive discipline, require advocacy to ensure a normative or policy adoption that will promote its sustainability and integration beyond the closeout of MGD17. End of 2022 MC, GoK, and USDA 5. MC, USDA, and GoK should explore systematic support for the introduction of school orchards/gardens in schools that do not already have them. June 2022 MC, GoK, and USDA Lessons Learned Relevance: Lessons Learned 1. The midterm and final evaluations highlight that this has been an adaptable program. Mercy Corps, GoK, and USDA have piloted and modified the program approach throughout implementation to improve relevance, impact, and meet participant needs at the institutional and student level. This flexible and adaptable approach to program design and implementation for success and impact should be lauded and replicated. 2. While the program is appropriate and relevant, sustainability remains a critical issue for all the MGD 2012 and 2017 schools. Effectiveness: Lessons Learned 1. The MGD17 SBC approach was part of an effective strategy to encourage very close involvement and partnership with local government and local communities. It is a best practice in all development programs that seek to extend behaviour change to the household or community levels. 2. The MGD17 donors and management team continually and consistently adjusted program design and implementation to respond to emerging needs or to be more successful and impactful; it is a best practice that should be applied more widely to ensure that programs remain relevant and ensure impact as circumstances/ conditions change or new data becomes available. This includes gender studies, household-level studies, and other unplanned activities supporting program success. Impact Lessons Learned 1. The SBC component has had a multiplying effect on the program's impact by changing behaviours in the school setting and participating children's households. 2. While the pandemic did cause some challenges to the program implementation, USDA, the MGD17 team, and the GoK were able to adapt the implementation and programming approach to ensure that the program was able to achieve its outputs and outcomes. Efficiency Lessons Learned 1. The flexible and adaptive implementation style employed at the outset and throughout MGD17 by USDA and MC permitted the program to innovate, ensure effectiveness, relevance, and impact, and 13 ultimately enable the program to successfully continue program implementation during the pandemic when many other projects halted or stopped implementation. 2. The data-driven and needs-based adjustments and modifications made to the SFP, literacy, infrastructure, and SBC components throughout the life of the award were critical to achieving the results and successes resulting from the 5-year MGD17 implementation. Sustainability Lessons Learned 1. The reliance on parental contributions and subsidies as a sustainability strategy for SFP in MGD17 schools in some program communities is simply not viable, and given the economic strife caused by the pandemic and rising commodity prices, these risks putting the sustainability of the SFP at all graduated schools in jeopardy. 2. Intention does not equal ability due to the financial implications of sustaining the SFP programs and the differing circumstances impacting each school and community; multiple and differing strategies are required to sustain SFP programs. 14 1. Introduction and Purpose 1.1. Project Context The McGovern-Dole International Food for Education and Child Nutrition Program (MGD) helps support education, child development and food security in low-income, food-deficit countries around the globe. The program utilizes the donation of U.S. agricultural commodities, as well as financial and technical assistance, to support school feeding and maternal and child nutrition projects. In the Kyrgyz Republic (Kyrgyzstan hereafter), specifically, MGD is designed to support the National School Feeding Program developed by the Ministry of Education and Science (MoES), while the 2017 support includes additional components targeting literacy and learning promotion. While Kyrgyzstan as a whole has made significant strides in both nutrition and education since independence, and remains the most democratic country in the region, the country has struggled for years with modernizing elements of its economy, including the agricultural sector, which is the backbone of rural regions and their economies in particular. Labor out-migration, consequently, to Russia and Kazakhstan, among other destinations, has been significant. The country has experienced significant economic and political tumult in recent years, which has had an impact on programming and coordination with government officials, due to high levels of turnover and shifting priorities. Most acutely, after a series of unlikely events, the disputed October 4, 2020 parliamentary elections resulted in Sadyr Japarov, a formerly imprisoned Kyrgyz politician, progressing from prisoner, to interim prime minister, to ultimately being elected as president, as of January 10, 2021. In April 2021, voters went on to approve dramatic constitutional changes that cemented Kyrgyzstan’s transformation from a parliamentary democracy to a state in which power is strongly vested in the president. Multiple outside observers, watchdogs, and foreign ministries have expressed their alarm at the extent of the changes and the perceived democratic backsliding. These dramatic events, which provoked significant street protests in the capital, Bishkek, took place against a backdrop of a global pandemic. While Kyrgyzstan appears to have largely avoided the strong surges of COVID-19 experienced elsewhere, travel restrictions and intermittent school closures have persisted into the spring of 2021, including during the data collection period for this activity. Inflation, including significant hikes in the price of food, has presented an additional burden for families and households. Demographic trends have, furthermore, resulted in a relative youth bubble and increasing classroom sizes. Even more recently, Tajik-Kyrgyz violence erupted in late April of 2021 in the Batken region, resulting in destroyed property, significant injuries, and 46 casualties. The fighting, which stemmed from a dispute over access to water facilities in a contentious border region, resulted in closed and destroyed schools. The violence, while tragic, did not directly affect sampling for this activity, and further underscores the ongoing challenges for any program dedicated to augmenting health and education in this evolving and fraught context. The events described above have affected MGD programming, at times dramatically. The endline evaluation, it should be cautioned, occurred during and immediately after this slew of events. It must be 15 noted that, while programmatic and implementation challenges are described below, participants, stakeholders, and students themselves demonstrated impressive creativity and resilience, which this report also attests to. 1.2. Project Description Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education (FFE) and Child Nutrition program in Kyrgyzstan since 2003. The program provides nutritional and technical support to education institutions, including public schools, kindergartens, and vocational schools. This program is implemented in support of the National School Feeding Program developed, as noted above, by MoES, which has been both a partner and key stakeholder throughout project implementation. The total budget for the 2017-2021 MGD program was $15,738,451.39. In 2017-2021, Mercy Corps supported 440 rural schools across the country by providing over 2,000 metric tons of supplementary commodities (enriched flour, lentils, rice, split peas, and vegetable oil). More than 90,000 primary grade students benefited from meals in schools prepared from these commodities. In addition to school feeding, the program includes the following complementary components, though these vary in intensity and scope, across the country: ● Literacy Promotion: Provision of supplementary reading materials to public schools and libraries, Positive Discipline training for primary grade teachers, as well as organization of out-of-school and community-based reading activities; ● Infrastructure Rehabilitation: To support winterization of schools (repair or replacement of old windows, roofs, heating systems, etc.); ● Sanitation and Hygiene: Installation of handwashing stations, repair or construction of improved school latrines; ● Capacity Building: Tailored training for school administration on project management and budgeting skills, improved knowledge on nutrition and hygiene in schools, etc.; ● Social and Behavior Change Communication (SBCC): Introducing improved nutrition and hygiene behaviors to the target communities through a network of trained community-based change agents. ● Potable Water Intervention: Improving access to safe drinking water by improving the water supply in target schools through identification and tailoring of ideal filtration/treatment solutions. This report presents the final performance evaluation of the 2017 Kyrgyz Republic MGD program. Its purpose, components, and findings are discussed in subsequent sections. 1.3. Results Framework MGD operates under two mutually reinforcing strategic objectives: to improve literacy of school-aged children and to enhance the use of healthy dietary practices (see Figure 1). Ancillary components bolster these objectives through the inclusion of supportive programming—the enhancement of effective infrastructure, including water, sanitation, and hygiene (WASH) facilities, and social and behavioral change reinforcement through the deployment of trained community members. The Theory of Change, in short, is that: 16 …providing a conducive school environment, reducing hunger and preventing illness, students will be able to participate more fully in class, resulting in better learning outcomes1 … While the evaluation team was unable to identify explicit assumptions, it is clear that some basic assumptions must hold to maximize the efficacy of MGD programming. These might include, for example, a regular school schedule, uninterrupted by long closures, as were experienced this past year. Even under such extraordinary circumstances, the program would need to assume mostly uniform access to the internet. Other relevant assumptions might include access to maintenance services, for new facilities, appliances, or upgraded infrastructure; minimal turnover of trained teachers and librarians; and regular participation of parents and the community. As these factors inherently vary, so too may program effectiveness and impact. To the extent that data allows, some of these assumptions are explored under Relevance in the Findings section. 1 https://docs.wfp.org/api/documents/WFP-0000068787/download/ 17 Figure 1. Intersection of McGovern-Dole Legislation and Results Framework2 2 From USDA (2017) Report to Congress: The Global Effort to Reduce Child Hunger and Increase School Attendance, Fiscal Year 2017, Source: https://www.fas.usda.gov/sites/default/files/2020-04/mcgovern-dole-fy2017.pdf 18 1.4. Purpose of the Evaluation This report serves as an endline program evaluation of the USDA Kyrgyz Republic MGD program, implemented by Mercy Corps throughout all seven provinces of Kyrgyzstan. The purpose of the evaluation is to: ● Assess and provide evidence on the performance and results of the program against set objectives and targets; ● Highlight program achievements and successes supported by evidence gathered during this evaluation; ● Evaluate the degree of school feeding sustainability in target schools after several continuous years of program implementation; ● Identify key lessons learned and provide specific, actionable, and evidence-based recommendations for the ongoing and future McGovern-Dole programs; and ● Assess how the achievements are viewed by program stakeholders: MoES officials, school staff, local community, parents of primary grade students, and local governments. The evaluation is broadly organized along the lines of the Organization for Economic Co-operation and Development-Development Assistance Committee (OECD-DAC) evaluation criteria,3 with overarching evaluation questions organized around Relevance, Effectiveness, Efficiency, Impact, and Sustainability (see following section). Given the influence of COVID-19 related school closures, lockdowns, and household ramifications an additional line of questioning examines specifically how COVID-19 has affected schools, their personnel, students, and parents. Evidence from this study and program reports suggests that, while the impact of COVID-19 is potentially significant, stakeholders have managed to develop creative workarounds to closed classrooms through the use of WhatsApp groups, audio books, and online reading programs, among other solutions. Both the positive and negative effects are examined through a combination of qualitative inquiry, on-site observation, and a literacy test, discussed in greater detail in the Evaluation Design and Methodology section. 2. Evaluation Design and Methodology 2.1. Evaluation Questions Mercy Corps established that the endline evaluation should answer the following 17 evaluation questions with the focus on sustainability of the intervention: Relevance 1. Was the program’s design an appropriate solution to meet the stated objectives and results? 2. How did stakeholders perceive the program’s design and its activities? 3. To what degree was program implementation aligned with the goals and objectives of the relevant government ministries (MoES, MoH)? Effectiveness 4. Did the program meet indicator targets? 3 See, for reference: http://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf 19 5. How do program staff and stakeholders perceive the effectiveness of the processes and resources used by the program? 6. Did program activities show signs of creating unintended positive or negative outcomes? If yes, which activities contribute to this? Efficiency 7. Have any programmatic or financial adjustments been made during the course of the program? Why? If so, were they appropriate to the situation? 8. To what degree was feedback from the previous evaluations and stakeholder input incorporated into the program? Impact 9. To what degree have there been changes in each of the following that are attributable to the program: (a) health, nutrition, and dietary practices at the schools; (b) government financial support for literacy and school feeding activities, including local government; (c) nutrition behaviors of families for children at home? 10. To the degree that attribution is possible, what combination of activities is responsible for the above impacts? Sustainability 11. To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue primary school feeding beyond the program? 12. To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue improved literacy instruction practices beyond the program? 13. Have the program activities contributed towards improved information sharing between primary schools and local and national MoES officials? 14. What key factors have contributed to the variation in target schools’ degree of feeding program success as measured through dietary diversity, continuous community support, and other aspects of program implementation? COVID-19 Impact 15. To what extent has the COVID-19 pandemic and associated disruptions in education, such as school closure, affected effectiveness and sustainability of school feeding programs established under the program? 16. To what extent has the COVID-19 pandemic and associated disruptions affected program outcomes for other components? 17. To what extent has Mercy Corps been able to adapt program implementation during the COVID￾19 pandemic? 2.2. Evaluation Design This evaluation exercise is a non-experimental performance endline evaluation. The evaluation was designed to address the evaluation questions, spanning the themes of relevance, effectiveness, efficiency, impact sustainability, and COVID-19 effects. The evaluation, therefore, leverages qualitative responses, detail, and rich description, to draw linkages between programming quality, implementation, and context whenever appropriate, and “outcomes,” which will also be gauged qualitatively. Additionally, the report includes an analysis of literacy promotion indicator data, a quantitative analysis of an administered literacy assessment test. Finally, the report includes an assessment of an administered observation 20 component, which examines the presence and condition of infrastructure and facility upgrades resulting in a more mixed-methods approach. The approach to the qualitative data most closely resembles a within-case study methodology in its broad sweep of relevant data organized by overlapping evaluation questions, while not attempting to explicitly determine causality or explain a phenomenon. That said, elements of contribution analysis, 4 complemented by an inherent comparative component (with data stemming from like-actors, and a theoretically equivalent intervention, from diverse regions within Kyrgyzstan) is leveraged to the extent that it assists with analysis and provides more nuanced findings. Contribution analysis, however, should not be confused with yielding attribution, or “impact” in any statistical, or conclusive sense. Causal Design evaluators intentionally coded qualitative data and assessed patterns, trends, and (in)consistencies in order to address evaluation questions, with limited insights into attribution. Coding is discussed in the Data Analysis Methods section. Contribution Analysis Qualitative data can be analyzed through a structured process that assists with parsing often complex, nuanced, and messy text and feedback into findings. Contribution analysis is utilized, to the extent possible and appropriate, given limitations, to organize this process. Contribution analysis is appropriate in this case in that it strives less to provide attribution, but rather addresses questions that “are concerned with the role different actors played in supporting the observed outcome, or enabling conditions for success.”5 By identifying outcomes first, and then leveraging data to determine how and where evidence maps to a Theory of Change, if possible, and how evidence supports a “contribution story,” a plausible linkage between programming and outcomes can be distilled. In practice, however, the full, iterative extent of contribution analysis—in which the Theory of Change is fully interrogated and alternative explanations fully explored through additional field work—cannot be deployed in this study because of time constraints, and the inability to travel to field sites. The evaluation team has taken care to be explicit about the levels of analysis in which “contribution stories” are bound, the possibility of alternative scenarios and explanations, and where data limitations inhibit more precise findings. The extent and diversity of qualitative inputs, nevertheless, allow for a comprehensive assessment and endline evaluation. 2.3. Sampling Methods KII and FGD Sampling strategy Causal Design used a purposive sampling strategy to ensure breadth of geographical (province/district) and school size representation across KIIs and FGDs among the 440 schools in the MGD intervention 4 See for example: Mayne, J. (2008). Contribution analysis: An approach to exploring cause and effect. International Learning and Change (ILAC) Brief, 16 5 Hopkins, Laura (2021). Tools and Tips for Implementing Contribution Analysis: A Quick Guide for Practitioners (see: https://www.itad.com/wp-content/uploads/2021/03/Contribution-Analysis-Report.pdf) 21 sampling frame. The sampling selection process included selecting 70 schools (a little over 15 percent of all MGD program schools) for KIIs and ten schools for FGDs based on the following selection criteria: 1. Selection for the Mid-Term Evaluation (MTE) a. Causal Design eliminated the 78 schools that were surveyed for the MTE from the total sampling frame to minimize the possibility of selection bias from the MTE and to achieve a broad spread of school evaluation between the MTE and final evaluation. i. In the 2018 cohort (see point #2, below) in Talas, both schools in the program were selected for the MTE. As such, Causal Design selected one school that was also covered in the MTE for this province/cohort. 2. Cohort Coverage a. Given that the MGD program intervened in different schools across different cohort years (2017, 2018, 2019), Causal Design selected 15 percent of schools from each cohort to ensure proportional coverage across cohorts. 3. Provincial/District Coverage a. To ensure geographical coverage across cohorts, Causal Design calculated the portion of program schools in each province across the three cohorts and applied that percentage to the total required sampled by cohort. i. Causal Design added an additional school in Issyk-Kul and Naryn for the 2017 cohort and in Talas for the 2018 cohort to ensure that at least one school in each province was sampled across each cohort. ii. Causal Design added one additional school in Osh for the 2019 cohort to account for a rounding issue and ensure that the sample reached 70 schools. b. Causal Design then selected one school in each district in each province (where possible), to ensure coverage across the province. 4. Number of Primary Students Across Schools a. After ensuring geographic coverage, Causal Design also selected schools based on primary student enrollment size, with the aim of covering a spread of school sizes across cohorts and districts. 5. FGD Selection a. FGDs were selected based on the same criteria as above, but from the 70-school subsample. The evaluation team selected 3 FGD schools in each cohort (4 for the 2019 cohort, as it is the biggest) using the following process: i. Causal Design selected one school per province and two schools in provinces with the most program schools (Chui, Osh, and Jalal-Abad). ii. Causal Design then ensured proportional coverage across cohorts by selecting the additional three schools (across Chui, Osh, and Jalal-Abad) based on cohort size. One additional school was selected from the 2018 cohort (Jalal-Abad) and two schools were selected from the 2019 cohort (Osh, Chui). iii. Causal Design then selected the additional schools from different districts in Chui, Osh, and Jalal-Abad. iv. Finally, Causal Design selected schools with some variance in primary enrollment size. 22 Table 1. Intended Sample Selection Ratios by Province and Cohort Province 2017 Cohort 2018 Cohort 2019 Cohort Total Chui 2 9* 11* 22 Talas 3* 1^ 1 5 Issyk-Kul 1^ 1 2* 4 Naryn 1^ 2 2 5 Osh 3* 3 8^* 14 Jalal-Abad 2 5* 8* 15 Batken 2* 1 2 5 Total 14 22 34 70 ^Added to ensure one in each province/cohort and one additional for Osh 2019 Cohort to resolve rounding issue *Selected for FGD There were some small changes between what was planned and what was actually sampled because of field logistics. These changes are included in Table 2 (with reasons for any changes denoted). Table 2. Actual Count of Sampled Schools Province 2017 Cohort 2018 Cohort 2019 Cohort Total Chui 3* 9 11 23 Talas 3 1 1 5 Issyk-Kul 1 1 2 4 Naryn 1 2 2 5 Osh 3 3 8 14 Jalal-Abad 2 5 7** 14 Batken 2 1 2 5 Total 15 22 33 70 *One school added to Chui schools in the 2017 cohort from the intended sample set. **One school removed from Jalal-Abad schools in 2019 cohort from intended sample set, due to COVID-19 related closures. Causal Design, with assistance from Mercy Corps program staff, used a census selection approach to select pertinent interviewees for province and national-level KIIs with government stakeholders and other NGO actors. Due to logistical considerations — the FGD team enumerators also interviewed provincial level 23 actors — Casual Design worked with Erfolg Consult (the data collection team, discussed more below) to ensure that the district-level DDoE and local government interviews were somewhat aligned with FGD locations. Where it was logistically and feasibly possible, Causal Design prioritized maintaining even gender distribution across FGDs and separating FGDs by gender. Causal Design also attempted, wherever possible, to keep an even gender split in the selection of KII respondents at the national, provincial, district, and school levels. Perfect gender parity was not accomplished, however, as the overwhelming majority of school-based staff and FGD participants (SBC activists and otherwise active parents) were women. Literacy Test Sampling Strategy Causal Design replicated the sample for the literacy test used during the baseline assessment and reports produced by Save the Children, using the same schools and a new cross-sectional survey of Grade 2 students. A list of schools is not included, intentionally, for purposes of de-identification, but is available. The sample for this assessment includes 718 Grade 2 students from a representative sample of 48 schools included in the first phase of Save the Children’s programming of the USDA MGD program in the Kyrgyz Republic. The 48 schools were resurveyed in the endline from the 50 schools included in the Baseline Report with two schools not included in the endline due to overlap with the Time To Read (TTR) program. TTR is a nationwide USAID-funded reading promotion program which works to improve literacy instruction in schools, distributes age-appropriate reading materials, and engages parents and families to literacy at home.6 To draw the initial sample, the baseline randomly selected the 50 schools from the 136 schools included in Phase 1 of Save the Children’s component which were stratified by Northern and Southern provinces. The baseline research team apportioned 20 Northern schools and 30 Southern schools to account for the larger average size of schools in the South (Osh, Jalal-Abad, and Batken). Table 3 presents a breakdown of the sample by province (province). In ex-post power calculations, the endline survey was statistically powered to reveal reading comprehension scores plus or minus 1.2 percentage points at the 95 percent level of confidence. Table 3. Literacy Test School Sample Province Schools Children Batken 5 80 Chui 14 195 Issyk-Kul 1 16 Jalal-Abad 15 240 Naryn 1 16 6 https://www.usaid.gov/kyrgyz-republic/fact-sheets/time-read-project 24 Osh 10 160 Talas 4 61 Total 50 768 Within each school, the enumeration team attempted to interview 16 students. If fewer than 16 students were available, the team interviewed all students available. The sample was intended to be representative of Grade 2 students. When more than one Grade 2 class was present at a school, the enumerators sampled as evenly from each class as possible. We did not stratify the sample by age or by gender. Data Collection Methods Given COVID-19-related travel restrictions, Causal Design subcontracted and collaborated with Bishkek￾based Erfolg Consult (hereafter “Erfolg”), who conducted a combination of FGDs with parents (including “change agents,” or parents involved in committees and parent-teacher associations) and KIIs with stakeholders across the implementation chain (from government officials at both district and local levels to school directors, teachers, librarians, and cooks). Causal Design conducted remote in-depth interviews (IDIs) with senior national government officials, as well as USDA, Mercy Corps, and World Food Program (WFP) staff. Causal Design also interviewed staff from Save the Children and USAID, both of which are overseeing or previously oversaw other literacy￾related programming in Kyrgyzstan. The KIIs and FGDs Erfolg conducted were primarily in Russian, though clarifications were periodically made in Kyrgyz when needed. Interviews conducted directly by Causal Design were in English and/or in Russian, with translation provided whenever necessary. Table 4. Summary of Interviews and FGDs Conducted Province Batken Chui Issyk￾Kul Jalal-Abad Naryn Osh Talas Grand Total School Cooks 5 22 4 15 5 14 5 70 Teachers/ Librarians 5 22 4 15 5 14 5 70 School Administration 5 22 4 15 5 14 5 70 FGDs (Parents and Change Agents) 1 2 1 2 1 2 1 10 Ayil Okmotu 1 2 1 2 1 2 1 10 25 District Department of Education 1 2 1 2 1 2 1 10 IDIs 10 Mercy Corps, 2 National Government, 3 International Org., 1 USDA 16 Total 18 72 15 51 18 48 18 256 FGD and KII tools were designed by Causal Design and approved by Mercy Corps (see Annex B), although Erfolg independently translated the tools into Russian for its enumerators. The evaluation team designed, organized, and remotely conducted a training of trainers (with lead enumerators) on March 29 and 30, 2021. A primary focus during these sessions was to resolve any translation issues and to ensure that enumerators fully understood all prompts and probes. Special care was taken to ensure that the “spirit” of the tool and its intentions were fully grasped, and to ensure that translations would be appropriate and sensible to enumerators and to respondents. Throughout data collection, Causal Design maintained clear and open communication with Erfolg, to respond to questions, track progress, receive updates on enumerator team safety and COVID-19 statuses, and otherwise assist with any troubleshooting. This was mostly accomplished through a consistent WhatsApp dialogue with Erfolg field coordinators. Causal Design, for purposes of quality control, also requested audio files of FGDs and KIIs which were used for backups and for clarification during periods of confusion. Additionally, Erfolg replicated a basic literacy test which was originally conducted at baseline (by Save the Children). Erfolg visited the same group of schools that were surveyed in the baseline, substituting out three schools because they had been included in a TTR assessment. Specially trained enumerators administered the same test with randomly selected Grade 2 students. Finally, Erfolg enumerators conducted structured facility observations based on a comprehensive tool developed for the mid-term evaluation (see Annex B). Some schools were designated for infrastructure improvements, but all facilities should have working kitchens and food preparation equipment, as well as safe and clean sanitary facilities, working heaters, unbroken windows, etc. The checklists provide a basic accounting of facility maintenance and working order. Monitoring Data Although evaluation questions are answered mostly through qualitative data,7 Causal Design conducted an overview of a select number of indicators utilized as part of Mercy Corps’ MGD monitoring process. Monitoring data consist principally of output data (versus outcome metrics), but whenever appropriate, are woven into the narrative to elucidate findings. More analysis is present in the Literacy Results and Observation Assessment sections of the report . 7 See Annex C for the Terms of Reference, and specifically its ‘Annex C,’ pp. 148-150 of this report, which addresses data collection preferences. 26 Additional Considerations Causal Design worked with Erfolg to track and monitor exposure to COVID-19, and general health and safety. Erfolg adhered to a Causal Design-led protocol, designed to ensure both the health and safety of enumerators, and to ensure that data collection activities did not contribute to outbreaks, or endanger participants. Fortunately, all enumerators completed field work without developing symptoms, and it appears that no outbreaks were associated with field visits. As noted already, IDIs with government personnel, NGOs, and donors, were conducted remotely. 2.4. Data Analysis Methods Data stems from a combination of primary, qualitative sources, observation checklists, and literacy tests. The primary, qualitative data constitutes the bulk from which evaluation questions are addressed and MGD performance is assessed. While the team has conducted an overarching analysis of the observation checklist and literacy tests scores (including descriptive statistics, gender and geographic disaggregation of literacy scores, etc.), the cornerstone of data analysis is the coding of qualitative data. Coding Primary Qualitative Data A Priori Coding Causal Design developed an initial set of codes, or codebook, informed by the evaluation questions themselves and the results framework (and target outcomes). This “first-cycle” of coding consisted of a deductive priori scheme8 intended to probe for the presence, absence, and patterns of key qualities, actions, and responses that characterize MGD support broadly and specifically, across schools and communities. Coding structured the data (text and responses to interview questions) to allow for: 1) a more objective weighting of findings; 2) identification of the consistency or idiosyncrasy of important observations (e.g. Is this a widespread phenomenon? Or is it isolated to one instance, in one location?); and 3) assessment of how the data reflect wider themes or fit into a bucket of relevant responses. Coding also facilitated backwards mapping the source of data, for verification and validation. Refining Codes for Analysis After coding an early tranche of transcripts, the initial set of deductive codes were complemented by additional inductive codes,9 or those based on “new” findings from the data. These may stem, for example, from competing interpretations among KII respondents, or from references to instances and episodes that challenge program assumptions or, conversely, may suggest a unique but important success. This phase, and its flexibility, was important to capturing bits of data which are crucial to program learning that otherwise might have been overlooked. With a codebook finalized, coding proceeded, conducted by two of the authors who were in regular communication to ensure consistency of application, to compare early results, and ultimately to ensure the best inter-rater reliability (IRR), and therefore analysis, possible. 8 Saldaña, J. (2016). The Coding Manual for Qualitative Researchers (3rd ed.). Thousand Oaks, CA: SAGE 9 Ibid. 27 Additional Considerations Monitoring data, literacy test data, and data from school observation checklists were disaggregated to the extent possible, primarily by province, but also by gender. Literacy test data, in particular, do include differences in the means of scores/results and their statistical significance across provinces and languages, when available. See the Literacy Results section for a complete summation of tabulated data. Secondary Consultant Recruited for Report Revision Mercy Corps and USDA engaged an external consultant to revise the USDA MGD17 FFE Final Evaluation Report to incorporate updated monitoring data to capture the no-cost extension period from September 2021- December 2021, with the added task to improve the analysis of the monitoring and evaluation data included throughout the report. Dr. Jones was engaged to conduct this work, and her analysis focused on the findings, conclusions, recommendations, and lessons learned. The executive summary was also part of the revised sections covered by Dr. Jones. 2.5. Evaluation Limitations While overall the evaluation team was provided sufficient data with which to conduct its analysis, as is apparent in thematic sections below, various factors impeded an even more complete, or robust, analysis. These complicating factors are separated by heading and discussed below. Data Quality ● Data quality overall was sufficient, and in places excellent, owing to the efforts of Erfolg field teams. The nature of working through a third party, however, means that the evaluation team was unable to follow up regarding areas of confusion, unclear or vague responses, or to probe further after an interesting observation. In short, the evaluation team’s analysis was limited to the transcripts it was provided. ● Quantitative data, the monitoring/output data in particular, were very challenging to trace. It appears that, due to transitions in implementers, reporting standards, data sources, and data collection methods have changed enough that calculating indicators was much more complicated than was assumed by all parties. As a result, those numbers are reported with some caution. Context ● Exposure to separate, nationwide U.S.-funded literacy programs made it confusing for mostly school￾based respondents, and the evaluation team, to distinguish responses, and attribute them to MGD interventions, specifically. These programs included two USAID-funded programs: Time to Read (TTR), a reading promotion program which works to improve literacy instruction in schools, distributes age￾appropriate reading materials, and engages parents and families in literacy at home; and Okuu Keremet, which works to improve the quality of reading and math teaching in primary schools by training primary school teachers and distributing supplementary learning materials.10 ● To mitigate the possibility of misattribution of results from other programs to MGD, qualitative tools included a brief preamble intended to ensure that respondents were assessing and discussing MGD, specifically. Enumerators were also informed of the existence of other interventions and instructed 10 https://www.usaid.gov/kyrgyz-republic/fact-sheets/time-read-project; https://www.usaid.gov/kyrgyz￾republic/fact-sheets/okuu-keremet-learning-awesome 28 to probe and confirm whether respondents were talking about MGD interventions throughout the interview if in doubt. ● Nonetheless, teachers and change agents made references to “training” and activities that were at times difficult to attribute to MGD versus other programs. ● It was at times difficult for the evaluation team to distinguish, from the responses, explicit program adaptations to COVID-19 from regular programming, in particular around WASH. To try and elucidate information specifically relating to MGD activities, tools asked specifically about known MGD COVID￾related adaptations and interventions, such as installing sinks and online reading camps. Nonetheless, many respondents remained confused about which programs were responsible for particular interventions. Analysis ● While a succinct Theory of Change is referenced above (see section 1.3), the evaluation team was unable to access a more fully developed theory of change or updated results framework. Comparing and contrasting against a more fully developed Theory of Change would have provided an additional analytical lever. Conflicts of Interest There are no conflicts of interest (COI) to report with respect to this evaluation. Causal Design is an independent monitoring, research, and evaluation organization with decades of combined experience providing top-tier evaluation products for clients globally. Causal Design had no previous involvement with MGD in Kyrgyzstan. It did, however, after an initial outreach to multiple firms, sub-contract Erfolg Consult, which Mercy Corps had worked with previously, for its in-country data collection needs. This relationship presented a potential COI risk, but Erfolg proved to be conscientious, effective, and liaised only with Causal Design during data collection. All findings are the result of structured data analysis and have had no influence from Mercy Corps, or others, in their derivation. 3. Findings Findings are presented by OECD-DAC evaluation criteria and corresponding evaluation questions. Quantitative data from literacy tests, observations, and available monitoring data are woven in to complement qualitative data where most appropriate and where those data augment analysis. Data is otherwise tabulated and reported in the Annex A. Therefore, this report will assess program performance across relevance, effectiveness, efficiency, impact and sustainability criteria, evaluation questions, and indicators/targets/ objectives. This report chapter will also highlight criteria specific conclusions, ratings, recommendations, and lessons learned. To guide the performance assessment process, the following rating system was employed to gauge and rate the project performance: Table 5. Performance Categories Rating System for Evaluation Criteria Highly Satisfactory The FY17 Mercy Corps McGovern-Dole Program fully achieves and exceeds criteria standards and components 29 Satisfactory The FY17 Mercy Corps McGovern-Dole Program almost entirely achieves criteria standards and components Moderately Satisfactory The FY17 Mercy Corps McGovern-Dole Program does not fulfill all criteria standards but demonstrates moderate completion of 70% of key components Moderately Unsatisfactory The FY17 Mercy Corps McGovern-Dole Program does not meet at least 50% of the criteria standards and aims Unsatisfactory The FY17 Mercy Corps McGovern-Dole Program does not meet 25% of the criteria standards and components Highly Unsatisfactory The FY17 Mercy Corps McGovern-Dole Program fails to address any of the criteria standards and components Relevance The OECD-DAC defines relevance as, “the extent to which the intervention objectives and design respond to participants, global, country, and partner/institution needs, policies, and priorities, and continue to do so if circumstances change.” OECD-DAC simplifies this by asking: “is the intervention doing the right things?”11 Three evaluation questions guide the treatment of relevance in this report. Broadly, these questions ask if MGD in Kyrgyzstan is designed appropriately, aligns with the objectives of MoES, especially, and consists of activities that are perceived well and embraced locally. The report highlights the appropriateness and alignment of the project with GoK policy, interest in terms of school feeding and literacy, and examines the design, appropriateness, and opportunities for improvement as it relates to eventual impact and sustainability. (1) Was the program’s design an appropriate solution to meet the stated objectives and results? (2) How did stakeholders perceive the program’s design and its activities? The main factor fueling the creation and implementation of the United States Department of Agriculture (USDA) funded McGovern-Dole Food for Education and Child Nutrition Program (MGD) in Kyrgyzstan since 2006 was due to low-performance rates. In 2009 and 2006, Kyrgyzstan was the lowest performer in reading, mathematics, and science literacy, among all the participating countries in the PISA test administered by the OECD and a consortium of the world’s leading international testing organizations. Over 80% of the Kyrgyz students did not achieve the global standard in any subject matter area.12 While there are many potential causes for Kyrgyzstan's poor performance on the PISA test, one that is undeniable is linked to food security and malnutrition as, “There are several reasons behind the learning crisis. First, children come to school unprepared—not enough stimulation of development at an early age, 11 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm#relevance-block 12 Halsey Rogers, March 23, 2018 30 malnutrition of children, lack of basic preparation for school has its effects.”13 According to the Food and Agriculture Organization of the United Nations (quoted above), Kyrgyzstan is a low-income food-deficit country among 51 countries globally, with around 26 percent of the population living in poverty. To address the underperformance of Kyrgyz primary school children, Mercy Corps has been implementing the United States Department of Agriculture funded McGovern-Dole Food for Education and Child Nutrition Program in the Kyrgyz Republic since 2003, and recently won an award for another iteration of the MGD program from 2021 - 2026. The McGovern-Dole 2017 aims to support the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which continues to be a principal participant and stakeholder for the McGovern-Dole 2017, along with selected primary schools. The project aims to support the implementation of the Kyrgyz government's national school feeding program within 430 primary schools and has the following core objectives, which include: ● Increasing knowledge of nutrition, health, and hygiene practices, safe food preparation and storage practices, and access to clean water and sanitation services by facilitating training for cooks, parents, and school officials and providing competitive grants to upgrade school facilities; ● Improving student attentiveness and reduce short-term hunger as a result of increased access to food by rehabilitating school kitchen infrastructure to meet government sanitation standards and providing daily school meals; ● Improving student attendance by providing daily school meals; ● Improving the quality of literacy instruction by ensuring better access to school supplies and materials; and ● Increasing the skills and knowledge of relevant school staff, as well as enhancing community and family support for education. The data suggests that the validity, relevance, and importance of MGD across the actors from the national to school level continues to be demonstrable and evident. Most respondents and stakeholders highlighted that there was no school feeding program before the introduction of the Food for Education programming, and many lauded the impact of the program's literacy component. All components of FFE 2017 are relevant, have added value, and are universally appreciated by government officials and school participants. 13 https://24.kg/english/79321_Crisis_in_education_How_Kyrgyzstan_can_overcome_it/ Spotlight Midline Box 1: At the midline 171 of 172 survey respondents felt that the program was relevant and appropriate to their school's needs. As it relates to the literacy component 97 of 104 respondents felt that the component was useful. Moreover, the respondents lauded the comprehensive nature of the program, the fact that the design covered infrastructure, literacy promotion and School Feeding. The programs relevance and appropriateness to the targeted communities and institutions was resoundingly acknowledged and lauded. The data from the final evaluation suggests continuity, with many discussing the need to expand the program to new schools and including components that will promote sustainability and durability these programs in graduated MGD 12 and 17 schools. 31 School Feeding Program Regarding the school feeding components of the program, respondents offered the following reflections regarding the program relevance and design.  “If there is a possibility to extend the scope of hot meals, one needs to extend it. It should be introduced in other schools as well. For example, modern kitchens were built in the schools where the Mercy Corps held its work. The same conditions should be created in other schools.” (IDI from local government in Batken, Leilek )  “Very high. We don't feed our children properly. Parents usually come home from work, somehow feed a child, offer chips, sweets, cookies, tortillas, etc. I've done that myself. After studying with the Mercy Corps, we already have a different attitude to nutrition of children. We are turning to the proper nutrition of schoolchildren. We learned how to calculate kilocalories for one day according to standards for children. Now children are waiting for a hot dish to be served. And before that, we only gave tea with milk and a bun. Now we do not give tea to children at all. Even the bun we make on the recommendation of a technocard from enriched flour. And we make different types of buns. These buns are made at the expense of those 10 som, which the state gives. We also try to diversify the menu together with the accountant based on the technocard. So, for a month, we develop four types of prospective menus for a month, i.e. for each week. Based on the prospective menu, the cook makes the current menu based on the possibility. So, for a week, we cook soup and porridge with buckwheat or rice, vegetables and fruits. Children are always impatient for breakfast, as parents do not make porridge at home. At first it was difficult, as they did not finish the bun, and some did not eat porridge. Some students refuse to eat breakfast at home in the morning, saying that they will eat at school. So, I think, even if it is a little, we are moving forward.” (IDI DDoE Naryn Ak-Taala)  “Hot meals for children is a big help, because in rural areas many families are poor, in conditions of migration and employment of parents, children do not have the opportunity to eat properly and they go to school hungry.” (Local government official in Osh)  “Hot meals are very important for school. When a child is well-fed, they will be attentive at the lessons. We have noticed the difference between the schools which are included in the hot meals program and the schools that are not included. We cooperate closely with teachers and can see the difference.” (Government official from Osh) The provision of “hot meals” is most consistently reported as the most significant benefit stemming from MGD programming, which is a culmination of the various interventions and support provided by the program in the form of commodity distribution, cooking/hygiene practices, infrastructure, WASH, and equipping kitchens. Hot meals were cited by administrators, teachers, and parents (even most librarians even cited meals first, before books) as fundamental to learning, fundamental to the health and wellbeing of students, and essential for struggling families, as well as school districts. Several respondents from FGDs and KIIs similarly argued that the hot meals are essential to poorer families from the area and “a relief” for parents. Other respondents, primarily teachers and librarians, highlighted that the fortified foods, and enriched flour was a crucial addition to the diet of local children. Several cited their increased awareness of nutritious foods and dietary diversity. But a considerable majority of respondents linked hot meals and nutrition to student attention, well-being, and achievement in their responses. 32 Literacy Promotion Component As it relates to the literacy promotion component, librarians, teachers, and parents cited access to books, positive discipline, and literacy extracurricular activities as critical to MGD success. The MGD reading materials were developed by national experts from MoES, assessed for translation services, and assigned grade levels. Regarding the literacy component respondents shared:  “We realized the importance of using Positive Discipline in the classroom and how important it is for a teacher to maintain a positive atmosphere in the classroom so that student performance indicators are at a good level.” Moderator: “How easy or difficult was it to use Positive Discipline in your teaching?” Respondent: “At first, it was difficult for us, because it was unusual for us, but over time, seeing the fruits of using Positive Discipline, it became easier for us to use it.”  “Yes, I got the support that we needed. Students are more light-hearted, communicative, and take initiative.” (Teacher in Osh, Alay ) For the most part, like the school feeding component, the MGD literacy program was also viewed as relevant and appropriate in design and content. The Positive Discipline training was cited by directors, parents, teachers, and librarians as having an observable, positive effect on teachers and students. MGD continues to be relevant and aligned with achieving the program objectives and aims. Notably, while different respondents may have a “favorite” component that they feel is most important to augmenting the student learning environment, what is key is that MGD—as one implementing partner respondent argued—is “transitioning it from pure humanitarian assistance into a more complex development assistance, which is what is needed in a context like Kyrgyzstan.” This evolution to integrated, complementary services makes each component more relevant in this context. Collectively, the responses from teachers, librarians, cooks, parents, and administrators are consistently and overwhelmingly positive. As a school director from Aravan, Osh, reported, “I consider the Mercy Corps project as one of the most useful projects in 4 years.” This consistency of positive responses from distinct respondent types, spanning geography, makes these results especially robust. Some of the critiques regarding the MGD design were related to concerns about sustainability which highlight the importance of school gardens and orchards, and consideration of the increased workload for participating school staff and officials. A school director from Batken, for example, noted that:  “The teachers didn't benefit at all. It was good for the children. They probably don't eat at home in the morning. And here at school they eat breakfast and study well. And for teachers, on the contrary, the work has increased, they are checked on more often, and they have more responsibility.” (2) To what degree was program implementation aligned with the goals and objectives of the relevant government ministries (MoES, MoH)? The collective response was overwhelmingly positive regarding alignment with the MoES and MoH. The program supported the introduction of hot meal programs at participating schools throughout Kyrgyzstan and contributed to the introduction and validation of the school feeding law in 2019. As a school director from Suzak district (Jalal-Abad) stated: 33  “Of course, this program complements the objectives of the ministries … The Minister of Education and the Minister of Health are working to improve the educational level and health of children. The programs and goals for Mercy Corps and the Ministry of Education are the same. They complement each other. The Mercy Corps program and the Sanitary Epidemiological Station readings are the same, they complement each other.” A school director from Naryn commented: “It definitely does not contradict. On the contrary, everything is going for the better.” Administrators often cited everything from hot meals, access to vitamin-fortified foods, and hygiene practices as most emblematic of the shared alignment with MoES and MOH. District Department of Education (DDoE) respondents were similarly positive, citing many of the same attributes of the program as clear evidence of the alignment while underscoring the importance of the program in meeting their objectives. This was the same for local government officials. MC staff shared that the MoH is a partner to the program as they address child nutrition and are involved in the school menu preparation. While MoH does not have a nutrition policy or guidance for school-age children, they are fully involved and engaged in the program. MoH has historically focused on children under five and has often neglected children over the age of five but are starting to include this age group in their discussions and dialogue.14 The data emerging from the final evaluation reinforce the midterm evaluation findings which resoundingly highlight the relevance and importance of FFE. In short, the program is appropriately designed to meet its core objectives. In addition, its implementation has been adjusted to ensure impact and success and will be examined under the efficiency section. Relevance Criteria Conclusions Conclusion Rating Observations The data emerging from the final evaluation reinforce the midterm evaluation findings, which resoundingly highlight the relevance and importance of FFE17. In short, the program is appropriately designed to meet its core objectives and is positively viewed by stakeholders at all levels. Satisfactory ● The school feeding component of McGovern-Dole 2017 remains highly relevant and in line with the GoK policies and strategic aims. ● The Kyrgyz School Feeding Policy was passed in December 2019. ● The project emphasizes teaching teachers positive discipline techniques and increasing access to reading materials and literacy. SBC activities are highly appreciated elements of the MGD design. ● As stated during the midline, the fact that the program does not teach improved literacy techniques to teachers and that TTR is working in the same schools on literacy, definitive attribution for the learning outcomes is impossible unless there is a comparative quasi￾experimental design employed. ● The program design has been adapted and modified from the outset to ensure the appropriateness and relevance to institutional and individual participants. 14 MC Staff interview, 12/2/2021. 34 Relevance Criteria Recommendations Recommendations Timeline and Prioritization Duty Bearer 1. GoK, MC, and USDA should meet to explore and consider the best approach to the literacy component in the future, particularly exploring whether traditional literacy training and positive discipline will be combined or whether the positive discipline approach will be abandoned. Given the recent literacy study results, there may be evidence that a combined approach is ideal. ASAP GoK, MC and USDA 2. USDA, GoK, WFP and MC should hold a symposium to explore the issue of sustainability of school feeding programs to fully explore what design components are required in order to promote the sustainability and independence of the GoKs SFP programs in graduated schools, especially in light of the current economic issues in the country. ASAP GoK, WFP, MC and USDA 3. USDA, GoK and MC should develop a white paper regarding lessons learned from the flexible and adaptable approach to implementation employed during the Implementation of MGD17. ASAP GoK, MC and USDA Relevance: Lessons Learned 1. The midterm and final evaluations highlight that this has been an adaptable program. Mercy Corps, GoK, and USDA have piloted and modified the program approach throughout implementation to improve relevance, impact, and meet participant needs at the institutional and student level. This flexible and adaptable approach to program design and implementation for success and impact should be lauded and replicated. 2. While the program is appropriate and relevant, sustainability remains a critical issue for all the MGD 2012 and 2017 schools. Effectiveness The OECD-DAC defines effectiveness as, “the extent to which the intervention achieved, or is expected to achieve, its objectives, and its results.”15 Here the report will take stock of FFE17’s achievement as the program has reached the end of its implementation after providing over five years of support to 440 schools and communities throughout Kyrgyzstan. The principal question we are looking at under this criterion is, Did the program meet its indicator targets? The McGovern-Dole 2017 program is a complex multi-sectoral program that has several components and key interventions like in the midline report for the purposes of analyzing the program performance; the analysis is structured into two main clusters. The first is a school feeding/enabling learning environment and the second is literacy and learning promotion. 15 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm#effectiveness-block 35 Within each cluster, there are various activities and outputs, including capacity building, materials, infrastructure, and so forth. The original performance monitoring plan (PMP) indicators are presented in full in Annex A. The program appears to have made significant progress in meeting its output goals, frequently surpassing target numbers and achieving outcomes, despite COVID-19. According to Mercy Corps’ own monitoring data, the MGD intervention demonstrated significant progress on many outputs, achieving or surpassing 21 out of 26 performance indicator targets. (1) Did the program meet indicator targets? School Feeding Program and Enabling Environment Table 6. Monitoring Data - School Feeding Program (SFP) Key Outputs FFE17 LoP Target FFE17 Midterm Status (% of completion) FFE17 Final Status Number of Schools Served 413 (84+329) 84+138+218 = 440 (107%) 440 (107%) Number of Students Reached (SFP) 88,730 95,358 (107.5%) 136,584 Number of School Cooks Certified 329 397 (F) 11(M) (124%) 493 (149.8%) # of SA Trained in Safe Drinking Water Technologies 260 146 (56.2 %) 469 (360.8%) Number of Daily Meals Provided 37,300,000 9,260,121 (24.8%) 27,228,136 (73%) Commodities Provided to Schools (in metric tons) 2,290 1492.821 (65%) 1,810 (79%)16 Number of Individuals Trained in Child Health and Nutrition as a Result of USDA Assistance 121 290 (239.7%) 798 (659.5%) Number of Kitchen Rehabilitation 329 251 (76.3%) 356 (108.2%) Number of Schools Using an Improved Water Source 413 411 (99.5 %) 440 (106.5%) # of Water Treat and Filtration Systems Installed 130 135 (103.8%) 135 (103.8%) Number of Individuals Trained in Safe Drinking Water Technology 130 459 (353.1%) 469 (360.8%) FFE17 sought to provide daily school meals to primary school children in grades one through four in targeted schools to create an enabling learning environment for primary school-age children. However, after the nationwide school closure due to COVID-19 pandemic, in May 2020 the MoES formally requested MC KG to distribute remaining commodities (received in 2019 for the 2019-2020 school year) to the families of the primary grade students as emergency aid. To support this process, the MoES committed to using administrative resources (school staff and space) to help package and distribute commodities, as 16 Initially, the volume of commodities LoP Target for the MGD FFE17 program was 2,290 metric tons, however, the number of distributed commodities is 1,810 metric tones, reasoned by the replacement of flour by lentils at the Y3 of the program. 36 well as provide all distribution records to MC for subsequent verification. MC conducted post distribution monitoring. The quantity of commodities was distributed with the calculation of each commodity standard portion per child per day from the remaining leftovers of provided commodities. As a result, all remaining 336.19 metric tons of USDA-provided commodities were distributed as take-home rations to the families of 74,729 primary grade students in 356 program schools. The distributed take-home rations were not reflected in the program indicators. The program’s approach aims to create a sustainable school feeding program at participating schools as a vehicle for improving learning outcomes for primary school children. To do this, the program begins with rehabilitating school kitchens, which includes infrastructure, both building rehabilitation and water source improvement. To this end, monitoring data summarized above highlights that to date, over 356 school kitchens have been rehabilitated, which also included purchasing of new kitchen equipment to support the execution of the improved hot meal program in participating schools and improved water sources. FFE17 supported improved access to safe drinking water through improved water supply in 440 schools. The program has achieved 108% percent of its target schools; according to the program monitoring data, these schools have received all the requisite school infrastructure and equipment inputs. As it relates to the improved water sources, over 100% of the targets have been reached.17 Kitchen Rehabilitation/Building School-level respondents were very positive about kitchen rehabilitation and the provision of kitchen equipment, which took place in all 356 FFE-supported schools in years 2 and 3 of the program, as well as the provision of dining room furniture, which was distributed to 153 schools in March 2021 based on a rapid needs assessment.18 Respondents noted that these activities had met a critical need, as they made school cooks’ jobs easier and schools would not have been able to buy these items or refurbish kitchens without Mercy Corps’ support. School observation data shows that the vast majority of FFE-supported schools have functional kitchen infrastructure and equipment, as shown in Figure 2. 17 The establishment of the water treatment system at each school was also complemented by an educational component designed to sustain the system’s proper usage and maintenance. 18 Mercy Corps, Biannual Report 7 October 2020 - March 2021 Spotlight Midline Box 2 At midline 123 of 138 surveyed respondents stated that the kitchen repairs and equipment had been completed on time, with the rest saying that they were planned or unsure. Also, only 12 respondents claimed that the work was done late due to government funding, school funding or other reasons. The data suggested that McGovern-Dole 2017 was on track and performing on the SFP component. 37 Figure 2. Kitchen Infrastructure and Equipment Note: Functionality data not collected for handwashing sinks and water supply in kitchens. Some schools noted unmet needs with regards to kitchen rehabilitation and equipment. Respondents noted for example, that they still do not have enough kitchen equipment, citing a need for larger quantities of smaller items such as utensils, pans, stoves, and dishes, which frequently broke. Requests were also made for school dining furniture. Another challenge mentioned by some schools was that school canteens were too small for the number of students. This could be linked to increased school enrolment due to increasing birth rates and the fact that school attendance and enrollment increase with hot meals, which is discussed further under impact as it relates to unexpected consequences. In terms of training the school staff for school feeding, including safe food preparation, management of commodities, and water systems, FFE’s monitoring data highlights that all targets have been surpassed substantially relative to these variables. The program trained school cooks in all targeted schools to ensure they have the qualifications to prepare safe, nutritious meals for primary school children. The qualitative data emerging from the final evaluation highlights that the trainings were systematic and of quality.  “Yes, it was difficult in the beginning and we did not know anything about the trainings. We did not know why we were sent to participate in these trainings. After the trainings, we started to understand and started using all these skills in our work. We learned a lot in these 2 months. We had challenges with water supply and we had to bring it from the street; that’s why we couldn’t use the water heater.” (KII Cook, Osh Alay)  “The trainings have helped us a lot. We now know how to continue with the feeding when the program ends. Mercy Corps supported us and our students in purchasing what we are lacking.” (KII Cook, Jalal Abad Suzak)  “There was no problem with the inventory and products. We didn't have a water filter. And before, during the rains, we had mud mixed into the water. This problem was solved when the filters were installed at the beginning of this school year.” (KII Cook, Naryn) 38 In line with the midline, the quotes above from the final evaluation highlight the benefit of the training for the school cooks, emphasizing the knowledge that the individual gained to do their work as a school nutrition staff. The last quote highlights the importance of the improved water sources provided by the program. Though some school cooks said they had found the trainings challenging at the beginning, almost all of the cooks interviewed were extremely positive about the trainingsthey had received under the MGD program. Respondents reported that they had learned a lot about cooking techniques, food safety and storage, nutrition, and reporting requirements and adapted their cooking practices accordingly. School cooks also viewed the in-person visits by field-based school feeding specialists very positively, noting these visits provided a good opportunity to learn, and that the specialists were helpful, friendly, and knowledgeable. Quantitative data also indicates that training for cooks was useful and put into practice by most trainees; 409 of the 493 of the people trained in safe food preparation and storage under MGD also demonstrated use of these new skills. Table 7 below summarizes how many school meals were provided and the numbers of children who received daily school meals. As noted previously, while MGD was able to surpass their target number of school-age children receiving daily school meals, they were not able to meet the targeted number of daily school meals, due to COVID-19 mandated school closures. Mercy Corps adapted to this by distributing food commodities as take-home rations, discussed further in the COVID-19 section. Table 7. Daily school meals and school-age children receiving meals as a result of USDA assistance Description Target Result Number of daily school meals (breakfast, snack, lunch) provided as a result of USDA assistance 37,300,000 27,228,136 (73%) Number of school-age (grades 1-4) children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance 88,730 136,584 When disaggregated by year and by province (noting that there are differing numbers of schools within each province) there is an apparent trend showing that most of the meals provided were facilitated in 2019, which suggests that the pandemic had an outsized effect on the program’s ability to meet this indicator. Figure 3. Total Number of Meals per Year by Province 39 Source: M&E Data Collected on Monthly Meals The food commodities provided were viewed as a vital contribution of the MGD project to schools during the course of the program. Parents highly appreciated school meal provision, and it was cited as a relief for many families, as some households faced challenges feeding children before going to school due to shortages of time and household resources. School meals were especially relevant and important for children from low-income families. One female teacher from Jalal Abad, explained:  “Not every family can fully provide for children. Now, not only in the region, there are no jobs all over Kyrgyzstan. Especially due to COVID-19, the last two years have been difficult for people to survive. If you talk to young parents, you find out that it is very difficult to feed the family. Bread and butter every day… Therefore, children at least get fortified food here. This is useful for both the children and their parents. There are parents who go to work in the morning and do not have time to feed their children... They eat here and go to their classes in a good mood.” Furthermore, extra or leftover food was given to children from low-income families, including those in higher grades that did not receive school meals, adding an additional benefit for those most in need. All respondents interviewed for this evaluation viewed the provision of school meals to be extremely effective in causing improvements to children’s attendance, attentiveness, and engagement in lessons. Students were described as being more “cheerful,” “joyful,” “active,” “interested,” and having better “perception.” As such, respondents linked school meals to literacy and other educational outcomes, including increased knowledge and interest in reading. Typical sentiments from interviewed respondents, in this case a male school director from Jumgal, Naryn, included the following:  “The primary outcomes are current perceptions in education. We as heads of schools are monitoring this issue. We are conducting conversations with primary school teachers, and they note that after eating hot meals, the children have an interest in learning, and their mood is better. When a child is hungry, he thinks about food and about home. If the stomach is gurgling, he will not be able to answer the question even if he knows the answer. But, after eating, the children’s mood rises.” 40 Overall, the program took a comprehensive approach to SFP that addressed not only commodities and training but also sought successfully to create the environment required to provide students with nutritious, hygienic, and healthy hot meals five days a week. Social Behavior Change To support the effectiveness and reinforce the sustainability of the SFP program activities, the program includes an aggressive social and behavior change (SBC) component. The SBC program was not originally part of the program; however, it is one of the adaptations made to reinforce and support the program's success by building community and school support for the MGD17 aims related to adapting new nutrition, hygiene and literacy practices. This aspect of the program has been so successful that MC received and responded to a request from the WFP to share their SBC policy and component design. They asked for this to be shared so that they can included in the future programming19. Below we highlight the main outputs as of the McGovern-Dole 2017 SBC activities: Table 8. Monitoring Data - Social and Behavior Change (SBC) Key Outputs MGD17 LoP Target MGD17 Midterm Status MGD17 Final Status SBC Training Themes Covered Social mobilization, breastfeeding, complementary feeding, dietary diversity and anemia prevention, handwashing, and junk food Social mobilization, breastfeeding, complementary feeding, dietary diversity, and anemia prevention, handwashing, and junk food Social mobilization, dietary diversity and anemia prevention, handwashing and sanitation, safe food storage and reading at home Number of Urban SBC events n/a20 12 marathons to support healthy diets and active lifestyle in small towns 12 marathons to support healthy diets and active lifestyle in small towns Number of Informational Ads Distributed Through Local Media (TV, social media, etc.) 36 55 (152.8%) 351 (975%) Number of Parents Trained in Child Health and Nutrition 2,105 872 (41.4%) 3453 (164%) (1876 F) 19 Interview with MC staff 12/1/2021. 20 There was no established target for this activity in the Agreement since the activity was designed and proposed at a later stage. 41  “SBC was part of our strategy to encourage very close involvement and partnership with local government and local communities. We spent a lot of time to make sure these actors understood program aims, design and their role in the program. The SBC was a critical aspect of reinforcing and promoting the program's success. When parents and local communities are supportive, the school staff can create a very enabling environment for the program's success. Community members, schools and local authorities participate in the implementation and the M&E to measure and determine program success and how to address problems.” (MC staff) The MGD SBC component started in 2018 and ran in three cycles of outreach activities, with trainings conducted both online and in-person depending on pandemic requirements. The topics covered included: nutrition, namely complementary feeding for infants, exclusive breastfeeding, dietary diversity and anemia prevention, and safe food storage practices; WASH, namely good handwashing practice, improved sanitation facilities and safe drinking water; support for literacy promotion and school feeding activities; as well as trainings on social mobilization and the SBC process.21 Further analysis of the gendered component of SBC activities, whereby Mercy Corps sought to involve more men and increase their interest in SBC topics, is discussed in the Efficiency section. Overall, respondents were optimistic about the effects of SBC activities. Change agents, and school staff who had experienced SBC trainings, noted that the topics were useful, frequently mentioning nutrition, sanitation facilities and handwashing. Although behavior change in targeted communities cannot be exclusively attributed to program SBC activities, change agents reported having changed their behaviors, and recalled instances where they had seen other community members changing their behaviors. Most changes were related to hygiene and nutrition and other topics such as breastfeeding and nutrition during pregnancy. Some illustrative testimonies include:  “When we told our friends and neighbors about personal hygiene (toilet doors should be tightly closed, that there should be a washbasin with liquid soap near the toilet), about proper nutrition (that it is necessary to diversify the menu, preferably during the day there are different dishes, and not just one), a month later, we already noticed that they listened to advice.” (FGD Respondents, Issyk-Kul)  “I noticed changes in my neighbors. They built a modern toilet and kitchen. All the people started to install hot and cold-water supply systems. Many bought big refrigerators. Some parents now make their daughter-in-laws breastfeed until 6 months and do not use artificial feeding. They see the results in their children as they are healthy.” (Female FGD Respondent, Suzak, Jalal-Abad) The SBC strategy was also extended to the literacy promotion component of the program, below a respondent stated:  “Yes, we have a WhatsApp group where we communicate with parents, students and conduct various reading activities. Regarding e-books, we just started to use them.” (Librarian, Talas-Talas ) It is worth noting that as a result of the pandemic that Mercy Corps adjusted its SBC strategy to ensure that program messages and community-level work could continue to reach participants and community members to reinforce and sustain the work that had been done since the outset of the program related 21 Mercy Corps, Biannual Report 7 October 2020 - March 2021; Mercy Corps, Biannual Report 6 April 2020 – September 2020; Mercy Corps, Biannual Report 5 October 2019 – April 2020; Mercy Corps, Biannual Report 4 April – September 2019 42 to literacy, nutrition and health for schools’ youth and their families. This was highlighted in the MGD Household-level study excerpt below.  Table 8 shows that in the baseline survey, 26% of caregivers participated in an SBC training on nutrition, 23% on hygiene and sanitation, and 24% on reading. By the endline, the same shares grew by 4, 13, and 3 percentage points, respectively, confirming the expansion of the SBC program coverage. In addition to the training, the SBC program promoted good practices through messages on TV and virtual groups. We observe a decrease in the proportion of caregivers that saw such information (from 68% to 53%). In contrast, we notice an increase in the level of caregivers' engagement with the virtual world. Certainly, during the pandemic, people got more engaged with virtual ways to communicate with their communities, and we can observe that in our data. Belonging to a virtual group like WhatsApp or Telegram grew by six percentage points; now, 76% of households make part of such groups. Moreover, the share of caregivers on a topical virtual group, where they discussed health, nutrition, food safety, hygiene, or sanitation, grew by 14 percentage points, from 17% to 30%.22 This excerpt illustrates how Mercy Corps’ SBC strategy adapted successfully to reach communities and participants despite the social distancing and end of face-to-face activities caused by the ongoing pandemic. This is a recurring theme that we see in how the program adjusted program design and implementation to respond to emerging needs or be more successful and impactful. Further evidence of the impact and visibility of MGD17’s SBC work is illustrated by WFP’s interest in learning about the approach. As a result last year, MC shared its experience of working with local communities on social and behavior change, video tutorials in Russian and Kyrgyz to deliver online training to school cooks, specifications of food products for improving the state procurement process, the list of graduation package for handover to graduating schools with WFP. MC shared videos and information materials on nutrition, reading, sanitation, and hygiene with the USAID Advancing Nutrition Project. MC contributed to the finalization of the national communication strategy, on which the community outreach activities are currently based.23,24 Infrastructure/Winterization A critical aspect of MGD17 strategy to create an enabling learning environment for primary school students is linked to the winterization and WASH components of the project. MGD17 sought to provide school infrastructure rehabilitation grants for winterization and to upgrade sanitation facilities in select 22International Security and Development Center. McGovern-Dole Food for Education and Child Nutrition in Kyrgyzstan – Endline Study Final report ( November 2021) 23 MC Staff interview, 11/24/2021. 24 While the MGD17 SBC approach relies on community members sharing messages to ensure local ownership and accessibility, the mission must note that some respondents, felt that it would be more effective for information to be shared directly by Mercy Corps trainers or field specialists, who were perceived to have more authority and to be more knowledgeable than local trainers. This is a question of feasibility, for the program to have reach they seek to change behaviors at the community level they must rely on community members and leaders to support the magnitude of outreach and messaging to have the impact the SBC strategy seeks to have. 43 target schools. At the midline none of the infrastructure work had begun; the current status, as the table 9 below illustrates, MGD17 managed to surpass its targets. Table 9. Monitoring Data - Winterization and WASH Key Outputs MGD17 LoP Target MGD 17 Midterm Status MGD17 Final Status Number of schools that received Infrastructure Repair / Winterization grants 50 0 73 (146%) Number of Latrines Built or Restored 50 0 59 (103.5%) Number of schools that received grants to install handwashing facilities to mitigate effects of COVID-19 n/a n/a 30 Mercy Corps conducted a needs assessment for the MGD17 schools, focusing on water sources, sanitation conditions, winterization, and handwashing facilities, to guide the selection of the infrastructure projects for winterization and WASH. In addition, it adapted its infrastructure approach to ensure that the work in WASH particularly would comply with worldwide standards related with construction. Originally Mercy Corps planned to give subcontracts to schools for the latrines; however, in 2019, Mercy Corps received a modification from USDA that allowed them to engage firms directly to build the latrines based on international standards latrine construction. This ensured that the latrines were safe and met international standards, including the use by handicapped students and adults. Before, there was some inconsistency in the quality of the latrines because schools were contracting the construction themselves using local firms with no knowledge of international standards. Four different companies were engaged to work for the MGD17 program; they included Arhstroiinvest, Orion, YugMag, and ArslanStroi. Firms were engaged based on the region of the country.25 School-level respondents widely praised infrastructure improvements. School-level and government representatives felt that without Mercy Corps, these repairs would not have been possible. The program also exceeded its target numbers of schools for infrastructure interventions. When evaluating the types of projects, the most prevalent were window replacements, followed by indoor toilet repairs and handwashing stations. Winterization Mercy Corps selected 52 projects across 35 schools to target for winterization infrastructure projects, including replacing or repairing windows, doors, floors, roofs, ventilation, and heating systems based on a needs assessment.26 Overall, respondents were appreciative of the infrastructure developments, noting that kitchens were warmer, and noted no problems. According to information from the school observations carried out as part of this assessment, infrastructure elements were largely implemented effectively, and in many cases, without problems. 25 Mercy Corps, Biannual Report #5 and MC Staff Interview 11/24/2021. 26 Mercy Corps, Biannual Report 7 October 2020 - March 2021 44 Figure 4. Installation and Functionality of MGD Infrastructure Components (collected during the final evaluation for a sample of schools) *Data not collected on installation of heating systems, roofs, and kitchen ventilation systems As shown in Figure 4, most or all schools targeted for window and door installation reported that these were installed correctly and all were in working order at the time of the visit. All schools targeted for roof repair and kitchen ventilation system installation reported having no problems. One of the four schools that received heating system installation/repair did not have a functioning heating system at the time of the visit, it was not clear why. Latrines & Water Stations For 28 schools, new gender-specific and persons living with disabilities-accessible latrines were constructed. For an additional ten schools, toilets were repaired. MGD17 covered 65-96 percent of the costs for all of these works, and local communities contributed the remainder in cash or in￾kind.27Feedback from school-level respondents on new or refurbished latrines and sinks were positive. Respondents reported that the students used the new sinks for handwashing, and school staff and students alike appreciated being able to use warm indoor latrines.  “Students have become more well-mannered, their interest in learning has improved. The children began to fully comply with the hygiene standards, as we had the opportunity to do this at school, for example. We have hot water, and a toilet was built.” (School librarian) Handwashing facilities were lacking in some cases. Though all of the schools sampled had handwashing facilities in or near the dining hall, half of them did not have handwashing facilities with soap and water within 2-3 meters of latrines. Respondents also appreciated the provision of napkins and soap for handwashing, especially in the context of COVID-19. In response to the pandemic, Mercy Corps solicited additional funding to provide WASH upgrades at 30 program schools in Kyrgyzstan. Literacy and Learning Promotion Table 10. Monitoring Data - Literacy and Learning Promotion 27 Mercy Corps, Biannual Report 7 October 2020 - March 2021; Mercy Corps, Biannual Report 6 April 2020 – September 2020 45 Key Outputs McGovern-Dole 2017 LoP Target McGovern-Dole 2017 Status Sept 2019 (% of Completion) MGD17 Final Status # of Supplementary Reading Materials Received 34,699 17997 (51.9%) 100823 ( 291 %) # of Librarians Trained 310 207 (46.5%) 895 (289 %) # of Teachers Trained in Positive Discipline 1290 2045 (158.5%) 2045 (159 %) # of Reading Corners 2580 1175 (45.5%) 3410 (132 %) # of Reading Clubs/home-made book competitions 430 118 (27.4%) 561 (131 %) # of Reading Festivals 42 14 (33%) 132 (314 %) # of Parents Trained in Literacy Promotion 6,450 3000 (46.5%) 4574 (152 %) Literacy Promotion The McGovern-Dole 2017’s literacy and learning promotion component sought to increase the amount of time children in the targeted schools spend reading and writing. The principal components of literacy and learning are; (1) Training in Positive Discipline for selected staff at participating schools; and (2) Literacy Promotion activities, which include supporting Parent-Teacher Associations (PTAs) to organize and implement family and community literacy activities.28 Lastly, regarding literacy promotion, all 356 targeted schools have established Reading Corners, some respondents noted that they did not have enough storage areas for books when they were distributed.29 One female librarian from Aksy, Jalal-Abad, recounted, “I always tell superiors that we don’t have proper conditions. Despite the fact that books are provided by the state and different organizations like yours, we still don’t have conditions where to place them... I have a request, before setting a task for us, please pay attention to our conditions.” Observation data also show that in around 10 percent of the sampled schools, Reading Corners are not fully accessible to primary grade students, either because the books are too high for small children to reach or because the classroom/library where the reading corner was closed at the time of the observation visit. Data from the literacy assessment concurs with the positive findings emerging from the MGD17 monitoring data, and the assessment observed an increase in reading comprehension. Specifically, the proportion of second-grade students tested who attained the specified threshold increased from 39 percent in the baseline analysis to 47 percent in the endline assessment that was conducted for this final evaluation.30 Although this increase may not be due solely to MGD programming, it nonetheless suggests 28 As discussed during the midterm evaluation, this component of the project was changed to avoid duplication with the USAID funded TTR program and shifted to positive discipline training for educators and literacy promotion community-level activities to promote literacy outside of the classroom. Also the literacy promotion activities included Storytime, reading clubs, parents training, reading festivals, and public awareness campaigns and supplementary reading materials appropriate to primary school reading levels 29 Mercy Corps, Biannual Report 7 October 2020 - March 2021 30 See Annex 2 for more detailed results from the literacy assessment conducted as part of this final evaluation. 46 a positive influence of MGD17 on literacy outcomes and is particularly notable given the challenges posed by school closures and the subsequent switch to online learning due to COVID-19. 31 Positive Discipline Positive discipline is focused on praising children and redirecting behavior so that they achieve improved learning performance. While the monitoring data here looks quite strong, the evaluation data suggest that the literacy and learning component has mixed results. It was difficult to ascertain and quantify the effects of MGD literacy promotion, which focused on extra-curricular activities in isolation given the other literacy interventions also underway at many MGD-targeted schools in Kyrgyzstan.32 As such, some respondents expressed confusion about which activities fell under which program. Many teachers interviewed said they found it initially difficult to grasp the concept of Positive Discipline, as it differs from the more formal teaching methods typically used in Kyrgyzstan. Despite this, many teachers reported using principles of Positive Discipline, such as treating boys and girls equally, speaking gently to children, and building individual relationships with students. They also reported seeing benefits, such as that lessons became more positive and “light-hearted”, students gained independence and confidence to speak their opinions, and teachers developed closer relationships with their students. Reflections from many teachers included:  “Under the Program we took part in trainings on Positive Discipline, where we learned that if a student made a mistake we as teachers shall gently and delicately explain that they made a mistake. Thanks to Positive Discipline, children began to study well, and we moved from old teaching methods and became closer to them.” (Male Teacher, Tyup, Issyk-Kul, 2018 Cohort)  “Before the project, we treated the children of the primary classes as little ones who do not understand much. After the project, we began to listen to them more, and despite their small age, they have very interesting thoughts. We thought that only excellent students are good ones, and the rest are not very good. Now we try not to divide them. If we listen patiently to a student, then he can do a lot of good things.” (Female Teacher, Uzgen, Osh, 2019 Cohort)  “I liked the element of positive discipline, which helps children very much, and describes how to work not only with high-performing children but also with low-performing ones. If a proper communication technique is applied to such children, they are happy and do their best to study well.” (Teacher 4/12/21)  “Of course, teachers have a lot of changes. The primary school teachers, in particular, have changed. They have already learned how to communicate with children and communicate with parents.” (Librarian, Osh Kara-Suu , 13 April 21) Monitoring data from Mercy Corps lesson observations indicate that some gaps in applying Positive Discipline remain. Although it was possible to distinguish between MGD support and other literacy interventions, respondents were largely positive about MGD literacy activities, including reading camps, book-making competitions, and family reading campaigns. Most children and their parents perceived these activities to be well-liked and attended. Respondents shared: 31 Under MGD, Mercy Corps distributed 100,823 textbooks and other teaching and learning materials, almost triple the original target of 34,699. 32 Mercy Corps. Biannual Report #5. 47  “I think the school camp helped the children to spend their holidays usefully, gained new knowledge, experience and, most importantly, interest in reading.” (Librarian)  “The most useful was the Reading Festival and the Reading Camp, but listening was the best. Because if a student first listens and can immediately retell it, it has a very positive effect on his mental development. But best of all, we had the work of parents with their children. Because until the 4th grade, many parents do not help their children, do not control them. And after the 5th grade, the children are already learning independently. Many parents do not know what the child's behavior at school is. When we talk about the actions of a child, they say that the child behaves differently at home, and at school it is completely different. This project has had a positive impact not only on the school, but also on the parents and children themselves.” (Librarian, Osh Kara-Suu , 13 April 21)  “It was interesting when we held educational festivals. There we attracted parents, sponsors, who also read books to children. With the help of the educational festival, we managed to collect about 50 books.” (Teacher)  “We have a Reading Festival. Parents take part with great pleasure and there are regular joint readings… When you explain how to improve and how to sell children on reading—it’s so simple and easy, and the parents are just thrilled. They did not used to think about it, and everyone was busy with their own business: a child does his own affairs and a parent does their own affairs. But when we advised them on how to attract children with reading, they started to implement it. And they have a result: all children started reading. There were children who did not read at all or read poorly. Now all of them read, and can retell and reread. And it’s easy because the parents are involved in this work. I think that it’s important that parents are involved.” (Teacher) The quotes above highlight the positive impact of the literacy promotion activities increasing interest in reading amongst students and families, as well as increasing access to books through donations. Mercy Corps Biannual Report (October 2020 to March 2021) states that 10,447 books were donated to 331 schools holding reading festivals in the reporting period alone.33 MGD activities also supported literacy promotion at home. Many respondents reported seeing increased parental engagement with their children’s literacy as a result of MGD activities. MGD activities may have led more fathers to become more involved in school activities and literacy promotion at home, according to evidence from school-level respondents and Mercy Corps staff interviewed for this evaluation. One librarian from Nookat, Osh, recalled, “There were a lot of changes. For example, previously, fathers did not read books at home, only mothers read, and now fathers are interested in reading books. Now even grandfathers read books and send us photos.” However, parents’ and especially fathers’ involvement 33 Mercy Corps, Biannual Report 7 October 2020 - March 2021 Spotlight Midline Box 3 At the midline the evaluation data suggested that the literacy and learning component has mixed results. For example, while 31 of 33 respondents claimed their school had received positive discipline training, only 19 stated that parents had been trained, and only 25 claimed librarians had been trained. Lastly, only 17 of the 33 claimed to have reading corners. This mixed performance was also reflected in the midline responses where several schools that had received the program did not remember having had the training. In other cases, respondents confused some activities with TTR. 48 in literacy promotion at home cannot be exclusively attributed to MGD and may have resulted from COVID-19 induced school closures. Naturally, the switch to online learning activities at home has required parents to become more involved in their children’s education. As explained by a female librarian from Tyup, Issyk-Kul: “Since education has become online, parents have begun to work more with children and literacy is increasing not only for the child, but also for the parents. Parents provide such information. They say that by teaching children, they themselves also learn a lot. Family changes were mandatory." The remaining indicators are discussed, where relevant, and are presented in full in Annex A. Effectiveness Criteria Conclusions Effectiveness Conclusion Rating Observations The MGD17 program is a complex multi-sectoral program the data shows highly effective performance across all of the outputs related to both the SFP/enabling learning environment component and the second Literacy and learning promotion components. Despite the school closures due to the pandemic, MGD17 was able to meet or surpass all of its pre-established objectives except those linked to meals served to students. Overall,the program took a comprehensive approach to SFP that addressed not only commodities and training but also sought successfully to create the environment required to provide students with nutritious, hygienic and healthy hot meals five days a week. Highly Satisfactory ● 108% percent of target schools have received all the requisite school infrastructure and equipment inputs for kitchen rehabilitation. ● As it relates to the improved water sources, over 100% of the targets have been reached. ● Most (75% or above kitchen equipment and materials were functioning at time of field visit. ● In terms of training the school staff for SFP, including safe food preparation, management of commodities, and water systems, the McGovern￾Dole Monitoring data highlights that all targets have been surpassed substantially relative to these variables. ● All winterization and WASH infrastructure targets were exceeded. ● Mercy Corps received a modification from USDA that allowed them to engage firms directly to build the latrines based on international standards latrine construction. ● MGD17 also made distributions of household rations to the families of participating primary school children. ● MGD17 also surpassed all targets linked to literacy and learning promotion, including extra-curricular activities, positive discipline training for teachers and the creation of reading corners. Effectiveness Recommendation Recommendation Timeline and Prioritization Duty Bearer 1. Current MGD17 data highlights that the indicator for meals served was not achieved; however, it is not clear if the household level rations for the families of participating primary school children were considered in this calculation. If not, some consideration for these distributions and the additional activities undertaken should be reflected in the MC PMP and performance assessments to demonstrate the quality of the effectiveness ASAP MC & USDA 49 and management demonstrated by the MGD17 management and oversight efforts. 2. If not already done, the program should share with the GoK the infrastructure and needs assessment framework to integrate this into their planning and prioritization standard operating practices should they so choose. End of 2022 MC, USDA & GoK 3. The MGD17 and USDA should develop guidance on delivering effective online training, SBC activities for child and adult learners participating in development and/or quasi-humanitarian programming. End of 2020 MC, USDA and GoK Effectiveness: Lessons Learned 1. The MGD17 SBC approach was part of an effective strategy to encourage very close involvement and partnership with local government and local communities. It is a best practice in all development programs that seek to extend behavior change to the household or community levels. 2. The MGD17 donors and management team continually and consistently adjusted program design and implementation to respond to emerging needs or to be more successful and impactful; it is a best practice that should be applied more widely to ensure that programs remain relevant and ensure impact as circumstances/ conditions change or new data becomes available. This includes gender studies, household-level studies and other unplanned activities supporting program success. 50 Impact The OECD-DAC defines impact as, “the extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects,” or “what difference does the intervention make?”34 The ultimate aim of the MGD17 program is to create an enabling learning environment and improve primary school children’s health and nutrition outcomes as a vehicle for improving their learning outcomes. While the data on school feeding and literacy promotion activities are important and it should not be undervalued that according to MC monitoring data, during the 2020-2021 academic year 87% of MGD17 schools have a diverse school menu (i.e. no meal repetition beyond twice per month and all food groups are included), this was not the fundamental objective of the program. The main aim of the program and impetus for the creation of the School Feeding Program (SFP) in Kyrgyzstan was to improve the poor learning outcomes of primary school children in the country. The reading assessment conducted as part of the MGD17 suggests that the program, along with all the other programs and donors working on improving learning outcomes in Kyrgyzstan, has positively impacted reading and comprehension among target primary school students’ children population. According to the literacy assessment, the most significant change is an increase in reading status. Figure 5 below suggests substantive improvement in reading comprehension among participating primary school children. The observed increase in reading comprehension is a notable achievement, particularly in the context of school closures and the move to online learning due to COVID-19, which respondents across the board reported was a challenge and was likely to result in lower educational attainment for students. Such changes are not solely attributable to MGD17, given that other literacy programs such as USAID’s TTR and Okuu Keremet are also operating in many MGD schools. However, the literature highlights the clear, conclusive link between nutrition and improved cognitive ability and learning outcomes.35 As such, the contribution of MGD17 to improved learning outcomes in participating schools is unquestionable based on the existing literature and research. Figure 5 highlights the findings from the baseline and endline reading assessments. 34 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm#impact-block 35 Benton, D. & Roberts, G. (1988). Effect of vitamin and mineral supplementation on intelligence in a sample of schoolchildren. The Lancet, 1, 140–143. Schoenthaler, S., Amos, S., Doraz, W., Kelly, M., & Wakefield, J. (1991). Controlled trial of vitamin – mineral supplementation on intelligence and brain function. Personality and Individual Differences, 12, 343–350. Benton, D. & Buts, J. (1990). Vitamins/mineral supplementation and intelligence. The Lancet, 335, 1158–1160. 51 Figure 5. Reading Status in the Language of Instruction (Baseline n=768, Endline n=718) The data highlights an 8% increase in the number of readers with comprehension, while the nonreaders remain at 5%. While this is a single-digit improvement in the reading comprehension results from the baseline, in the backdrop of the ongoing pandemic and school closures, these results are promising, and they suggest that the MGD17 is contributing to reversing malnutrition that had previously undermined the performance of school-age children in Kyrgyzstan. They also suggest that continued programming and innovation is required to ensure that the learning outcomes among targets continue to trend in the right direction and reverse nutrition deficits that undermined learning outcomes previous to the introduction of school feeding hot meal programs in Kyrgyzstan. (1) To what degree have there been changes in each of the following that are attributable to the program? 2) To the degree that attribution is possible, what combination of activities is responsible for the above impacts? (a) Health, Nutrition, and Dietary Practices at the Schools According to the available data, MGD17 has strong findings related to changes in health, nutrition, and dietary practices at schools. Monitoring data shows that 87% of program schools have a diverse school menu (i.e. the same meal is not repeated more than twice per month, and all food groups are included). The monitoring data also shows that 78% of program schools have school feeding (full meals) five days per week. Both of these findings are as of the 2020-2021 academic year. These findings are confirmed by the USDA-funded Household Study conducted by the International Security and Development Center, which states that “the data reveals that virtually all children in our sample were receiving school meals during the academic year they were interviewed.” At baseline, 95% of the children were receiving a hot meal in school, and in the endline, the same share increased by four percentage points, reaching full 52 coverage.36” All schools where staff were interviewed reported that they had started serving hot meals to students. Interview respondents shared the following:  “We used to serve children with baked rolls and tea for 7 KGS allocated by the state. We also gave them cookies. But since our cooperation with Mercy Corps, we have started serving hot meals 5 days per week and we will keep going with this program till the present day.”  “We have been cooperating with Mercy Corps Food For Education program since 2018. There were big changes after joining the hot meal program. We have meals, but parents do not pay attention to enriched meals. We have been supplied by Mercy Corps with food commodities which ensure growth, improve thinking and developing of a student’s body. Speaking about changes, our student has got the first place on mental arithmetic in Suzak district. And we thank Mercy Corps for this.” (KII Director Jalal-Abad Suzak)  “Mercy Corps is a good community. Besides hot meals, they have provided special equipment and equipped our kitchen. Mercy Corps has trained us comprehensively on how to conduct paperwork, correspondence. All staff who visit us are professionals. They gave us logbooks, and trained on how to fill in them. They taught not only students but us as well. Conditions in the kitchen were improved to feed students.” (KII Director Jalal-Abad Suzak) It is difficult to disentangle the factors that have led to the major changes in health, nutrition and dietary practices at the schools; however, what is clear to most program participants is that much of the changes that have occurred within these programs would not be feasible without the MGD17 program support. The quotes highlight everything from the commodities, training, kitchen rehabilitation, monitoring visits, and coaching were critical to program success. Moreover, all aspects of this comprehensive program have been refined and adjusted over the past five years of implementation to address weaknesses and ensure success. Key to these changes were the kitchen rehabilitations that enabled school cooks to cook on-site, which they did not have before. The provision of in-kind foods was also seen as a key factor. All schools reported using the commodities distributed by Mercy Corps, though as previously noted, many raised the issue that some commodities were new to both children and cooks. Trainings for cooks on food storage, nutritional practices, and cooking techniques were also reported to be significant. As noted previously, 409 of the 493 of the people trained in safe food preparation and storage put these new skills into practice. Cooks interviewed reported following lessons from the trainings, Cook’s Guide, and recipes from Mercy Corps to ensure that food provided at school was healthy and nutritious. They reported providing varied meals throughout the week and serving healthy dishes such as pea soup, porridge, compote, lentils, and buckwheat. Some which they had never cooked before. Cooks reported changing their cooking methods to be healthier, such as boiling rather than roasting or frying foods, and measuring foods by the gram. The MGD17 robust and comprehensive package of infrastructure, commodities, training, and SBC work together to support participating schools to deliver successful and consistent hot meal programs. 36 International Security and Development Center. McGovern-Dole Food for Education and Child Nutrition in Kyrgyzstan – Endline Study Final report (November 2021) 53 (b) Government Financial Support for Literacy and School Feeding Activities, including Local Government Government support for school nutrition has increased due to advocacy by Mercy Corps and other stakeholders under the MGD17 program. Mercy Corps and WFP supported the government to refine and pass an updated version of the Kyrgyzstan national school feeding law in January 2020. In particular, Mercy Corps advocated for a more efficient allocation of the dedicated government budget under the updated school feeding law. Under this updated law, all schools receive 7 KGS per child per day for school feeding, increasing up to 10 KGS per child per day in high-altitude and remote areas. More significantly, Mercy Corps and WFP were instrumental in passing a bylaw in June 2020, which outlines a list of 52 locally available food commodities which may be purchased for schools using government funds. This bylaw aims to ensure that schools meet minimal nutritional requirements for young children in terms of nutritional quality and menu diversity.37 As such, the passage of this bylaw has positive implications for the sustainability of efforts to provide high nutritional quality school meals across the country. However, a number of school-level respondents noted that the amount allocated under the School Feeding Law is still not enough to fully support school meals, especially in the context of increasing food prices as a result of COVID-19. Some schools from the 2017 and 2018 cohorts reported changing their menus, such as removing expensive commodities like meat, and even reducing the number of days they could provide hot meals. Although schools tried to fill gaps through parent contributions, these contributions were not always enough, and not all parents could contribute. As some respondents recalled: ● “The government only gives us 7 KGS. Once a week we used to cook a dish with meat, but this year we were without it. There were green peas from Mercy Corps. Green peas taste better and cook faster. Children eat it better. The old yellow peas take a long time to cook, and some remain uncooked, hard depending on the suppliers… And it was noticeable to the students, since we used to give hot meals for 5 days, but now we give hot meals on only 3 days… And on the other two days we give tea and bread. Before we used to give them hot meals 5 days a week thanks to Mercy Corps products. Of course, it is very noticeable. You can't buy anything with 7 KGS.” (Female Cook, Kemin, Chui, 2017 Cohort) ● “During the pandemic, they told us not to collect money from the parents of the students. But at that moment it was difficult for us because 7-10 soms, which the state gave us were not enough for school meals… There were such misunderstandings among the parents... At that moment there were two opinions; most of the parents wanted to pay for hot meals and others did not want to pay.” (Female DDoE Representative, Ak-Suu, Issyk-Kul) These results should be interpreted in the context of a spike in food prices due to COVID-19, and therefore cannot be assumed to be representative of the long-term sustainability of school meals. 37 Mercy Corps: April 2020-September 2020 Biannual Report 54 There is much less evidence of increased financial support for literacy activities by local or national governments, with the exception of the adoption of a Positive Discipline training module by the Kyrgyz Academy of Education. Some school-level respondents noted that local Ayil Okmotu members and other local officials participated in Reading Festivals and other distinct programs, including reading books to children, and donating books to schools. However, the overall lack of substantial government support for literacy promotion suggests that future programs and implementers may need to advocate for this to ensure resources and capacities linked to literacy persist after MGD closeout. (c) Nutrition Behaviors of Families for Children at Home Respondents interviewed frequently reported seeing changes in nutrition behaviors at home among themselves, and to a lesser extent, among others in the community. The school meals program and children themselves were reported to be major driving forces of changing nutrition practices at home. Under MGD, children became more knowledgeable about nutrition, and began to ask their parents to cook more at home. For example, one mother in an FGD from Ak-Suu, Issyk-Kul, recalled, “When I was cooking fried buns, my daughter, who is studying in the first grade, made a remark to me that it was so harmful to cook on the stove, that it would be better in the oven, please.” Similarly, one MoES respondent recalled:  “The children got knowledge and skills about proper nutrition, and they started to ask their parents ‘Why [do] we cultivate these kidney beans for business and not use them in our daily diet for nutrition?’ Because these kidney beans will help us to use balanced diet.’” Many respondents recalled instances of children requesting their parents to cook the same food as they received at school. For example, a female director from Jayil, Chui said:  “When the students come home, the parents ask what they ate at school. Students say they ate pea soup, lentil soup. Even during the holidays, I could hear the children telling their parents: “Why don't you cook lentil soup? Don't you know what lentils are?” Parents then tried to cook lentils at home.” Some respondents, however, noted that parents struggled to replicate school meals at home, as they could not find lentils and green peas in local markets. Cooks were also reported to be an important source of knowledge for parents, with frequent reports of parents asking cooks how to make certain dishes. For example, a male cook from Aravan, Osh, recounted, “Children come home and ask parents to cook the same meals as at school. Parents come and look at how our cook’s work. For example, parents did not see lentils and they came and saw how to cook it. And we tell them how to do it.” A female teacher from Naryn stated:  “Parents come to school and see what children are eating… The parents' committee buys food every month and monitors its consumption and looks at the reports of the dining room. And at the same time learn how to cook a particular dish. And they themselves say that they also began to cook such dishes at home.” As previously noted, nutrition trainings and SBC were reported to spark changes in nutrition behaviors, particularly for change agents and those who received trainings directly from Mercy Corps. For example, a DDoE respondent from Ak-Talaa, Naryn said, “Parents usually come home from work, and feed a child, offer chips, sweets, cookies… I've done that myself. After studying with the Mercy Corps, we already have 55 a different attitude to nutrition of children. We are turning to the proper nutrition of schoolchildren.” These findings concur with the USDA Funded household survey where it states the following:  “In terms of practices at home, we find that diets are significantly better for caregivers who participated in SBC training than those who did not (Columns 1, 3, and 5). Moreover, our more robust estimations show some positive impacts of the training on the quality of diets at home (Columns 2, 4, and 6). Specifically, the odds of having dietary diversity in the household above the critical value of 4 food groups increased by 49%, based on the Minimum Dietary Diversity for infant and Young Child Feeding indicator” (MDD-IYCF). 38 The independent household level study funded by USDA and conducted by the International Security and Development Center demonstrates conclusive evidence that the MGD17 training and SBC activities have indeed had a positive impact on the dietary habits and practice of households with primary school children at schools participating from MGD17.39 This is a major success of the program and has the potential, if sustained over the long-term, to reinforce and expand the positive impact of MGD17 on primary school children’s health, wellbeing, and learning outcomes. (3) Did program activities show signs of creating unintended positive or negative outcomes? If yes, which activities contribute to this? Increased Burden Though some respondents noted that the school feeding program had increased their responsibility and made their jobs more burdensome, citing in particular the need for additional paperwork, the majority felt that the benefits made this additional burden worth the effort.40 Many parents have also taken on the added unexpected burden of contributing to school feeding programs at their kids' schools, in fact, MGD17 monitoring data shows that over 57% of program schools have regular (monthly) financial contributions from parents (on average between 100 - 200 KGS per child per month). The program has begun to view the contributions of parents as a signal of program sustainability; however, it was not an intended outcome of the program design and also presents an increased burden on impoverished families within these communities. Despite the usefulness of the trainings overall, evidence suggests that MGD trainings could create an additional burden for participants, particularly for women who still had to complete teaching, cooking, and family duties. A female librarian from Moskovskii, Chui noted, “Even if teachers are interested and want to have good results, they have their own families and homes, where they play the role of mothers, wives, and cooks. At school, teachers cannot be involved outside of lessons, and according to the program, 38 International Security and Development Center. McGovern-Dole Food for Education and Child Nutrition in Kyrgyzstan – Endline Study Final report. (November 2021) 39 Not all schools, however, reported that home nutrition behaviors had changed, and as previously noted, not all SBC change agents were successful in creating nutrition-related behavior change. Resource constraints prevented some from changing their diets at home. This particularly emerged as a problem in Panfilov, Chui, where a female teacher stated, “I have not noticed any changes in the nutrition of families. When you speak with parents, they tell you what they do not have. I propose that they just buy 2 kg of carrots and make a salad, to add garlic, onion and eat at home. They say with what money they have to buy. Other parents completely care about nutrition.” 40Interestingly the government increased salaries for teachers significantly, but not for the school cooks, that is why the program gives the school cooks rations for their personal wellbeing. 56 training is conducted on Saturdays. It is necessary to have time to do both household chores and to have time for classes at school.” Evidence of training fatigue was also found among some teachers interviewed, who were also taking part in USAID-funded training activities related to Okuu Keremet and TTR. Another female teacher from Alamydyn, Chui remarked, “You know we are so tired of these trainings. There are too many trainings… All the trainings helped us and each of them was useful. It is never late to learn but it is too much for us.” The online format of trainings could compound frustration and difficulties. One teacher from Karakyl, Jalal Abad, explained: ❖ “Truth be told, there were a lot of projects in the primary classes and the teachers were tired. There was “Reading together”, then “Let's read”, then “Okuu keremet”. Also USAID, Mercy Corps and others… They went on in a row and it was hard for the teachers… The work was endless, there were few breaks. It is psychological suffering for teachers. I will say openly, I also suffered… This is an energy loss and a negative impact on health. All day at the computer, if there is no Internet at home, you sit at your neighbor's. Online was very difficult.” The potential burden and frustrations that training activities can pose demonstrate that it is important for programs to be mindful of other competing interventions and demands on participants’ time. Training activities should be coordinated and adapted, with a focus on modifying timings and formats, in order to avoid causing undue hardship for training participants. Perceived Care MC staff shared that the distribution of rations to families during the pandemic lockdown period transformed how communities saw the MGD17 program; the communities saw that MC and donors cared beyond their targets and program. When kids were in lockdown, MC began distributions to the families of primary children and these small rations were appreciated, and it demonstrated care for their wellbeing. All the primary school children's familiesreceived the rations that the kids would have received if they went to school. The families came to the school to pick up their rations, the MOEs took on the burden of the logistics surrounding this, and they managed this on their own and reported back to MC.41 Children’s Increased Interest in Reading and Books When asked whether the program had any unexpected consequences or results, many school-level respondents reported being surprised by some of the program’s outcomes. Among these were children’s increased interest in books and learning in general, as well as changes in their behavior, creativity, and independence following MGD feeding and Positive Discipline activities. One librarian from Moskovskii, Chui, explained, “Unexpected was the desire of children to read bound books. Because every child has gadgets, they cannot be torn away from their phones… The number of students who are interested in books has increased.” School-level respondents were also surprised to see the program’s effects on increasing parental involvement and engagement with their children’s learning, with parents even becoming more interested in reading themselves. These may be very intentional outcomes, but the surprise they generated among respondents speaks to the challenges programming faced and, to some extent, the project’s success in making inroads with target populations. 41 MC Staff Interview, 12/2/2021. 57 Empowerment Training activities were linked to empowerment and changes of status for some trainees. According to Mercy Corps’ October 2020 March 2021 Biannual Report, 95 change agents from across Kyrgyzstan, including 51 women, planned to participate in the April 2021 local council elections, having gained confidence, motivation, and recognition after engaging with SBC, public health, and nutrition issues in their local community.42 Though it has not been confirmed, one Mercy Corps respondent interviewed recalled that some of these candidates were successful, having gained the trust and votes from community members. Trainings for school cooks also increased their social status. One Mercy Corps respondent recalled that the job of a school cook, which is not well-paid, typically attracts people of lower socio-economic status. However, their employability has increased as a result of building skills through MGD trainings: “In a lot of cases, they're becoming the only certified, highly skilled cook for [the] whole village. And they, suddenly, are becoming very popular for any private businesses like restaurants or coffee shops, or even for homemade cooking.” Their new knowledge has helped school cooks gain recognition as professionals and nutrition experts in the community and has consequently brought them more respect. In addition, feedback from the school cooks and teachers and librarians that received online trainings, one of the immediate outcomes was related to the increase in their perceived social status. In fact, when they were doing physical monitoring visits, intensive trainings, and learning around how to do the distance learning and protocol. Cooks and librarians said that their social status and family status had increased since an international agency was furnishing them with materials to participate in meetings and trainings. This led community members and their family to view them with more importance. This was also the case with the school cooks before the pandemic, but during the pandemic this expanded to teachers, librarians, and the community change agents.43 Another effect of increasing cooks’ capacity, however, could be increased turnover at schools, given the new opportunities that are open to them. Cooks from schools who were interviewed complained that their salaries are low and hours long, and directors noted that turnover among school cooks is high. In order to avoid this pitfall, more may need to be done to support school cooks, such as advocating for higher salaries and better hours. Mainstreaming Training Efforts Another unexpected success of MGD’s capacity-building efforts is the adoption of MGD training materials by educational institutions around the country. For example, as of March 2021, MoES and MoH were reviewing video lessons on cooking methods developed under MGD to be shared with Technica, Vocational Education, and Training (TVET) schools across Kyrgyzstan.44 A training module on Positive Discipline was also accepted by the Kyrgyz Academy of Education, which could help to mainstream positive education methods across the whole country. The inclusion of these materials in the formal training institutions across the country could be a significant step toward mainstreaming positive 42 Mercy Corps, Biannual Report October 2020 - March 2021 43 MC staff Interview, 12/2/2021. 44 Mercy Corps, Biannual Report October 2020 - March 2021 58 education methods and good school nutrition and cooking practices across Kyrgyzstan and promote the long-term sustainability of these new methods. Enrollment and School Transfers School-level respondents noted that not only had student attendance improved under the MGD program, but that enrollment had also increased at program schools throughout the course of the intervention. Though increasing school enrollment is largely due to the increasing birth rate and population in Kyrgyzstan in general, combined with mandatory school enrollment, some respondents specifically noted the role of school meals as a pull factor, with children transferring to schools where hot meals were provided. For example, a local government official from Osh noted, “Many children live with their grandparents, and they do not have the opportunity to eat well; therefore, hot meals in schools have a very positive effect on them, and there have been cases when parents transfer their children to schools where Mercy Corps implements its program.” Some schools reported struggling to feed the additional students. For example, a female director from Osh, Uzgen, recounted that they had been forced to decrease portion sizes: “A student used to be provided 2 slices of bread and now 1 slice; if the hot meals were 250 grams, they were reduced up to 210 grams… The student’s attendance has increased now, and everyone knows about the Mercy Corps.” As noted previously, some schools also reported that their canteens were too small for the number of students. Given the context of increasing population and the role of school meals as a pull factor, school feeding, and infrastructure activities should account for the likelihood of increasing student numbers when planning activities, such as making sure new infrastructures can accommodate additional students and providing additional food to schools when student numbers increase. Spillover According to interviewed respondents, some non-targeted MGD schools had been inspired to conduct similar literacy activities, such as reading camps and book festivals, as well as to start their own hot meals programs. Respondents also noted that trained teachers shared their new knowledge with other teachers, including those of higher grades than those targeted under MGD. Training district-level officials also meant that district-level specialists could pass on knowledge to non-targeted schools. These spillover effects are a positive indication for indirectly increasing program coverage for participants and the sustainability of program teachings. (2) To what extent has the COVID-19 pandemic and its associated disruptions affected program outcomes for other components? ❖ “The MGD17 outcomes were not disrupted at all. We did everything except feeding the kids, we did more infrastructure, latrines, and handwashing, and a lot of the literacy promotion activities were conducted remotely. I do not think we were disrupted at all, but we learned how to adapt and be more efficient and effective. We did not stop when many other programs had to stop.45” (MC Staff Interview, 12/15/2021) 45 MC Staff Interview , 12/15/2021. 59 MC staff were quite clear that, despite the challenges presented by the global pandemic, they were able to overcome them through a creative and adaptive management style that has led the MGD17 process with the flexible and needs-driven oversight of the USDA. The monitoring and evaluation data suggests that the program has achieved its principal aims as highlighted above, with the only caveat being that school closures did not allow the program to meet the indicator of the number of meals provided to students. However, the household distributions made to primary school children’s families and the added WASH infrastructure outputs were unplanned humanitarian support that the program provided. Most of the activities had to be suspended in mid-March 2020 when the schools were suddenly closed across the country due to the pandemic. However, in the majority of the target schools, teachers and librarians were able to quickly adapt to the new circumstances and work with their students and communities remotely using WhatsApp and other communication tools to ensure that their students keep reading while at home. Because this was a sudden and rather drastic change in many communities that are not used to working remotely and using online tools, the MC literacy promotion team organized a series of technical support sessions to share tips on working remotely and suggest various technical solutions tailored to the local context. At the time of reporting, most program schools continued implementing literacy promotion activities remotely and regularly reported to MC through WhatsApp and email by sending photos and videos of their work. Teachers, librarians, cooks, and school administrators consistently report that the COVID-19 pandemic and related school closures have challenged parents tremendously. For example, one teacher at a primary school in the 2018 cohort in Batken province said “It was difficult in the beginning. Many children do not have smartphones, and some of them did not understand how to use applications. Literacy levels have decreased. Children became lazy.” School staff also said that the arrival of the COVID-19 pandemic also encouraged parents to appreciate teachers and schools more than they would have otherwise. Librarians and teachers also widely reported that parents became much more involved in their children’s education and literacy promotion during the pandemic in most, but not all, cases. These factors will likely translate into even greater improved literacy outcomes over time due to renewed respect and esteem for teachers, as well as increased engagement of many parents. Impact Criteria Conclusions Impact Conclusion Rating Observations The data highlight an 8- percentage point increase in the number of reads with comprehension, while the nonreaders remain at 5%. While this is a single digit improvement in the reading comprehension results from the baseline, in the backdrop of the ongoing pandemic and school closures, these results are promising and suggests that the efforts of the GoK and MGD17 are reversing the years of malnutrition that had Satisfactory ● Evaluation, monitoring and independent household study evidence suggests that there has been significant progress improving health, nutrition and dietary practices at school and in the home. ● The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. ● MGD17 monitoring for 2020-2021 shows that 87% of program schools have a diverse school menu (i.e., the same meal is not repeated more than twice per month, and all food groups are included). 60 previously undermined the performance of school-age children in Kyrgyzstan. ● MGD17 monitoring for 2020-2021 shows that 78% of program schools that have school feeding (complete meals) five days per week ● MGD17 monitoring for 2020-2021 shows that 57% of program schools have regular (monthly) financial contributions from parents. ● The program performance and impact suggest that while the pandemic presented challenges to program implementation, it did not block the MGD17 from achieving its program outcomes related to SFP, literacy promotion and learning outcomes for primary school children at participating schools. Impact Recommendations Recommendations Timeline and Prioritization Duty Bearer 1. To fully explore the impact of the MGD17 program, a quasi￾experimental design is required that compares the performance of MGD 17 schools with WFP and schools that do not have hot meal programs. MC should work with GoK to explore the possibility of conducting this type of study. There should also be control for TTR and related literacy programming to ensure that the true value￾add of MGD holistic approach is captured. March 2022 MC, USDA & GoK 2. An internal study of MGD12 & MGD17 graduated schools to explore the continued impact and sustainability would also be beneficial for exploring lessons learned and best practices for ongoing and future programming. There should be some attempt to explore the impact of the ongoing pandemic and how to support the GoK and local communities to find solutions. End of 2022 MC, GoK and USDA Impact: Lessons Learned 1. The SBC component has had a multiplying effect on the program impact by changing behaviors in the school setting and in participating children’s’ households. 2. While the pandemic did cause some challenges to the program implementation, USDA, the MGD17 team, and the GoK were able to adapt the implementation and programming approach to ensure that the program was able to achieve its outputs and outcomes. 61 Efficiency The OECD-DAC defines efficiency as, “the extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way;” in other words, “how well are resources being used?”46 Here, we will explore how the MGD17 was managed to ensure performance and impact. This includes questions related to the management and oversight of the program including programmatic and financial adjustments, feedback incorporation, and how the program adjusted to COVID-19. (1) Have any programmatic or financial adjustments been made during the course of the program? Why? If so, were they appropriate to the situation? (2) To what extent has Mercy Corps been able to adapt program implementation during the COVID-19 pandemic? MGD17 was managed from its outset with an adaptive and results-oriented lens and leadership. This allowed us to address design duplications (with other USG funded programming) at the outset of the program and flexibly and appropriately adjust the program design and budget allocations as new challenges and information emerged. Here we highlight the most critical adjustments made during the program life. Beginning with the adjustments highlighted by MC staff as the most critical programmatic and financial adjustments. ● Adjustment 1: COVID-19 household-level rations; due to COVID-19 shutdowns in May 2020 when it became clear that the schools would not reopen. Schools closed on March 16, 2020, and we realized the risk that if schools would not be reopened, that the commodities would not last and would spoil if left in the schools. MC worked with the MoES to allow the schools to organize the distribution of the commodities as rations to families of the primary school children who would normally receive meals as part of the SFP. The MoES and schools organized and led the distributions, while MC provided guidance. Some schools took what was left and distributed it equally. Others provided the rations to vulnerable families. The schools themselves decided on this approach. ● Adjustment 2: Increased distributions transported to the schools; in February 2021, MC assessed the commodities distribution process and revised and developed new SOPs for distribution. The MGD17 team realized that managing, transporting, and storing such large quantities of commodities was a tremendous burden for many schools, especially those far from urban centers, particularly Bishkek. Thus, MGD17 began bringing the commodities to schools, and the number of distributions increased from 1 per year to 3-4 per year. MC assumes the transport to the schools and manages the security, hygiene, and preservation of the commodities for a longer time. This approach is better for their program schools, elevating a major burden and is also preemptive in case of another lockdown due to COVID-19. ● Adjustment 3: The literacy component; when Save the Children ceased their operations in Kyrgyzstan in 2019, MGD17 integrated the literacy program into the SFP program and utilized the same approach for the monitoring and oversight for both components. This adaptation was also predated by the redesign of the literacy program in the original design, as they planned to work with teachers and books, but TTR was doing this, so they brought in the positive discipline, which was the most significant modification. 46 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm - efficiency-block 62 o It is also worth noting that the literacy team organized a reading marathon amongst participating schools; each school had a book by their selected community leaders and celebrities on camera and then put it on WhatsApp or Facebook, and then passed the virtual baton to a new school. o In the face of COVID-19, the literacy team worked with schools to organize online reading camps in lockdown. They were using WhatsApp, Skype, Facebook, and students and families used their own personal devices to participate. MC provided guidance and digitized their books so that teachers and librarians could use them. MC Kyrgyzstan won an award for innovation from MC for this work. ● Adjustment 4: Infrastructure; MGD17 designated infrastructure projects on needs-based assessment. Also, they increased the WASH as a response to the needs created by COVID to facilitate school openings as well. Winterization was not as critical as WASH for school functioning and the SFP. The needs assessments were also a critical step because they confirmed the needs related to SFP and latrines and identified schools that had the infrastructure but needed to be renovated or rehabbed. Most schools did not have the infrastructure, so they also identified 32 schools that did not meet the minimum requirements for WASH and sanitation. MC then managed the infrastructure process, hired four companies, and managed the rehab and infrastructure. This was significant and urged by the pandemic putting WASH at the center of the program and priorities for school reopening. Many schools have handwashing in their dining area, but these are not available outside of the cafeteria, which closes when meals are done; MC sought to address this through new infrastructure projects in 32 schools. These handwashing stations were by the schools' entrance, so now everyone has access. ● Adjustment 5: SBC was not part of the original program design. Parents were originally to be trained for 16 hours but this approach was not practical. In early summer 2018, MGD17 looked at adding the SBC component; they began with a basic USDA approach, and then did a gender assessment in 2019. They then revised the curriculum to be more appropriate in terms of audience and content for parents including men and SFP/nutrition. Registration rates for men in the SBC program are up to 30% now, a major increase from 9.5% in the first year of SBC implementation. COVID-19 Disruptions Overall, COVID-19 had a severe effect on the context under which Mercy Corps operated in Kyrgyzstan and necessitated a number of program changes and adaptations. These included pivoting school feeding to take-home rations, moving trainings and extra-curricular literacy activities online, and expanding MGD’s WASH component, such as distributing liquid soap and providing grants to schools to establish handwashing stations with running water. Program adaptations to COVID-19 were largely received positively and contributed to making MGD relevant and effective despite the challenging circumstances. However, the online format of trainings and activities sometimes posed challenges. Another result of COVID-19 was that school closures in many cases also led to increased parental involvement in their children’s education. However, as touched upon previously, the negative economic impacts of COVID-19 present an obstacle for parents’ ability to contribute to school feeding programs, and consequently, sustainability. (2) How do program staff and stakeholders perceive the effectiveness of the processes and resources used by the program? In general, all program stakeholders are very positive about how the MGD17 was managed and how the resources were used. Whether school staff, government representatives or MGD17 staff the general 63 sentiment was that the management and resources employed to implement the program were effectively employed and managed. School Administration  “They give us the necessary and useful products for children. We used to have problems with delivering products to school. We had to deliver from Naryn to Chaek, and it costs a lot of money. Moreover, there are requirements for the vehicle which delivers the products. Recently, the Mercy Corps themselves have solved this problem and are delivering products to the school themselves. And it was very difficult for us, aiyl okmotu could not help, and the parents did not want to understand. We covered the delivery expenses ourselves. Then we asked Mercy Corps to solve this problem, because finding a vehicle to deliver to the village is a big problem. We couldn't use a vehicle transporting coal or animals. We needed a special vehicle for transporting products. The main thing is that we are provided the necessary products. We have an issue with meat. Since meat prices have increased so much, no one is buying it. Even the tender was not announced. So currently we cook without meat.” (KII Director, Naryn Jumgal )  “I don't have additional plans regarding the program, but the project works now and we together with parents would like to extend the program on providing hot meals.” (KII Director, Jalal-Abad Aksy)  “I would leave as it is. Because we see how the program works. Children have become better at studying, vocabulary has increased, teachers are working well on this program.” (KII Director, Issyk￾Kul) AO & DDoE  “I rate it at a high level. Mercy Corps finances a lot of schools in Kyrgyzstan. This is a great amount of money. We express gratitude to Mercy Corps for the work it does.” (IDI Local government, Jalal-Abad Suzak)  “I would leave it as it is. We will not be able to make more than our budget allows us.” (IDI Local government, Jalal-Abad Suzak)  “I would leave it as it is.” (IDI Local government, Talas Bakai-Ata Kyzyl-Oktyabr)  “The changes may be made depending on requirements of life and society. For example, growing apple gardens gives the opportunity not to buy fruits and use your own products.” (IDI Local government, Naryn Ak Talaa)  “Enlightenment, enriched food, installation of sinks, latrine,” (IDI Local government, Naryn Ak Talaa )  “I would leave it the same.” (IDI Local government, Batken Leilek )  “I really like this program, I would like to just wish it a success, since, in my opinion, everything has been worked out to the smallest details and there are no complaints, therefore, everything needs to be worked at this pace.” (IDI DDoE, Talas Bakay-Ata)  “It is such a pity that this program came to its end. The only thing I would like is that they would continue to work. Also, the groceries that I have listed above, such as: rice and oil, I would leave them, because after the pandemic the situation in the village is difficult. Secondly, I would leave the cleaning supplies to the napkins and sinks.” (IDI DDoE, Talas Bakay-Ata) MC Staff 64  “We spent donor resources very efficiently and very effectively; we sought the best supplier and deals for the program in terms of price and quality of commodity or resource. We sought to be efficient and timely in line with the contract and USDA procurement regulations. In the program, they changed the program monitoring approach, they were doing monthly before, we changed it to hiring field specialists that were based in the field, so this minimized the travel cost of the project so that staff were based in the field, and they were able to be based in the field. We have different categories; we had training field specialists for the food component. We also had field-based engineers that had supervisors that managed their work remotely. They report to them monthly; they use WhatsApp to communicate on a regular basis.” (MC Staff Interview, 11/24/20221)  “Positive perception, the MOEs and MOH, provided letters of support immediately, without hesitation. MC is advocating for the continuation of the program asking the donor to continue the program because of the relevance and need for the program's continuation.” (MC Staff Interview, 12/2/2021) The school and government representatives confirm the responsiveness and good stewardship reflected in the management and implementation of MGD17. They highlighted the responsiveness of MC staff and repeatedly stated that they would not change anything or would like more activities or schools added to the program. Many were reluctant or apprehensive about the end of the program. MC staff concurred with this position or expressed their pride about their stewardship and management of MDG17 over the last five years. (2) To what degree was feedback from the previous evaluations and stakeholder input incorporated into the program? To explore how the program has incorporated the feedback from previous evaluations and stakeholders, we have included a table of the recommendations from the midterm evaluation, which integrates feedback and recommendations from the program stakeholders, participants and implementers; as well as the analysis and observations from the evaluation team, for further detail see table 11. Table 11. Midterm Recommendations and Current Status Relevance Recommendations Timeline & Prioritization Duty Bearer MC Management Response as of January 2020 November 2021 & Comments 1. MC should also be clear with nomenclature and avoid calling the promotion activities a literacy component when TTR is clearly a literacy-focused project working in the same schools. January 2020 MC and literacy team MC will coordinate with Save the Children HQ and USDA the nomenclature. Was done in 2020 and resolved. 2. MC should organize a meeting with the GOK and school reps, to take stock of the literacy and learning promotion activities and determine the appropriate way forward to maximize the outcomes ASAP GoK, School Reps, MC To maximize the literacy promotion and learning outcomes MC is planning to conduct the meeting with MOES, DDE and school representatives to share and discuss the Finalized, due to the Covid-19 pandemic and the desire to encourage the participation of external stakeholders (Ministry of Education, Ministry of Health, local authorities, school administrations 65 associated with this element of FFE17. Blind acceleration of activities may not be enough to yield meaningful results. results and plans of the Literacy Component of FFE17. The meeting is scheduled for February - March 2020. There is also a new Manager position in the south office for the literacy promotion component, which is also in compliance with MC structure. and etc.) the meeting place in September 2020 online. 3. Gender training is a critical step in ensuring that staff considers gender and diversity in their daily work and planning. It will be important for each of the staff to reflect and document how they individually will integrate what they have learned from the training into their work at the individual or component level. ASAP MC All team members have committed in their annual self-development plan to consider gender and diversity in their daily work. Gender and diversity refresher training for staff members is planned to be conducted in February 2020. Trained staff members will provide updated individual plans on how to integrate what they have learned from the training into their work to respective managers. GEDSI and SG trainings were completed for all MC staff following standard MC guidelines (the first training was conducted on June 21- 22, 2019, the second one - on March 4-5, 2020). 4. It will also be necessary for the MGD17 program and support staff to hold a workshop to determine how they will integrate findings from the gender assessment into the program during the final years of the project. Once gender assessment complete MC Workshop on integration of findings from the gender assessment into the program is planned to be conducted with support of external Gender experts. Prior to the gender integration workshop MGD MC program team will come up with proposed ideas for integration that will be discussed with the entire team. Completed, this was addressed via SBC revised interventions. 5. MC should consider the recommendations from the program participants and assess what they can include in the next two years of program implementation. Additional recommendations that may not be addressed during the remainder of the September 2020 MC MC has planned the series of workshops for program participants and exchange visits to the best performing sites in February- March 2020. As one of outputs of this activity key findings and recommendations will be Completed, due to COVID19 pandemic most of the group workshops were canceled starting April 2020. Priorities changed due to epidemiological situations as all schools were closed and students studied from home for an extended period that reflected in the quality of education in general. However, 66 program should be shared with GoK as they continue the SFP without donor support. documented to be shared with GoK in September 2020. experience exchanging between program participants continued via WhatsApp groups among school cooks, school administrators, and district departments of education specialists. Effectiveness Recommendations Timeline and Prioritization Duty Bearer MC Management Response as of January 2020 November 2021 & Comments 1. SBC has no literacy component; MC should look at integrating literacy into the SBC planning and activities to the degree possible. (10/7/19, MC staff) April 2020 Program component leaders are designing integration points of literacy activities into the SBC activities. As one of the topics for the next year SBC rural outreach activities will include the literacy promotion module. The module and corresponding IEC materials will be developed jointly by SBC and Literacy teams. Key messages focused on family members support to children on reading will be distributed by program volunteers-activists in all program target sites. This activity will be complemented by broadcasting of literacy promotion video spots on national regional TV channels as well as other SBC platforms. Completed, literacy promotion was included as one of SBC components immediately and broadly disseminated via social media channels as of 2020. 2. The impact study should be designed to include clear questions regarding the SBC activities that occurred in the communities, their impact in addition to the questions regarding knowledge and behaviors at the household level. (2) Also, literacy should be included in the impact study to look at community￾level knowledge and behaviors linked to learning and literacy promotion at the community level. May 2020 MC Done Completed, this is included in the household-level research that culminated in the Nov. 2021 report. Phase one (“baseline”) was done in early 2020 right before the onset of the pandemic, and phase two (“endline”) was completed in Nov 2021. However, a quasi￾experimental design with WFP and schools that do not have hot meal programs would be the best design to fully explore the MGD-FFE model's impact in Kyrgyzstan. 67 3. MC will have to be very aggressive in the final two years of the project in order to ensure that all infrastructure projects are completed in a timely and effective manner. It will be important to develop a very clear, well-coordinated strategy with the infrastructure team with regular reporting and updates to MC senior leadership with external oversight from an external oversight from an infrastructure expert. May 2020 MC Activities toward the quality improvement of performance and outputs of infrastructure projects have already been initiated. To support these activities, professional engineering companies were contracted to ensure 1) capacity building of the infrastructure team, and 2) the regular monitoring and reporting on all infrastructure projects. For the third year of the MGD program, around 100 infrastructure projects will be completed with USG compliance and the new MC monitoring and reporting approach. Completed, all infrastructure projects were completed during LOP. 4. The evaluation data suggests that extensive analysis/assessment is required of whether schools have received a comprehensive literacy and learning promotion package. In addition, as MC is merging the monitoring data from the literacy and learning promotion component into their databases, some work should be done to validate the quality and accuracy of that data. April 2020 MC Literacy Managers along with other component Managers will be trained in January 2020 to strengthen data analysis/assessment skills and will be obligated to perform regular data analysis and reporting. Technical Literacy MTE will be conducted in mid￾February, and the same methodology will be used to be consistent with the baseline conducted in 2018. Done 5. MC senior leadership needs to apply some of the lessons learned from the SFP to the literacy and learning component regarding follow￾up coaching and monitoring. Improved coordination and integrated management of the two components will also be important moving forward. January 2020 MC Well established and practiced follow-up coaching and monitoring activity of the school feeding program have been replicated to the Literacy Component. The same structure of field specialists and data collection and data management is implemented as of October 2019. Done 68 6. The findings suggest that FFE17 needs to provide some support to schools for developing strategies for schools and the GoK to successfully manage infrastructure projects including the budget, oversight and quality control. September 2020 MC The program will incorporate this recommendation in the work plan and discuss it with GoK and development partners. Once agreed upon, a strategic approach will be introduced to successfully manage infrastructure projects including the budget, oversight, and quality control will be introduced to schools. Due to the COVID pandemic, Gov priorities changed significantly, and the main focus was done for more WASH-related infrastructure improvement at schools like hand￾washing stations, which was done very successfully and jointly with local Gov. 7. MC should consider providing guidance and recommendations for schools and GoK/MoES (1) regarding regulations for the grading and staffing requirements for school nutrition programs and (2) administrative timeline and process for schools to submit paperwork for nutrition staff to be graded and salaried appropriately on time. Both these actions should help to address the challenges raised related to workload and salary. September 2020 MC Program will incorporate this recommendation into the work plan and will discuss it with GoK and development partners. Under discussion with WFP, UNICEF, Nutrition Alliance, and others 8. MoE’s ministry-level infrastructure is lacking, and all school nutrition and teacher training relies on one person; MC, WFP, and USDA to advocate and support the construction of a MoES team that can fully oversee responsibilities for SFP as they prepare for the closeout and handover MC will discuss this issue with WFP and USDA. MOE has been undergoing numerous leadership changes since 2020, and new leadership priorities were mainly focused on COVID￾related education outcomes, not necessarily SF. 9. MC, WFP, Gok, and USDA need to develop an exit and handover strategy for all SFP interventions as they prepare to close out. The sustainability model mentioned in FFE17 guidance, as well as recommendations for MoES staffing and all ASAP MC, WFP, USDA, GoK MC drafted its exit strategy and recommendations for MoES along with other integral issues for sustainability. Done 69 other issues, should be addressed in the exit strategy. Efficiency Recommendation Timeline and Prioritization Duty Bearer MC Management response as of January 2020 November 2021 & Comments 1. If MC and the GoK are not able to find a solution for banning flour that has tested positive for GMOs, they should prepare a strategy with the GoK on how they will inform schools in advance so that they can prepare for the change in their commodity baskets. ASAP MC, USDA, and GOK Likely the international lab tests, both from Germany and from Russia, have not been confirmed of GMOs in wheat flour. The remaining lot of wheat flour will be distributed according to the distribution plan in February 2020, but no wheat flour is to be distributed by the program in 2020. Ongoing, no news for FFe17, but negotiation with the Gov is ongoing for the expected flour commodities as of the FFE2021-26 period 2. MC should employ the best practices in oversight/monitoring and sustained/remote engagement across the components by conducting joint monitoring visits for literacy and learning as well as introducing WhatsApp groups to the literacy and learning engagement and learning model. January 2019 MC Done Completed and results are observed in the literacy program results and outputs. 3. MC should consider a limited benefits package that can be offered to contractors to provide some healthcare support in consideration of sick days. The updated payment mechanism for the contractors was done as of December 2019. Completed 4. MC leadership should consider reinforcing the SBC and governance liaisons with regional/provincial level focal points to help expand the effectiveness and impact of the program and SBC activities. February 2019 MC MC conducted 41 meetings with 835 regional/district level governance representatives. MC programs’ accomplishments, ongoing activities and plans were on the agenda of the meetings. One of the important issues that have been discussed was sustainability of the program. An output of these meetings is that all Completed 70 local government reps agreed to allocate funds in the annual budget to kitchen equipment. Efficiency Criteria Conclusions Efficiency Conclusion Rating Observations McGovern-Dole 2017 performed well across all the dimensions of efficiency. The evaluation data shows solid financial and programmatic management performance. The MGD17 team and USDA applied its extraordinarily adaptive and innovative leadership style to ensure the program was managed for results and address emerging needs and crises like the pandemic. Highly Satisfactory ● The program made multiple programmatic and management adjustments to ensure program success and efficiency, which included everything from early revision to the program design, adjusting the infrastructure approach repeatedly to adjusting program components to respond to the ongoing pandemic. ● MGD17 leadership and USDA were able to maintain program implementation during the pandemic and added household rations to primary school student’s families, online training and activities for the SFC and literacy promotion components. The program also amped up the WASH component to encourage the school reopening and fight the spread of COVID-19. ● The program performed well in terms of financial and personnel management efficiency. ● The program oversight and management processes are viewed positively by most stakeholders. ● Lastly, most if not all feedback emerging from the midterm evaluation has been addressed or completed despite delays caused by the pandemic. The midterm recommendations are the result of the combined feedback and recommendations from stakeholders across the program. Efficiency Recommendations Recommendations Timeline and Prioritization Duty Bearer Given MGD17’s and USDA’s experience and success with program continuation and remote/virtual activities during a pandemic, MC and USDA should develop guidance documents and best practice materials that can serve as guidance and training materials for related programming and staffing to prepare for future shutdowns and similar situations. End of 2022 MC, USDA and GoK 71 Efficiency: Lessons Learned 1. The flexible and adaptive implementation style employed at the outset and throughout MGD17 by USDA and MC permitted the program to innovate, ensure effectiveness, relevance, and impact, and ultimately enable the program to successfully continue program implementation during the pandemic when many other projects halted or stopped implementation. 2. The data-driven and needs-based adjustments and modifications made to the SFP, literacy, infrastructure, and SBC components throughout the life of the award were critical to achieving the results and successes resulting from the 5-year MGD17 implementation. Sustainability In the context of this evaluation, this report considers sustainability to refer to the ability of participants to achieve and continue the two primary program outcomes—namely, school feeding and improved literacy promotion—after “graduating” from the Mercy Corps intervention. This description of the term grows out of the language of the evaluation questions themselves and adheres to the OECD-DAC definition of sustainability as an evaluation criterion: “the extent to which the net benefits of the intervention continue or are likely to continue.”47 In this light, sustainability was assessed based on the best judgment of the likelihood of program outcome continuation after the intervention period. Though the National Law on School Feeding bodes well for sustainability, the amount of money allocated under this law (7 to 10 KGS per day per student) is not considered sufficient to provide students with an acceptably diverse diet. Evidence to indicate the long￾term sustainability of government support for improved literacy instruction practices is yet more sparse. Arguably the other starkest challenge to MGD programming is its sustainability. Repeatedly, interview respondents told us how much they wanted Mercy Corps to continue its programming, and how difficult it would be for the schools to sustain school feeding—let alone improved literacy promotion— independently. As a cook in Jalal-Abad, remarked: “I think [that] if Mercy Corps leaves, we'll probably be like orphans, on our own. We would like to continue working with them.” That said, there are signs that some schools are successfully striving toward continuing school feedings with the support of parents and, in some relatively rare cases, local governments. (1) To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue primary school feeding beyond the program? Almost unanimously, school administrators expressed a strong desire to continue the program, with the vast majority also saying that they wished to have Mercy Corps’ support continue. Staff at a minority of schools flatly declared that they could not continue the school feeding program without Mercy Corps’ support. Many school staff expressed intent or desire to continue school feeding but also noted they had no or very limited capacity to do so as illustrated by the quotes below:  “We have planted potatoes and we will plant carrots and will keep planting every year. The state allocates by 7 KGS. We will plan. When developing the menu, I will take into account students and we will calculate how much we will get from potatoes, fruits and of course, we will ask for support from 47 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm 72 parents, and we will act on this basis. ……As I mentioned earlier, we worked with parents and have informed them. ……As per Order of the Government all schools should have hot meals until 2024. That’s why Mercy Corp supported us, and it is our duty to continue with the program.” (KII Director, 4/5/2021)  “We have plans. Parents mostly have possibilities to support in autumn, when they have a harvest. They gather nuts and apples. We are planning to hold conferences. It is the first stage. Secondly, there are good people in our village who help low-income families. We can name them as sponsors. We will work with them……. There are no challenges. There were no parents who said that they would stop supporting. Many of them say that they need to support themselves further as children get used to having meals.” (KII Director, 4/13/2021)  “As I said before, we will be provided flour. The rest of the products can be provided by parents. Maybe one of the parents will finance the meat, or they will collect more than 10 KGS. Maybe the state will allocate not 7 KGS, but 10 or 12 KGS per student. Our president has changed and maybe he will pay more attention to the development of children. If not, we will look for sponsors. I would also like to appeal to our businessmen. …..We need to work and have a desire. We need to be able to find an approach. We need to look for donors or write a project.” (KII Director, May-2021)  “The republican budget allocates an amount for each student. I also hope that parents will help us.” (KII Director 4/13/2021) As these quotes demonstrate, most school administrators see the value of the hot meal programs and are committed to continuing these programs. However, there are doubts about how they will continue the program once the USDA programming is concluded, most plan to seek a combination of three things (1) more support from the government, (2) support from parents and community members or (3) new donors to help support the continuation and sustainability of their hot meal programs. According to Mercy Corps’ monitoring data, parent and community contributions are already a critical sustainability model. In fact, 546 public-private partnerships facilitating regular parental or community contributions to support school feeding programs have been established at target schools due to MGD17 assistance, far surpassing the target of 453. In total, just over 1.6 million USD in cash and in-kind contributions were donated by private entities (parents and other community members) to target schools in order to support school feeding programs and other related activities, more than tripling the target of 479,000 USD (over the duration of the program). The quotes above also highlight the potential for orchards/horticulture/school gardens as markers of the increased likelihood of sustainability.48 In these cases, the staff and community are investing resources of their own in the food production enterprise, which is likely to have a cascading effect on further local resource allocation. Most schools intend to continue providing hot meals to students during the school day, but do not have definite plans about how to do so. Although the national government contributes 7 KGS (10 KGS in particularly remote or high-altitude communities) per day per student to schools for school feeding, universally school 48 MGD17 monitoring data shows that 113 schools of the 440 participating schools have gardens established through the School Orchards project funded by a private donor to complement their school feeding program. Some schools have their own gardens, developed without support form USDA, no data is available regarding these privately developed gardens and orchards. 73 directors and other staff said that this amount was not sufficient by itself to provide an acceptably diverse diet for students. 49 Some schools do not feed children a full week without parental support. There may be some regulatory hurdles preventing local food procurement in certain cases, which reportedly limits parental contributions. One primary school cook:  “It would be possible to use parents as sponsors, but we have to get products from the place that won a tender. This is for [the] purpose of safety. God forbid anything happens to the children. But it would have been better if we had been given a document to receive food from the parents. Many parents would like to be sponsors. Yes, for one year our parents helped us, but then we were told that we could not do this without proper documents.” (KII, school cook) Although it was not clear from the data, we suspect that this documentation issue is around the provision of meat products, in particular, as other respondents also mentioned this regulation. In addition, parents from different regions and villages, have very different abilities and inclinations to pay for hot school meals. Moreover, the economic situation in Kyrgyzstan since the start of the pandemic has worsened considerably, and parents’ ability to finance school feeding is limited. We do not see this limitation as reflective of a lack of enthusiasm for the program. For example, at a primary school in Chui, that started receiving Mercy Corps support in 2018, parents declined to contribute after the program support ended.50 A cook at the school said:  “There was a meeting at the school level where all parents participated, and they totally refused. Initially, after the quarantine, when the first grade started to study in September, Mercy Corps had not yet delivered food, so we asked the milk carriers to help us with the purchase of flour. We asked, but there was no help.” Some schools saw parents begin to contribute but stop during the quarantine. For example, one school in Chui was receiving parent contributions before the COVID-19 pandemic started, but it is no longer. A cook at the school, which currently has hot meals four days a week and tea and buns the fifth, said:  “Parents have supported us for two years. We had good meals in 2018-2019. Parents supported and [the] meals were very good. We had hot meals all 5 days. Now we don’t have support from parents, and they don’t collect. Now we have only 7 KGS [from the state] and food commodities from Mercy 49 It must be noted that there is a range in terms of extent of continuation of school feeding. Some schools that we are reporting here as having achieved sustainable school feeding are schools that do not have hot meals all five days a week after the end of the Mercy Corps MGD intervention there. For example, a cook at a primary school in Kemin in Chui province, from the 2017 cohort, described how they were only able to provide hot meals three days a week after Mercy Corps concluded its support. They said that although parents supplemented the state allotment, on the “other day[s] there is only tea and bread. Before we used to give them hot meals 5 days a week thanks to Mercy Corps products. Of course, it is very noticeable. You can't buy anything with 7 KGS.” 50 Oftentimes, it was difficult to parse out distinctions between intent and capacity to continue school feeding. School staff often seemed to want to believe that parents would contribute, without having received any firm commitment that they would—let alone having received consistent, sustaining support. For example, a director at a primary school in Moskovskii, Chui said, “I believe in [the plan] because our parents are friendly and they do not have arguments, so they will help us for sure...If this project ends, parents are ready to help us.” 74 Corps. Parents would agree, but because of quarantine, because children studied and then did not study, their parents stopped providing.” This illustrates that the school administrator’s sustainability approach based on parental contributions is precarious. Given the ongoing pandemic and the economic conditions in various parts of Kyrgyzstan, it suggests that the potential for the continued sustainability of the hot meal program may be undermined if they rely heavily on parental or community level contributions. Local and National Authorities on School Feeding  “During the pandemic, they (MC) told us not to collect money from the parents of the students, so we did not collect it. But at that moment it was difficult for us because 7-10 soms, which the state gives us, were not enough for school meals. The parents came and said that there was no need to stop this since children sometimes do not eat at home and, but they eat at school because they like school food more than the home food. But, of course, there were also parents who said that the state should provide schools with food. There were such misunderstandings among the parents. I think that every parent should think about the health of their child and ensure a good life. At that moment there were two opinions, most of the parents wanted to pay for hot meals and others did not want to pay and even wrote applications.” (IDI DDoE, May 2021) Regarding the responsibility for ensuring the sustainability of the hot meal programs at MGD17 schools, there seems to be a tension between whether the responsibility falls to the parents or the government. In this quote above from a DDoE representative, the onus is placed on parents to do more, while the parents feel the government should do more. This seems to be the fundamental challenge confronting the MGD17 schools as the program comes to an end. Occasionally, Ayil Okmotus have committed to at least considering replacing the aid that Mercy Corps provided, although local governments rarely seem to have the available funds to fully replace the support a school has received from Mercy Corps. A common scenario is represented by a primary school in Tyup, Issyk-Kul, in the 2017 cohort where a cook said:  “We thought that the children could eat all 5 days a week. We are working in this direction. We applied to the Ayil Okmotu and they said that they would consider this issue. Most importantly, I believe that parents will support us, from the very beginning they have been supporting us.” Fundamentally GoK and its partners USDA, MC, and the WFP have put in place the basic structures to support sustainability at the national level, with the passage of the National Law on School Feeding, along with the provision of the 7 or 10 KGS per student per day provide the legal and financial framework for school feeding to continue nationwide. However, the data suggests that increased resources are necessary for schools to provide and sustain a diverse, nutritional daily diet for each student. 75 (2) To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue improved literacy instruction practices beyond the program? As it relates to the literacy component the data from the interviews were mixed relative to the likelihood that the activities will be sustained: School Administrators on School Feeding  “It is not easy. If the Mercy Corps program ends, it will be hard for us. …….It is also not easy. Everything depends on money. And organizing electronic versions is also not easy. But we will do our best.” (KII Director, Naryn Jumgal)  “To contribute with the program, Mercy Corps leaves us with samples and experience. It has trained us. To continue the program, I will give instructions to my deputies to include it into the plan. I hear that those products that we did not get during the pandemic, Mercy Corps is distributing, and we would like to thank Mercy Corps for this. I have also heard that Mercy Corps will distribute humanitarian aid. I am glad.” (KII Director, Jalal-Abad Suzak )  “Yes, we have experienced teachers who are mentors. We also hold conferences and meetings…. We will look for other organizations through the websites, and ask sponsors to support in terms of purchase of books…..I think they are feasible. We will make an effort.” (KII Director, Jalal-Abad Aksy )  “Regarding the resources we have good books. We have good resources as I told you. We received new books from USAID. We have a lot of resources. We sometimes organize contests for children and parents, and they are interested. We previously held contests and in the future, we will continue to work for children to increase the level of literacy and give an incentive to them. ….Our programs, material resource base supports us to achieve these goals. Parents help me. They work closely with us, participate in various contests. I try to connect parents with the school and the school with the community.” (KII Director Issyk-Kul )  “Of course, the work will continue. We will always search on the internet and use other resources available to us.” (KII Director, Chui Moskovsky )  “It is necessary to continue the same program. In the planning for the next year, we have included all the programs that Mercy Corps has shown us. Camp, reading fairy tales, parent community, Reading Together program, Reading Couple program, Baby Books program, Reading of the Year program, research programs, etc. All these plans have been implemented. All our resources are sufficient for this. Everyone will contribute to implementation of these programs: the children, parents, teachers, and the school administration.” While most of the respondents were affirmative about their desire to continue the literacy promotion component activities, the contributions above show that some of the school directors highlighted that it would be quite challenging to continue the literacy and learning promotion work without support. In contrast, others were positive about their ability to utilize the training and tools provided by MGD. Some schools have developed plans to work with local governments, parents, graduates, or other sponsors to further build their book collections. One school director discussed their plans to continue the literacy promotion work:  “I would like to strengthen our library ourselves, too, but we will not always wait for the Mercy Corps’ help. We find sponsors. There are also our graduates who are ready to help in this matter.” 76 While this type of statement is not particularly detailed as far as the approach is concerned, it nonetheless indicates an enthusiasm to continue the literacy promotion techniques and capacities that Mercy Corps helped instill. The other respondents either could not or did not discuss any such plans. Local and National Authorities on Literacy Like the school staff, despite their strong desire to continue the literacy program amongst local and national representatives, there is not a consistent sentiment regarding whether this is feasible and how they would sustain the literacy promotion activities at the closure of the programs as illustrated by the contributions from local and national representatives below. Ayil Okmotu (AO, local self-government executive body)  “I think the schools will continue. We will provide funding to schools from the JSC.” (IDI Local Gov Talas Bakai-Ata )  “We mainly support schools with office equipment, computers. We arrange e-books. We finance at our own cost. But we can’t finance it to the full extent. It should be the task of the Ministry of Education.” (IDI Local Gov Batken Leilek )  “We can help in compliance with the law. Each mayor’s office or ayil okmotu are not the same. There are a lot of them subsidized. There are a lot of ayil okmotu which can’t support rehabilitation even in the small school. No two towns are alike. We pay 60-70% of the school rehabilitation costs. We can’t direct finances only for education.” (IDI AO Chui Issyk-Ata ) District Level Department of Education (DDoE)  “We have a lot of plans to improve the students’ literacy. In summer, during the holidays, there are camps where they teach literacy, also there is a Jailoo development center where we teach children on different topics and work every summer, but we did not work during the pandemic. There is a program "Keremet Okuu", which includes 22 schools, and we are planning to add other schools. We also educate parents and improve their literacy. There is a school for parents where they are taught how to behave with children in different situations, how to bring up and how to motivate the child to study, because there are parents who did not study at all.” (IDI DDoE Issyk-Kul Ak-Suu)  “The fact that we have all the conditions makes our plans feasible. There are manual books, so we will be able to improve our knowledge.” (IDI DDoE Talas Bakay-Ata) The data suggests that given the lack of budget and the fact that most of the literacy and learning promotion work except for the positive discipline and reading corners are considered extracurricular, there is no certainty regarding the ability of these stakeholders to prioritize the literacy and learning promotion activities. Moreover, whereas the hot meal programs are supported and mandated by law, the literacy promotion activities such as positive discipline and related extracurricular activities do not have this institution and legal foundation for sustainability and continuation. The mission notes that MGD17 developed instructions related to the Positive Discipline and new reading methodologies were approved by the MoES and accepted/certified by its Department of Continuing Education for teachers. This is a promising sign of the integration of the approach beyond the end of the program.51 51 MC Interview, 12/2/2021. 77 (3) Have the program activities contributed towards improved information sharing between primary schools and local and national MoES officials? The evaluation data did not show MGD activities contributing to improved information sharing or communication between primary schools and local/national MoES officials. While school officials and staff, along with government officials, believe MGD programming complements the efforts of the relevant ministries., school administrators and staff, when asked, did not report experiencing improved communications with government officials. As the feedback from interview respondents below highlights, school administrator sentiments regarding the collaboration with local and national authorities were mixed with some feeling that the collaboration is strained and fruitless while others more so very positive. Interestingly there was more consistency relative to the quality of the collaboration from AO and DDoE staff: School Administration on Literacy  “Everything can be solved through negotiations. But our aiyl okmot is powerless, they will not solve. They don't even understand how to solve it. They think that school belongs to the director, and he must solve everything. I tell them that the school is on the balance of the aiyl okmotu, which means that they must solve all the problems. You receive taxes, the state leaves part of the taxes at your discretion. And to solve problems, we need a strong head of local government.” (KII Director, Naryn Jumgal)  “I can't evaluate. We did not have any problems during this period. We fulfilled what we promised 100% and we have support.” (KII Director, Jalal-Abad Aksy)  “We get books according to the age of students. The quality of education has improved.” (KII Director, Jalal-Abad Aksy)  “Both the district education department and the organization have the same direction. It promotes literacy. We had no problems on this issue.” (KII Director, Jalal-Abad Aksy)  “We closely work with local education authorities and communicate with them. We always invite them to our contests, and they also give us methodological support. We are in good relations with the Ministry of Education. We comply with all their instructions. They sometimes organize contests for us, and children participate and win prizes in the contests.” (KII Director, Issyk-Kul)  “Yes, there were changes. We cooperate with the Ministry of Education on the secondary classes from the fifth to the ninth grade. They learned how to assemble the video and needlework, they participated in contests, wrote essays, and won prizes.” (KII Director, Issyk-Kul)  “They have never prevented us from implementing programs and projects.” (KII Director, Chui Moskovsky) The findings illustrate that from the perspective of the school administration staff, there is no consistent transformation of the collaboration and information sharing between the tripartite of actors that is attributable to MGD17. The findings do not negate the efforts of MGD17 to improve information sharing between schools and local and regional education authorities through the program activities; joint round￾table discussions & workshops that were facilitated even during the COVID lockdown period using online platforms and bringing participants from all over the country. However, based on the evaluation data, there are opportunities to support these actors to formalize and standardize how they collaborate and share information across the tripartite. 78 (4) What key factors have contributed to the variation in target schools’ degree of feeding program success as measured through dietary diversity, continuous community support, and other aspects of program implementation?  “It is natural that there were differences in the degree of success of the nutrition program in schools, which depended on many factors, such as access to resources (local government, which are subsidized, could not help schools), remoteness from the district, province centers, differences in infrastructure (some schools lacked special premises for catering and canteens), management skills of school principals, support from parental committees, etc.” (Interview MC Staff, 11/24/21) The quote from the MGD 17 staff member above highlights the reality of the complexity of factors that explain variation in performance that we see at the school level. There are a multitude of factors that include the location, school staff engagement and leadership, local government/community engagement and finally, parental ability to subsidize the SFPs of their children’s schools. This quote from a DDoE representative highlights the key role of parents, local authorities, and communities in the success of SFP programs:  “Parents see and feel that hot meals help students to be healthy and to study well. That’s why they do not spare neither strength nor resources. The snack bar of a secondary school which was supposed to have its kitchen redesigned was recognized as unfit due to the small number of square meters by Sanitary and the epidemiological station in ayil okmotu. The parents supported instead and with the help of ayil okmotu, the premises was expanded and equipped for a kitchen. The deputies of local councils, heads of villages, and teachers were involved in this work. The deputy of the local council visited the opening ceremony of the kitchen. A lot of useful work was done with the support of local self-government, parents, and community.” (IDI DDoE, Osh Kara-Suu) This quote suggests that the program's success is contingent on how well this consortium of actors mobilize with the MGD17 team to implement and support the program rollout and continuation. Because of resource constraints and the importance of behavior change at the institutional level, all of these actors must collaborate in order to support SFP success. Another factor that was highlighted are the orchards and gardens, which provide extra/needed commodities and resources that can be used to support diet diversity for primary school children; a DDoE official reinforced this idea:  “There are schools that have opportunities. Sometimes they themselves come up with suggestions and they have opportunities, but no project. They wanted to organize hot meals using a subsidiary farm. Now, out of 18 schools, 13 have a subsidiary farm.” (IDI DDoE Naryn Ak-Taala) The quote also highlights the importance of the school administration's entrepreneurial spirit and desire to see the SFP programs succeed. These efforts, whether reaching out to other donors, parents, creating gardens or orchards or otherwise, increase the magnitude of success and likelihood of sustainability.52 52 Interestingly, one respondent answered this question with the following: “I have communicated with school directors from schools which are supplied by UN jointly with Russia. It turns that they (MGD Schools) are supplied better in terms of equipment, utensils, pots, shredding machine for onion and potato. But they (WFP schools) supply not like Mercy Corps, they have just supplied the flour. And Mercy Corps even during pandemic was distributing food commodities.” (KII Director, Jalal-Abad Suzak Kumush-Aziz) 79 (5) To what extent has the COVID-19 pandemic and associated disruptions in education, such as school closure, affected effectiveness and sustainability of school feeding programs established under the program? The MGD17 program was not able to meet its indicator targets for school feeding due to the COVID-19 pandemic and related school closures throughout the country. That said, significant adaptations were made to adjust the program and make the most impact possible despite the circumstances. While MGD17 was unable to reach the indicator for the number of hot meals per student servings, there were leftover food commodities in school warehouses. At the request of the Ministry of Education, these commodities were distributed to households in May 2020 by the Ministry of Education and school administrations. This was part of the adjustments that MGD17 made to meet the unanticipated needs presented by the pandemic. Many school employees and government officials reported that WASH and infrastructure improvement efforts predating the COVID-19 pandemic helped prepare students and communities to ward off the disease, occasionally crediting the interventions for keeping the illness out of their communities altogether. One government official in Issyk-Kul province stated:  “Schools have set up entry filters which were not used earlier. We learned to wear masks and stick to hygiene measures. Teachers arranged class hours on cleanliness and up-bringing, keeping social distance, avoiding groups of people. As a result, children mastered these topics. I think that through such activities we prevented COVID dissemination cases among children.” The arrival of the COVID-19 pandemic reinforced the importance of improved sanitation and hygiene, thereby likely having an effect of improving the legitimacy and buy-in of the intervention overall. Similarly, although students’ parents and families appreciated take-home rations, the end of hot meals at schools negatively impacted students’ health and well-being compared to what families were accustomed to before the pandemic. In addition, the economic situation in Kyrgyzstan as a result of the pandemic will no doubt make it more difficult for parents to contribute financially due to the growing cost of food products necessary for hot school meals. This was concurred and further explained by MGD17 staff:  “The food commodity prices increased due to COVID and this is a major challenge to program sustainability. They (MC)are meeting about this to see how to resolve this with the GoK and international community. The MoEs was to draft a document regarding the issue for the president's office. The parents' contribution has become a major issue and they are pushing this point forward." This economic reality detracts from any increased enthusiasm and related desire parents may have for financing school feeding after the support from Mercy Corps draws to a close, which has the potential to diminish program sustainability in general because even the government budget will not provide the quantity and quality of commodities as it did before the pandemic. This requires reflection and action from the GoK and MGD17 staff to consider the way forward to ensure the sustainability and continuation of the SFP program in the current economic climate. 80 Sustainability Criteria Conclusions Conclusion Rating Observations Fundamentally GoK and its partners USDA, MC, and the WFP have put in place the basic structures to support sustainability at the national level, with the passage of the National Law on School Feeding, along with the provision of the 7 or 10 KGS per student per day provide the legal and financial framework for school feeding to continue nationwide. However, the data suggests that in order for schools to provide and sustain a diverse, nutritional daily diet for each student, increased resources are necessary. Moreover, the legal and financial framework for the continuation of the literacy and learning promotion activities are not in place to ensure the continuity of these activities. Moderately Unsatisfactory ● Though the National Law on School Feeding bodes well for sustainability, the amount of money allocated under this law (7 to 10 KGS per day per student) is not considered sufficient to provide students with an acceptably diverse diet. Evidence to indicate the long-term sustainability of government support for improved literacy instruction practices is yet more sparse. The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. ● Many school staff expressed intent or desire to continue school feeding but also noted they had no or very limited capacity to do so. ● Regarding the responsibility for ensuring the sustainability of the hot meal programs at MGD17 schools there seems to be a tension between whether the responsibility falls to the parents or the government. ● The data suggests that given the lack of budget and the fact that most of the literacy promotion work with the exception of the positive discipline are considered extracurricular, there is no certainty regarding the ability of these stakeholders to prioritize the literacy and learning promotion activities. ● The findings illustrate that from the perspective of the school administration staff there is no consistent transformation of the collaboration and information sharing between the tripartite of actors that is attributable to MGD17. ● Because of resource constraints and the importance of behavior change at the institutional level all actors including the school administration, local authorities and communities and parents must collaborate in order to support SFP success. ● This pandemic and the economic crisis resulting is increasing the cost of commodities and diminishing the capacity of parent to subsidize school feeding after the support from Mercy Corps draws to a close, which has the potential to diminish program sustainability in general because 81 even the government budget will not provide as much commodities as it did before the pandemic. Sustainability Recommendations Recommendations Timeline and Prioritization Duty Bearer 1. MC, Gok, and USDA need to develop an exit and handover strategy for each graduated school that is specific to their needs, context, and circumstances. End of 2022 MC, USDA, & GoK 2. The exit strategy development should also explore the formalization of the communication and collaboration between schools, AO, and DDoE following the departure of MC. End of 2022 MC, GoK, and USDA 3. GoK, MC and USDA should consider working together to develop school-level sustainability strategies that extend beyond parent contribution and subsidies. June 2022 MC, GoK, and USDA 4. The literacy promotion components, particularly positive discipline, require advocacy to ensure a normative or policy adoption that will promote its sustainability and integration beyond the closeout of MGD17. End of 2022 MC, GoK, and USDA 5. MC, USDA, and GoK should explore systematic support for the introduction of school orchards/gardens in schools that do not already have them. June 2022 MC, GoK, and USDA Sustainability: Lessons Learned 1. The reliance on parental contributions and subsidies as a sustainability strategy for SFP in MGD17 schools in some program communities is simply not viable, and given the economic strife caused by the pandemic and rising commodity prices, these risks putting the sustainability of the SFP at all graduated schools in jeopardy. 2. Intention does not equal ability due to the financial implications of sustaining the SFP programs and the differing circumstances impacting each school and community; multiple and differing strategies are required to sustain SFP programs. 5. Conclusions MGD appears to have been quite successful overall, as a development program. While inconclusive by itself, quantitative data from literacy tests and output indicators suggest that, even with COVID-19 interfering with school schedules and access to the most essential parts of MGD programming, student trajectories were nevertheless positive. Qualitative data, meanwhile, collectively illustrates a diverse team of vested personnel dedicated to student performance, working together across diverse sectors. This is perhaps MGD’s most notable achievement—effectively transitioning over the years from a relatively narrow humanitarian aid program to an interwoven set of complementary, wrap-around services that create a positive learning environment for young students. 82 The list of KIIs and FGDs for this evaluation—including national government officials, school administrators, cooks, teachers, librarians, and parents—is itself emblematic of the diverse array of stakeholders who have been enrolled and are now coordinated around one project with the unified purpose of serving young students and creating the conditions for them to prosper. Qualitative data stemming from this same diverse array of respondents overwhelmingly suggests commitment to MGD’s desired outcomes. The uptake and implementation of training into real-world classroom and kitchen settings, the creation of novel adaptations to minimize the disruption caused by COVID-19, and the outreach and transparency to include the community and parents, in particular highlight the strength of this buy-in. This is itself a significant and notable set of achievements. The consistency with which such a broad array of respondents demonstrated their own engagement, enthusiasm, and commitment is notable and impressive. As the previous sections have noted, there were some complications. Most obviously with the coherence of the program alongside other similarly designed and overlapping programs, including other US￾supported literacy programs. MGD must find a way to feasibly coordinate with other donors, particularly USAID, however challenging that may be bureaucratically, to truly maximize their effectiveness and impact. While it was not called out as a specific section, OECD-DAC has introduced “coherence” and the coordination and sensible overlap among donor-supported programming in a given context as a core evaluation criterion among those examined here, underscoring its significance. Relevance Conclusion Rating Observations The data emerging from the final evaluation reinforce the midterm evaluation findings, which resoundingly highlight the relevance and importance of FFE17. In short, the program is appropriately designed to meet its core objectives and is positively viewed by stakeholders at all levels. Satisfactory ● The school feeding component of McGovern￾Dole 2017 remains highly relevant and in line with the GoK policies and strategic aims. ● The Kyrgyz School Feeding Policy was passed in December 2019. ● The project emphasizes teaching teachers positive discipline techniques and increasing access to reading materials and literacy. SBC activities are highly appreciated elements of the MGD design. ● As stated during the midline, the fact that the program does not teach improved literacy techniques to teachers and that TTR is working in the same schools on literacy, definitive attribution for the learning outcomes is impossible unless there is a comparative quasi￾experimental design employed. ● The program design has been adapted and modified from the outset to ensure its appropriateness and relevance to institutional and individual participants. Effectiveness Conclusion Rating Observations 83 The MGD17 program is a complex multi-sectoral program; the data shows highly effective performance across all of the outputs related to the SFP/enabling learning environment component and the second Literacy and learning promotion components. Despite the school closures due to the pandemic, MGD17 was able to meet or surpass all of its pre-established objectives except those linked to meals served to students. Overall, the program took a comprehensive approach to SFP that addressed commodities and training and sought successfully to create the environment required to provide students with nutritious, hygienic, and healthy hot meals five days a week. Highly Satisfactory ● 108% percent of target schools have received all the requisite school infrastructure and equipment inputs for kitchen rehabilitation. ● As it relates to the improved water sources, over 100% of the targets have been reached. ● Most (75% or above kitchen equipment and materials were functioning at time of field visit. ● In terms of training the school staff for SFP, including safe food preparation, management of commodities, and water systems, the McGovern￾Dole Monitoring data highlights that all targets have been surpassed substantially relative to these variables. ● All winterization and WASH infrastructure targets were exceeded. ● Mercy Corps received a modification from USDA that allowed them to engage firms directly to build the latrines based on international standards latrine construction. ● MGD17 also made distributions of household rations to the families of participating primary school children. ● MGD17 also surpassed all targets linked to literacy and learning promotion, including extra￾curricular activities, positive discipline training for teachers and the creation of reading corners. Impact Conclusion Rating Observations 84 The data highlight an 8- percentage point increase in the number of reads with comprehension, while the non-readers remain at 5%. While this is a single-digit improvement in the reading comprehension results from the baseline, these results are promising in the backdrop of the ongoing pandemic and school closures. They suggest that the efforts of the GoK and MGD17 are reversing the years of malnutrition that had previously undermined the performance of school-age children in Kyrgyzstan. Satisfactory ● Evaluation, monitoring, and independent household study evidence suggest that there has been significant progress in improving health, nutrition, and dietary practices at school and in the home. ● The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. ● MGD17 monitoring for 2020-2021 shows that 87% of program schools have a diverse school menu (i.e., the same meal is not repeated more than twice per month, and all food groups are included). ● MGD17 monitoring for 2020-2021 shows that 78% of program schools that have school feeding (complete meals) five days per week ● MGD17 monitoring for 2020-2021 shows that 57% of program schools have regular (monthly) financial contributions from parents. ● The program performance and impact suggest that while the pandemic presented challenges to program implementation, it did not block the MGD17 from achieving its program outcomes related to SFP, literacy promotion, and learning outcomes for primary school children at participating schools. Efficiency Conclusion Rating Observations 85 McGovern-Dole 2017 performed well across all the dimensions of efficiency. The evaluation data shows solid financial and programmatic management performance. The MGD17 team and USDA applied its extraordinarily adaptive and innovative leadership style to ensure the program was managed for results and address emerging needs and crises like the pandemic. Highly Satisfactory ● The program made multiple programmatic and management adjustments to ensure program success and efficiency, including everything from early revision to the program design, adjusting the infrastructure approach repeatedly, and adjusting program components to respond to the ongoing pandemic. ● MGD17 leadership and USDA maintained program implementation during the pandemic and added household rations to primary school students' families, online training, and activities for the SFC, and literacy promotion components. The program also amped up the WASH component to encourage the school's reopening and fight the spread of COVID-19. ● The program performed well in terms of financial and personnel management efficiency. ● Most stakeholders positively view the program oversight and management processes. ● Lastly, most if not all feedback emerging from the midterm evaluation has been addressed or completed despite delays caused by the pandemic. The midterm recommendations result from the combined feedback and recommendations from stakeholders across the program. Sustainability Conclusion Rating Observations Fundamentally GoK and its partners USDA, MC, and the WFP have put in place the basic structures to support sustainability at the national level, with the passage of the National Law on School Feeding, along with the provision of the 7 or 10 KGS per student per day provide the legal and financial framework for school feeding to continue nationwide. However, the data suggests that in order for schools to provide and sustain a diverse, nutritional daily diet for each student, increased resources are necessary. Moreover, the legal and financial framework for the continuation of the literacy and learning promotion activities are not in place to ensure the continuity of these activities. Moderately Unsatisfactory ● Though the National Law on School Feeding bodes well for sustainability, the amount of money allocated under this law (7 to 10 KGS per day per student) is not considered sufficient to provide students with an acceptably diverse diet. Evidence to indicate the long-term sustainability of government support for improved literacy instruction practices is yet more sparse. The 2019 and 2020 by law illustrate that the GoK continues to support school feeding and nutrition. However, there is little concrete evidence to show increased government support for literacy activities. ● Many school staff expressed intent or desire to continue school feeding but noted they had no or minimal capacity to do so. ● Regarding the responsibility for ensuring the sustainability of the hot meal programs at MGD17 schools there seems to be a tension between whether the responsibility falls to the parents or the government. ● The data suggests that given the lack of budget and the fact that most of the literacy promotion work with the exception of the positive discipline are considered extra-curricular, there 86 is no certainty regarding the ability of these stakeholders to prioritize the literacy and learning promotion activities. ● The findings illustrate that from the perspective of the school administration staff, there is no consistent transformation of the collaboration and information sharing between the tripartite actors that is attributable to MGD17. ● Because of resource constraints and the importance of behavior change at the institutional level, all actors, including the school administration, local authorities and communities, and parents, must collaborate to support SFP success. ● This pandemic and the economic crisis resulting in increasing the cost of commodities and diminishing the capacity of parents to subsidize school feeding after the support from Mercy Corps draws to a close, which has the potential to diminish program sustainability in general because even the government budget will not provide as many commodities as it did before the pandemic. 6. Recommendations In this section, Causal Design has utilized a Findings, Conclusions and Recommendations (FCR) matrix, which simply organizes validated findings into a coherent table. Themes are filled in first, followed by findings. This step provides an additional robustness check, as findings are tabulated (for frequency, or presence across multiple sources) and detailed for inclusion into the matrix. With findings noted, the evaluation team transitions to drawing conclusions, or addressing the “so what?” of the myriad findings and complex details. Staggering this process ensures that conclusions are not hurried assumptions, but rather clearly flow from the totality of evidence. Moreover, the process through which recommendations are arrived at is plainly visible for readers and invested stakeholders. Table 12. USDA Kyrgyz Republic MGD Findings, Conclusions, and Recommendations (FCR) Matrix Findings Conclusion Recommendation Relevance Hot meals were cited by a majority of respondents, across roles (librarians, cooks, administrators, etc.) as a A hot meal, in this context, is a driver of attendance, provides needed access to nutritious foods, and assists in creating a positive learning environment. Continue to work with schools on mainstreaming and prioritizing hot meal preparation. Though, notably, this is already being accomplished. 87 pivotal component, for families and educators. This is magnified, qualitative data suggest, in rural, poorer districts. Books, particularly accessible, age-appropriate books, were also cited as critical to engagement and fostering a conducive learning environment. Kyrgyzstan￾specific books (regarding history and poems) appear to be in short supply. Accessible, age-appropriate books are a scarce resource in this context, but are essential to increasing exposure and student engagement, with likely positive literacy promotion effects. Continue to work with librarians to ensure access to books and consider options for how to swap/rotate books between proximate schools. Work with local suppliers to source and distribute more Kyrgyzstan-relevant books (folklore, poems, and history).* While most respondents suggested that students adjusted over time, unfamiliar varieties of foods, especially lentils, caused some frustration among cooks, owing to waste, in particular, if students did not eat them. Most respondents were bullish on access to nutritious, diverse, fortified foods, but unfamiliar varieties, and inaccessible foods (at local markets, for example) may limit their impact. Work with cooks to derive menus and meal options that mix local and imported foods and are potentially replicable at home. Effectiveness School cooks have gained social status and new employment opportunities through MGD trainings. Because school cooks have new opportunities, while salaries remain low and working conditions at schools remain challenging, cook turnover could disrupt school feeding. Mercy Corps might advocate for additional financial and workplace support for school cooks, through advocating that governments allocate funds to school cooks’ salaries under the School Feeding Law, or through parents’ contributions. Teachers and librarians frequently praised the advent of Positive Discipline as a teaching strategy in their classrooms. Also, the PD module was approved by the The Positive Discipline methodology is novel and appreciated in the Kyrgyzstan educational context and is very likely an essential part of the stronger relationships that parents, teachers, and Consider continuing Positive Discipline training for teachers, as it is a welcome component of the improved literacy package, and also consider lobbying for its inclusion in the teacher training institute(s) to 88 Kyrgyz Academy of Education for use in schools. librarians identified between educators and students. reach more students outside of intervention schools. Some staff at schools that received MGD infrastructure support reported that canteens and/or toilets were nonetheless insufficient for the student population, which often grew significantly during the intervention. MGD and hot meals in particular not only increase attendance, but may also, when combined with the current youth bubble, spur increased school enrollment numbers overall. Ensure MGD fund allocation and infrastructure plans take into account the likelihood of increased student numbers as a result of MGD activities acting as a pull factor. Many school staff and other actors interviewed spoke about significant unmet needs at typically more remote schools lacking running water or other basic requirements necessary for hygiene and the provision of hot meals. Despite the strong achievements of MGD in schools, where running water already existed or could relatively easily be established, unmet needs will persist and exacerbate inequities in Kyrgyzstan until all schools have running water and hot meals. Consider allocating additional funding to infrastructure interventions, including water access, as these are a critical need among many schools across Kyrgyzstan. The need to participate in trainings and additional extra￾curricular activities can pose an additional burden. This is a particular risk for teachers, who receive other trainings from other implementing partners in Kyrgyzstan and women, who make up the majority of primary school teachers, administrators, and cooks, and face a double burden with both work and home duties. Involvement in too many trainings and activities, in addition to other home duties, may cause frustration and exhaustion among participants. Ensure all implementing partners conducting training activities as well as extra-curricular activities collaborate to ensure that activities do not overlap or cause undue burdens. Schedules and formats for trainings may need to be made on a case-by-case basis and should be gender-sensitive. Training fatigue and frustration can be enhanced by difficulties accessing online Technical difficulties when participating in online Consider additional trainings on digital literacy and other provisions to overcome technical difficulties, 89 trainings due to limited digital literacy and internet access. These difficulties may be greater for older recipients. trainings may reduce the effectiveness of trainings. such as assistance accessing the internet. These trainings may also be necessary to ensure that the format of online training does not cause an additional undue burden. Anecdotal evidence suggests that there can be some inconsistencies in the advice given to school cooks by different inspecting stakeholders. School cooks may be confused by the competing advice given and best practices may not be followed. Clarify and ensure that all inspectors, from all government and non-government organizations, give advice based on a single set of guidance and regulations to avoid inconsistencies. Refer to a single gold-standard set. Efficiency Very few school-level respondents reported contacting Mercy Corps with any feedback or suggestions for how to improve programming. School staff may not feel that it is warranted or desirable for them to give feedback to Mercy Corps about the programs it is implementing in their schools and communities, thereby restricting “buy-in” and local ownership, while also potentially limiting the program’s prospects for success and sustainability. Consider establishing a clear formal feedback and communication mechanism to solicit and manage stakeholder feedback that may not be sufficiently addressed by field monitors, while carefully managing expectations regarding request fulfillment and what is beyond Mercy Corps’ scope/capacity to change. Impact Despite on-the-ground challenges, most respondents depicted a committed group of diverse actors who have embraced trainings and learnings, and who are actively deploying them to enhance the learning environment for their students. Students are benefitting from the multi-faceted nature of MGD programming and have demonstrated greater engagement and interest. Future programming should continue to seek to learn lessons and leverage the solid foundation of local support and engagement developed through MGD programming. 90 While difficult to interpret in isolation, quantitative indicators suggest that students are—even with COVID-19 undoubtedly interrupting progress— improving their reading skills. The qualitative feedback suggesting greater engagement by parents, students, teachers, librarians, and cooks appears to be having a positive effect on learning outcomes. Quantitative indicators, including improvements in listening and reading comprehension at endline over baseline (see Annex A) most notably, reinforce the qualitative results. Future programming should seek to continue the support initiated through MGD programming to further build upon the currently achieved results. MDG influenced behavior at home; enhanced food storage, safety, and preparation in schools; and positively influenced government support. MDG programming has the design and capacity to positively affect reading and nutrition among participants in the immediate term, mostly, but sustaining these gains remains the crux. Future programming should seek to continue the support initiated through MGD programming to further build upon the currently achieved results. Sustainability Government funding of 7-10 KGS per child per day is not enough to maintain nutritious school meals. Parent contributions and ability to contribute vary widely, and not all parents are able to contribute enough to enable schools to continue providing nutritional school meals. Rising food prices exacerbate the Without steady and predictable parental contributions, in the absence of significantly increased governmental support, hot school meals will not be sustainable beyond the term of the Mercy Corps intervention. Consider working with the MoES, Parliament, and other relevant government entities to formalize a parental contribution system in order to secure sustainable school feeding nationwide. In addition, donors and international organizations should advocate the government to allocate more funding towards school meals, if necessary targeted in areas where 91 need for funding for school meals. parents are unable to supplement school meals budgets. Book distribution and improved capabilities of teachers and librarians to promote reading should have long-term impact in terms of improved literacy instruction. Absent additional sustained funding from the government, and/or parents, nutritious and diverse school meals in many cases will likely not be maintained when Mercy Corps’ food distribution stops. While this is critical to the learning environment, teaching techniques and access to age-appropriate books should sustain recent advances in literacy. Continue to work with librarians to ensure access to books and consider options for how to swap/rotate books between proximate schools. Work with local suppliers to source and distribute more Kyrgyzstan-relevant books (folklore, poems, and history). * Prior to implementation, any donor coordination of literacy improvement programming in Kyrgyzstan did not prevent programmatic overlap. When implementers encountered programmatic overlap, the donors were not made aware. Given that the donor representatives were unaware of the overlap among literacy programming, there is a strong possibility that the overlap can happen again and limit the coherence of MGD efforts. Donor should better coordinate literacy promotion efforts in Kyrgyzstan with existing and future literacy promotion and early childhood education interventions to maximize coherence. ** Relevant Recommendations Relevant Recommendations Timeline and Prioritization Duty Bearer 1. GoK, MC, and USDA should meet to explore and consider the best approach to the literacy component in the future, particularly exploring whether traditional literacy training and positive discipline will be combined or whether the positive discipline approach will be abandoned. Given the recent literacy study results, there may be evidence that a combined approach is ideal. ASAP GoK, MC and USDA 92 2. USDA, GoK, WFP, and MC should hold a symposium to explore the issue of sustainability of school feeding programs to fully explore what design components are required in order to promote the sustainability and independence of the GoKs SFP programs in graduated schools, especially in light of the current economic issues in the country. ASAP GoK, WFP, MC, and USDA 3. USDA, GoK, and MC should develop a white paper regarding lessons learned from the flexible and adaptable approach to implementation employed during the Implementation of MGD17. ASAP GoK, MC, and USDA Effectiveness Recommendation Timeline and Prioritization Duty Bearer 1. Current MGD17 data highlights that the indicator for meals served was not achieved; however, it is not clear if the household level rations for the families of participating primary school children were considered in this calculation. If not, some consideration for these distributions and the additional activities undertaken should be reflected in the MC PMP and performance assessments to demonstrate the quality of the effectiveness and management employed by the MGD17 management and oversight efforts. ASAP MC & USDA 2. If not already done, the program should share with the GoK the infrastructure and needs assessment framework to integrate this into their planning and prioritization standard operating practices, should they so choose. End of 2022 MC, USDA & GoK 3. The MGD17 and USDA should develop guidance on delivering effective online training, SBC activities for child and adult learners participating in the development and/or quasi￾humanitarian programming. End of 2020 MC, USDA and GoK Impact Recommendations Timeline and Prioritization Duty Bearer 93 1. To fully explore the impact of the MGD17 program, a quasi￾experimental design is required that compares the performance of MGD 17 schools with WFP and schools that do not have hot meal programs. MC should work with GoK to explore the possibility of conducting this type of study. There should also be control for TTR and related literacy programming to ensure that the true value-add of the MGD17 holistic approach is captured. March 2022 MC, USDA & GoK 2. An internal study of MGD12 & MGD17 graduated schools to explore the continued impact and sustainability would also be beneficial for exploring lessons learned and best practices for ongoing and future programming. There should be some attempt to explore the impact of the ongoing pandemic and how to support the GoK and local communities to find solutions. End of 2022 MC, GoK, and USDA Efficiency Recommendations Timeline and Prioritization Duty Bearer Given MGD17's and USDA's experience and success with program continuation and remote/virtual activities during a pandemic, MC and USDA should develop guidance documents and best practice materials that can serve as guidance and training materials for related programming and staffing to prepare for future shutdowns and similar situations. End of 2022 MC, USDA, and GoK Sustainability Recommendations Timeline and Prioritization Duty Bearer 1. MC, Gok, and USDA need to develop an exit and handover strategy for each graduated school that is specific to their needs, context, and circumstances. End of 2022 MC, USDA, & GoK 2. The exit strategy development should also explore the formalization of the communication and collaboration between schools, AO, and DDoE following the departure of MC. End of 2022 MC, GoK, and USDA 3. GoK, MC and USDA should consider working together to develop school-level sustainability strategies that extend beyond parent contribution and subsidies. June 2022 MC, GoK, and USDA 94 4. The literacy promotion components, particularly positive discipline, require advocacy to ensure a normative or policy adoption that will promote its sustainability and integration beyond the closeout of MGD17. End of 2022 MC, GoK, and USDA 5. MC, USDA, and GoK should explore systematic support for the introduction of school orchards/gardens in schools that do not already have them. June 2022 MC, GoK, and USDA 95 Annexes Annex A: Indicators Table 13. MC IPTT Indicator Results Indicator # Description Indicator Definition Target Result MGD SO 1: Improved Literacy of School-Aged Children 26 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Proportion of learners who attain the specified threshold at the end of two grades of primary schooling. Unknown Baseline: 39% Endline: 47% Program Result: Improved Quality of Literacy Instruction (MGD 1.1) Interim Result: Improved Literacy Instruction Materials (MGD 1.1.2) 2 Number of textbooks and other teaching and learning materials provided as a result of USDA assistance Number of teaching and learning materials provided as a result of USDA assistance, including materials designed specifically for this activity and standard materials available for purchase. Materials may include textbooks, workbooks, supplementary reading materials, etc. 34,699 Total: 100,823 Interim Result: Increased Skills and Knowledge of Teachers (MGD 1.1.4) 5 Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance Number of primary school teachers/educators/teaching assistants in target schools who demonstrate consistent use of improved techniques and tools in their classrooms, particularly with regards to literacy instruction. 645 Total: 1,933 6 Number of teachers/educators/teaching assistants trained or certified as a result of USDA assistance Number of primary school teachers/educators/teaching assistants in target schools who attend USDA-supported training 1,290 Total: 2,045, Female: 1,971 Male: 74 96 events and workshops on improved teaching techniques and tools, particularly with regard to literacy instruction among primary school students. Interim Result: Increased Knowledge of School Administrators (MGD 1.1.5) 4 Number of school administrators and officials trained or certified as a result of USDA assistance Participants: school director and/or deputy director, accountant, Parent Committee and/or Board of Trustees members, school supply/maintenance manager Topics include school feeding program management skills, budget development and accountability, food commodities storage rules and maintenance, roles and responsibilities of PCs and Boards of Trustees, MC accountability policies, data collection requirements, and other areas. 1,021 Total:1,163 Male: 895 Female: 268 3 Number of school administrators and officials in target schools who demonstrate use of new techniques or tools as a result of USDA assistance Number of school administrators and officials who are applying new knowledge and skills obtained at USDA-supported training events and workshops on a regular basis. 714 701 Program Result: Improved Attentiveness (MGD 1.2) Interim Results: Reduced Short-Term Hunger (MGD 1.2.1) and Increased Access to School Feeding (MGD 1.2.1.1) 15 Number of daily school meals (breakfast, snack, lunch) provided as a result of USDA assistance -A school meal may include a breakfast or lunch meal, or a snack provided in the mornings or afternoon during the school period. 37,300,000 Total: 27,228,136 97 -A school meal is counted each time it is provided to a student in a USDA-supported project. -School meals can be prepared in schools. 16 Number of school-age children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance Number of primary school students (grades 1-4) receiving daily school meals. 88,730 Total: 136,584 17 Number of social assistance Participants participating in productivity safety nets an as result of USDA assistance Number of primary school children (grades 1-4) regularly receiving hot meals at USDA￾supported schools. Same as above Same as above 27 Number of individuals benefiting directly from USDA – funded interventions Direct participants include primary school students receiving daily school meals, adults trained throughout the program (parents, teachers, directors, cooks, etc.) 93,906 Total direct participants: 144,007 28 Number of individuals benefiting indirectly from USDA-funded interventions Indirect participants may include parents of direct participants (students grades 1-4) who do not participate in USDA-supported trainings. 133,000 Total indirect participants (parents): 145,800 Program Result: Improved Student Attendance (MGD 1.3) Interim Result: Improved School Infrastructure (MGD 1.3.3) 7a Number of education facilities rehabilitated/constructed as a result of USDA assistance (kitchen infrastructure and equipment) Number of school kitchens rehabilitated through a USDA￾funded project. Rehabilitation may include installation of new windows and doors, whitewashing walls, installation of hand washing facilities and water heaters, and installation of 329 Total: 356 138 (2018) 218 (2019) 98 cooking and storage equipment (stoves, fridges, cupboards, etc.). 7b Number of education facilities (school buildings, classrooms, latrines, etc.) rehabilitated/constructed as a result of USDA assistance (school infrastructure) Number of school buildings that received reconstruction or rehabilitation support through USDA-funded projects. This support may include installation of heating systems, roof repair, installation of new windows and doors in classrooms, purchase of new classroom furniture, construction or repairs of hand washing and sanitation facilities, etc. 50 Total facilities (schools that received grants): 111 Interim Result: Increased Student Enrollment (MGD 1.3.4) 1 Number of students regularly (80%) attending USDA￾supported classrooms/schools This indicator measures the number of male and female primary school students attending school regularly (80% of classes or more, grades 1-4). The indicator goes beyond a one￾time measure of attendance collected at a single point in time during the school year and attempts to measure consistent school attendance during a given school year. Number: Children Disaggregation: Gender, Age Group 79,857 87,480 8 Number of students enrolled in schools receiving USDA assistance Number of primary school students (grades 1-4) formally enrolled in USDA-supported schools who directly benefit from USDA assistance (i.e. receive regular school meals). 88,730 Total: 136,584 MGD SO 2: Increased Use of Health and Dietary Practices 99 Program Result: Improved Knowledge of Health and Hygiene Practices (MGD 2.1) and Increased Knowledge of Nutrition (MGD 2.3) 18 Number of individuals trained in child health and nutrition as a result of USDA assistance Number of parents at target schools who receive training and informational materials on child health and nutrition as a result of USDA assistance. Topics for training will include rational nutrition, dietary diversity, cooking master class (for parents), food safety, nutrition during first 1,000 days, WASH, etc. 121 79853 19 Number of individuals who demonstrate use of new child health and nutrition practices as a result of USDA assistance Number of trained parents who regularly use new knowledge and techniques on child nutrition and health. 84 486 Program Result: Increased Knowledge of Safe Food Prep and Storage Practices (MGD 2.2) 20 Number of individuals trained in safe food preparation and storage as a result of USDA assistance Number of cooks from target schools who attend professional training on food preparation and safe storage practices and receive professional certification upon completion. 329 Total: 493 Female: 482 Male: 11 21 Number of individuals who demonstrate use of new safe food preparation and storage practices as a result of USDA assistance Number of trained and certified cooks from schools who use learned food preparation and storage techniques regularly. 230 409 Program Result: Increased Access to Clean Water and Sanitation Services (MGD 2.4) 22 Number of schools using an improved water source Number of target schools that have an improved water source, whether already existing or installed/improved through USDA-supported projects. 413 440 53 MC Comment: Under the SBC component, activity under this indicator was expanded to include community￾based nurses and health care workers as well. 100 23 Number of schools with improved sanitary facilities Number of target schools that have improved sanitary facilities (toilets), whether already existing or installed/improved through USDA-supported projects. 317 35654 Foundational Results: Increased Capacity of Government Institutions, Increased Government Support, Improved Policy and Regulatory Framework 12 Number of educational policies, regulations, or administrative procedures in each of the following stages of development as a result of USDA assistance (stage 1) MC, MoES and MoH will review and discuss the current stage of development of the national school feeding program in order to identify which evidence and lessons learned gathered by MC might be useful for this process. Pre-identified focus areas include human capacity development for managing school feeding programs, creating menus and supporting dietary diversity, engaging parents and wider community in supporting and managing school feeding programs, etc. Key partners: MoES and MoH 3 3 12 Number of educational policies, regulations, or administrative procedures in each of the following stages of development as a result of USDA assistance (stage 2) The second stage includes public debate and/or consultation with stakeholders on the proposed new or revised 2 2 9 Number of Parent Teacher Associations or similar “school” governance structures contributing to A Parent Committee is a school governance body that consists of parents who meet at least 4 times during the school year to participate in and contribute to 413 440 54MC Comment: TMC allocated more sanitation improvement grants to improve the sanitation facilities based on needs during the pandemic. 101 their schools as a result of USDA assistance the learning process and other school activities, including management of school feeding programs and other related activities (infrastructure repairs, purchase of furniture, etc.). This support may include direct financial (cash) or in-kind (produce) contributions. Interim Result: Increased Engagement of Local Organizations and Community Groups (MGD 1.4.4) 10 Number of public-private partnerships formed as a result of USDA assistance Number of public-private partnerships formed at target schools due to USDA assistance, i.e., number of target schools with regular parental or community contributions to support school feeding programs (cash or in-kind). This partnership may be formalized through a written agreement between the school's administration and PC or Board of Trustees. 453 559 11 Value of new public and private sector investments leveraged as a result of USDA assistance Value of cash and in-kind contributions by private entities (parents and other community members) to target schools in order to support school feeding program and other related activities. 479,000 USD 1,603,391 USD 102 Annex B: Data Collection Instruments The following instruments were intended to guide the semi-structured interviews that were conducted as part of the data collection for this evaluation. As a result, the questions asked in the actual interviews varied somewhat from the questions in the tools below. In the case of interviews that CD conducted directly (with national-level government officials, MC program staff, and USDA, USAID, WFP and Save the Children staff), the instruments were modified slightly before the interviews in order to tailor the planned questions to match the experiences and roles of the individuals interviewed. In all cases, the interviews were designed to be conversational, with optional probes and follow-up questions used whenever appropriate. School Teacher / School Librarian Tool Hello, my name is ________________. I work with Erfolg Consulting in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools in Kyrgyzstan by providing over 2,000 metric tons of food to them. This program has also promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community-based reading activities, plus making infrastructure improvements for schools and promoting positive social behavior change through a network of trained community-based change agents. Goal of this Interview: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this interview because we would like to learn more about how we can sustain the outcomes achieved thus far Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Background 1. Can I start by asking your name and position at the school? 2. How long have you worked at this school? 3. Have you been involved in trainings or workshops associated with the program in the past two years? a. [If yes:] What were the topics covered in the trainings you took part in? Relevance 103 4. When you reflect on all of the activities, projects and goals, what were program elements, from your perspective, that were most helpful to you, other teachers, and especially students? 5. If you were in charge of re-designing this program, what elements would you change? a. Why would you make these changes? 6. Were you involved in helping change, or adapt, activities to make them work better for your students and community? If so, what did that process look like? (Ex: Who did you work with? What did you change? Were there options for you to share your point of view on what you thought worked or needed to change?) Effectiveness 7. Based on your experience at this school, and with the goal of maximizing the literacy and health of students, would you have allocated Food for Education resources differently or the same? a. [If differently, ask:] How would you have used the resources differently? i. Could you please provide specific examples (probe: online resources, in-kind resources to families, improvements to infrastructure, transportation, parent awareness, etc.) ? 8. What, if anything, have you learned in terms of professional development through this program? a. [For teachers:] Mercy Corps has begun an initiative designed to encourage teachers to use Positive Discipline in the classroom. How useful, if at all, was this initiative for you? i. How easy or difficult was it to use Positive Discipline in your teaching? ii. Did you get the support that you needed for this shift? iii. What, if any, results did you see? iv. Which, if any, different topic might you have preferred to learn about for your classroom? b. [For librarians:] How has your role changed in the school, if at all, through this program? i. To what extent would you say that you have received enough support from Mercy Corps? 1. To what extent would you say that you received enough support from your local school(s)? Local government? 9. Was there anything you observed, as a result of the Food for Education program, that you did not expect -- affecting either school staff, parents, or students? a. [If yes:] Can you please explain? 10. In light of Covid-19 and related school closures, which tools or activities did you pursue that: a. Had the best student engagement? b. Students could access the most easily? c. Fostered literacy and learning the best, even if done remotely? d. Had the most parental engagement or support? 11. Thinking back to your response to the Covid-19 pandemic and these changes you made responding to school closures and social distancing requirements, what role, if any, did Mercy Corps play in supporting you during this time? Efficiency 12. I know that some schools also received support from other literacy programs, like RTI/Okuu Keremet or Chemonics/Time to Read. If these programs have been active in your school or a library, which worked better? a. Why do you say that? b. Which program was more difficult to work with? c. Why? 13. What, if any, changes have you seen in how the Mercy Corps Food for Education program was run during the past few years? Please be specific. (Probe: number and quality of trainings, observable infrastructure improvements, children affected, effective administration/communication, etc. 14. Have you ever provided feedback to administrators, Mercy Corps, or any other related officials regarding the program? a. [If not:] If you had feedback, would you feel comfortable providing it? Impact 104 15. Can you comment on any changes you have observed (in the last 2 years) in the general health, nutrition, and dietary practices at school? a. During this time, has there been any additional support from local or national government for school feeding and/or literacy? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? b. Also during this time, have you observed any changes in the nutrition behaviors of families of students at home? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? 16. Can you comment on any changes you have observed (in the last 2 years) in terms of literacy promotion practices, both at school and at home? a. What specifically have you done differently, as a teacher or librarian, to augment student learning/literacy, over the last 12 months? (Probe: new technologies, WhatsApp groups, e-books, parent groups, reading buddies, etc.) Sustainability 17. When this program ends, what plans or resources are in place to continue improved literacy instruction practices for primary school students? (probe for specifics re: access to resources, online or otherwise parent groups, etc.) a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? Covid 18. Describe the direct effects of Covid-19 on your students with respect to the health and literacy goals of the program. a. How did families and children respond to school closures and new ways of teaching? 19. What changes did you observe to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. b. To what degree were students/parents able to capitalize on new technologies? (like WhatsApp, audio or e-books, etc.) i. What affected a student’s learning under these new conditions, most? (Please be specific – limited internet, electricity, devices, learning curve with new platforms, for teachers or students, etc.) c. We are particularly interested in the online reading camps over summer 2020. Were you involved with such a camp? i. [If yes:] How easy or difficult was it to plan the camp(s)? 1. What, if anything, worked not so well with this online reading camp? 2. What, if anything, worked well? 20. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any questions about the program, concerns, or other types of feedback, please do not hesitate to call CARM at (0700-889-971). School Cook KII Tool Hello, my name is ________________. I work with Erfolg Consulting in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) 105 McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools in Kyrgyzstan by providing over 2,000 metric tons of food to them. This program has also promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community-based reading activities, plus making infrastructure improvements for schools and promoting positive social behavior change through a network of trained community-based change agents. Goal of this interview: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this interview because we would like to learn more about how we can sustain the outcomes achieved thus far. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Background 1. Can I start by asking your name and position at the school? 2. How long have you worked at this school, and were you involved in trainings or workshops associated with the program? a. [If yes:] What were the topics of the trainings or workshops? Relevance 3. What were design elements of the Food for Education school feeding program, from your perspective, that helped you the most to provide nutritious meals to students? a. What could have been improved? 4. If you were in charge of re-designing this program, what elements would you change? a. Why would you make those changes? Effectiveness 5. How effective was the WhatsApp group for Cooks? a. Did it help to have your former trainers moderating the groups, for example? 6. How do you feel you grew in terms of your professional development through the school feeding program, if at all? a. What skills did you learn through this program, if any? b. How do you feel you contributed to this program, if at all? 106 c. What challenges did you have, If any? 7. Based on your experience at this school, and with the goal of maximizing the health of students, would you have allocated Food for Education resources differently or the same? a. [If differently, ask:] How would you have used the resources differently? i. Could you please provide specific examples (probe: in-kind resources to families, improvements to infrastructure, transportation, parent awareness, different foods, etc.) ? 8. Was there anything you observed, as a result of the Food for Education program, that you did not expect -- affecting either school staff, parents, or students? a. [If yes:] Can you please explain? Efficiency 9. What, if any, changes have you seen in how this program was run during the past few years? Please be specific. (Probe: number and quality of trainings, observable infrastructure improvements, children affected, effective administration/communication, improved foods, access, etc.) 10. Have you ever provided feedback to school administrators, Mercy Corps, or any other related officials regarding the program? a. [If yes:] What feedback did you provide? b. [If not:] If you did have feedback, would you feel comfortable providing it? Impact 11. Can you comment on any changes you have observed (in the last 2 years) in the general health, nutrition, and dietary practices at school? a. During this time, has there been any additional support from parents, the community, or local or national government for school feeding? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? b. Also during this time, have you observed any changes in the nutrition behaviors of families of students at home? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? c. What specifically have you done differently, as a school cook, to increase the nutrition of students in the past 2 years? (Probe: new technologies, WhatsApp groups, cooking lessons, etc.) Sustainability 12. When this program ends, what plans or resources do you think are in place to continue providing food for primary school students? a. [If plans, including preliminary sustainability plans, and resources present]: Are these plans feasible in your estimation? (probe for specifics) i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 13. Please describe any interactions you may have had with local and national Ministry of Education and Science or Ministry of Health officials about the Food for Education school feeding program recently. Please be specific. 14. We are also very interested in your work directly with Mercy Corps. How have these school visits been for you? a. How useful, if at all, are they? b. Do they fit well with your schedule? c. Do you think they should continue to be as long and frequent, or more frequent? 107 Covid 15. Describe the direct effects of Covid-19 on your students with respect to the health goals of the program. a. How did families and children respond to school closures? 16. What changes did you observe that were taken to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. b. To what degree were you able to capitalize on new technologies? (like WhatsApp, online cooking classes, etc.) 17. Did home distribution of food rations from the Food for Education program occur in this community? a. [If yes:] Were you involved? i. What, if anything, could have been done to improve the process? Closing 18. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any questions about the program, concerns, or other types of feedback, please do not hesitate to call CARM at (0700-889-971). School Administrator KII Tool Hello, my name is ________________. I work with Erfolg Consulting in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools in Kyrgyzstan by providing over 2,000 metric tons of food to them. This program has also promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community-based reading activities, plus making infrastructure improvements for schools and promoting positive social behavior change through a network of trained community-based change agents. Goal of this interview: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this interview because we would like to learn more about how we can sustain the outcomes achieved thus far. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] 108 Thank you very much for agreeing to take part in this interview. Introduction 1. May I start by asking your name and position at the school? 2. How long have you worked at this school? 3. How would you rate your familiarity and level of interaction with Mercy Corps and MGD over the last several years (or life of program)? Relevance 4. What were the program design elements, from your perspective, that were most helpful to you as an administrator, and/or teachers and students? 5. If you were re-designing this program, what elements would you change to maximize either efficiency or effectiveness? (please be specific; probe for infrastructure, food, trainings, resource to teachers, librarians, students, etc.) a. What would you keep the same? (please be specific; probe for infrastructure, food, trainings, resource to teachers, librarians, students, etc.) i. Why do you say that? b. Did you make any changes, or adapt the programming to adjust things for your school and community that you felt made it better? i. If so, please be specific; and why were these changes necessary? 6. Based on what you know of Ministry of Education and Science and Ministry of Health, do you think the Food for Education program, as a whole, conflicted or complemented the objectives of the ministries? a. As an administrator, did the Food for Education program make your job easier or harder? b. In what ways; please be specific? (probe for ‘coherence’ issues – or complications with overlapping donors, programs, or even Ministry objectives, as well as any issues related to local context (dietary needs or preferences, travel distances, remoteness, access to internet, local environment and latrine building/engineering, etc.) Effectiveness 7. Based on your experience at this school, and with the goal of maximizing the literacy and health of students, would you have used Food for Education resources differently or the same? a. [If differently, ask:] How would you have used the resources differently? i. Could you please provide examples? ii. Did you make any changes that you felt were necessary? 1. If so: What were the effects from your point of view? 8. To what extent do you feel that you personally gained skills or developed professionally through the course of this program? a. In what ways, if any? 9. I’m going to read a list of activities from this program. For each, I’d like to hear if you and your school participated in these components, and, if so, how they were implemented. Please also tell me which of these components was most or least useful, and which elements the community, parents, or AO supported you with, as well. a. Kitchen repair? b. Food commodities support? c. Identifying additional budget sources for a full food menu for students? d. Literacy promotion? e. Trainings? f. Infrastructure support? g. Social Behavior Change? 10. Was there anything you observed, as a result of the Food for Education program, that you did not expect -- affecting either school staff, parents, or students? 109 a. [If yes:] Can you please explain? (Please be specific) (Possible probes, though try to avoid leading: added stress among staff, additional upkeep responsibilities, additional reporting; conversely: improved morale, better student performance, increased parental support and engagement) Efficiency 11. What, if any, changes have you seen in how this program was run during the past few years? Please be specific. (Probe: actors involved, distribution of resources, effectiveness of communication, training, reporting) 12. Were you ever provided information regarding previous evaluations or reviews of the Food for Education program? a. [If yes:] If you provided feedback on these previous evaluations, did you see any evidence of changes, or program adaptations, that aligned with your feedback, or otherwise? 13. How well over all do you feel you have been communicating, if at all, with Mercy Corps staff, and other actors involved in this program, like USDA, World Food Program, etc.? Impact 14. Can you comment on any changes you have observed (in the last 2 years) in health, nutrition, and dietary practices at school? Be as specific as possible, and describe any related outcomes (probe: fewer sick days, higher test scores, etc.?) a. During this time, has there been any additional support from local or national government for school feeding and/or literacy? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? b. Also during this time, have you observed any changes in the nutrition behaviors of families of students at home? i. [If yes:] What in particular have you observed, positively or negatively? 1. What specifically drove these results, from your perspective? a. Why do you say that? Sustainability 15. When this program ends or ended, what plans, or resources are in place to continue providing full hot meals for primary school students? a. [If plans, including a sustainability plan, and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? (probe for specifics) 16. As you may be aware, the National School Feeding Law was passed in 2020, mandating full hot meals in schools, as well as infrastructure, hygiene, and quality standards that Mercy Corps has been lobbying for. How do you feel about this law? a. To what extent do you feel that your school can implement the law? 17. What, if any, initiatives, or adaptations to the program did you personally take to improve it or make it more sustainable? a. If so: may you describe these? b. What did this require? (Probe: obstacles to or advantages in place to make these changes) 18. When this program ends, what plans, or resources are in place to continue improved literacy instruction practices for primary school students? a. [If plans, including a sustainability plan, and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? (probe: parental engagement, outside support, teacher dedication, etc.) ii. [If no:] What makes them unlikely to sustain? (probe for specifics, whether budgets, working parents, remoteness, internet access) 110 19. How well do you communicate with local and national Ministry of Education and Science officials, if at all? a. Have there been any changes in the past few years in this communication process? i. [If yes:] What changes have taken place? (probe: substance and focus of conversations, alignment with MGD objectives?) 1. Why, in your opinion, have these changes occurred? Covid 20. Describe the direct effects of Covid-19 on your students with respect to the health and literacy goals of the program. a. How did families and children respond to school closures? 21. What changes did you observe, by staff or students/parents, to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. b. What do you think has been the most creative and/or effective response by teachers, cooks and librarians to minimize the negative effect of school closures? 22. [If the Administrator did not bring up MC support:] I understand that Mercy Corps may have provided your school with take-home food rations, funding for new sinks, and other COVID responses. How useful, if at all, was this support? 23. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any questions about the program, concerns, or other types of feedback, please do not hesitate to call CARM at (0700-889-971). Local Government and District Department of Education KII Tool Hello, my name is ________________. I work with Erfolg Consulting in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools in Kyrgyzstan by providing over 2,000 metric tons of food to them. This program has also promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community-based reading activities, plus making infrastructure improvements for schools and promoting positive social behavior change through a network of trained community-based change agents. Goal of this interview: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this interview because we would like to learn more about how we can sustain the outcomes achieved thus far. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly 111 to us – your voice is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by asking your name and position in government? 2. How would you generally rate your: a. familiarity with Food for Education, and b. engagement with Mercy Corps and/or USDA Relevance 3. If you were re-designing the Food for Education program, to maximize its impact and sustainability where you live and work, what elements would you change? (be specific). a. What would you keep the same? i. Why (for any of the above)? b. What did not work well, from your point of view, if anything? 4. How well, if at all, did the Food for Education program complement the goals of the Ministry of Education and Science and the Ministry of Health? a. Why do you say that? b. Did the Food for Education program make your job, and affecting your Mission, easier or harder? i. Why do you say that? (please be specific and provide any examples) Effectiveness 5. Based on your experience in this region, and with the goal of maximizing the literacy and health of students, would you have allocated Food for Education resources differently or the same? a. [If differently, ask:] How would you have used the resources differently? i. Could you please provide examples? ii. Why are these changes needed in your opinion? iii. Were any of these changes enacted by you, or communicated to Mercy Corps as necessary? 6. Was there anything you observed, as a result of the Food for Education program, that you did not expect - - affecting either school staff, parents, or students? a. [If yes:] Can you please explain? 7. What, if anything, did you learn from any Mercy corps trainings you may have taken part in in the last two years? 8. How well do you feel that you communicated with Mercy Corps staff in the past two years? a. What, if any, changes have this engagement led to? b. [For DDoE:] Have you gone on a monitoring visit with Mercy Corps staff to a school or schools in your district? i. [If yes:] How useful, if at all, were these visits? Efficiency 9. What, if any, changes have you seen in how this program was run during the past few years? Please be specific. 10. Were you ever provided information regarding previous evaluations or reviews of the Food for Education program? a. [If yes:] If you provided feedback on these previous evaluations, did you see any evidence of changes, or program adaptations, that aligned with your feedback, or otherwise? 112 11. [Only for DDoE:] How, if at all, has this program led to any changes in your office's support for literacy activities in recent years? a. [If changes named:] Please explain what led to these changes. 12. [For both DDoE and AO:] What about in your support for school feeding programs in recent years a. [If changes named:] Please explain what led to these changes. Sustainability 13. When this program ends, what plans, or resources are in place to continue providing food for primary school students? a. [For AO only:] Has your office officially registered equipment provided by Mercy Corps to the school kitchen? i. Has your office registered the water filtration system at the school, if it has one, in order to allocate budgetary expenditures to maintaining it in the future? b. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 14. When this program ends, what plans, or resources are in place to continue improved literacy instruction practices for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 15. How well do you communicate with local school officials? a. For example, do you discuss with the school administrators allocating funds for infrastructure projects, in terms of “own contributions?” b. Have there been any changes in the past few years in this communication process? i. [If yes:] What changes have taken place? 1. Why have these changes occurred? Covid 16. Describe the direct effects of Covid-19 on students in this area with respect to the health and literacy goals of the Food for Education program. a. How did families and children respond? 17. How has Covid-19 affected your own office’s budgets, resources, and any plans for school improvement? a. I know that many plans fell through, but some districts had new funding for sanitary and hygiene projects. What happened here? 18. What changes did you observe to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. 19. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any questions about the program, concerns, or other types of feedback, please do not hesitate to call CARM at (0700-889-971). Social Behavior Change Activist FGD Tool Hello, my name is ________________. I work with Erfolg Consulting in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. 113 In 2017-2021, Mercy Corps supported 440 rural schools across the country by providing over 2,000 metric tons of supplementary commodities (enriched flour, lentils, rice, split peas, and vegetable oil), benefiting over 90,000 primary grade students. In addition, this program has promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community￾based reading activities. It has also supported infrastructure rehabilitation for schools, improved sanitation and hygiene facilities, built capacity with school administrators on project management and budgeting, improved knowledge of nutrition and hygiene in schools, and promoted social behavior change through a network of trained community-based change agents. Goal of this Discussion: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this focus group discussion because we would like to learn more about how we can sustain the outcomes achieved thus far. Managing Expectations: We cannot guarantee that every suggestion raised during this discussion will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed. Informed Consent: With your permission, I will be recording this discussion. What you say will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom. Do you agree to participate in this discussion, and to our conversation being recorded. [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. First off, could you raise your hand if you are a Social Behavior Change agent with Mercy Corps? a. Also, please raise your hand if you are a member of the Parent Teacher Association at this school. Relevance 2. What elements of the Food for Education program, from your perspectives, were most helpful to you, and your children, their learning, and health? a. Why do you say that? 3. If you were re-designing this program: a. What elements would you change from your perspective as a parent? Why? b. What would you keep the same? Why? c. What did not work well, from your point of view, if anything? Please be as specific as possible and provide an example, if possible. Effectiveness 4. What topics that you covered as an SBC change agent were the most useful for your community? a. Which topics were the most interesting or new for you personally? b. Which topics were most challenging to communicate to your community? 5. What changes, if any, did you see in your community after you distributed key messages to your neighbors? 6. Based on your experience in this community, and with the goal of maximizing the literacy and health of your children, would you have used Food for Education resources differently or the same? a. [If yes, ask:] How would you have used the resources differently? 114 i. Could you please provide specific examples (probes: online resources, in-kind resources to families, improvements to infrastructure, transportation, etc.)? 7. Was there anything you observed, as a result of the Food for Education program, that you did not expect -- affecting either school staff, yourselves, or your children? a. [If yes:] Can you please explain? 8. In light of Covid-19 and related school closures, which tools or activities where you involved with, or observe, that: a. Your child/ren were most engaged with? b. Your child/ren could access the most easily? c. Fostered literacy and learning the best, even if done remotely? d. Had the most parental engagement or support? 9. Of the activities above that were most beneficial, why do you think that was so? (Probe: teacher creativity, filled a need, accessible technology, student demonstrated real benefit from, etc.) Efficiency 10. What, if any, changes have you seen in how this program was run during the past few years? Please be specific. (Probe: number and quality of trainings, observable infrastructure improvements, children affected, effective administration and communication with parents, etc.) 11. Have you ever provided feedback to administrators, Mercy Corps, or any other related officials regarding the program? a. [If yes:] What feedback did you provide? b. [If not:] If you did have feedback, would you feel comfortable providing it? 12. Please tell us what you know about CARM? a. How did you use it in your community? Impact 13. In the past two years, have you changed what you feed your family at home as a result of the Food for Education program? i. [If yes:] What, in particular, have you changed? 1. Why have you changed this? 14. In the past two years, have you changed how much you work with your child on reading, at home/after school? i. [If yes:] How have these habits changed? 1. What inspired these changes? How did you learn these approaches? 15. Describe (generally) your level of involvement with the school and as a Social Behavior Change agent. i. What has been the effect on you, the school, or your family, through this participation? Sustainability 16. When this program ends, what plans, or resources do you believe are in place to continue providing food for primary school students? a. [If plans, including preliminary sustainability plans, and resources present]: Are these plans feasible in your estimation? (probe for specifics) i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 17. When this program ends, what plans, or resources do you believe are in place to continue improved literacy instruction practices for primary school students? (probe for specifics re: access to resources, online or offline parent groups, positive discipline, etc.) a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 115 18. Is there anything, as a parent, you think must still happen to sustain the best effects of the Food for Education program? Please describe. Covid 19. Describe the direct effects of Covid-19 on your children with respect to the health and literacy goals of the program. a. How did families and children respond to school closures and new ways of teaching? 20. I understand that since the Covid-19 pandemic started, as change agents you have helped spread messages about hygiene, health, viruses, and vaccines in your community. How has it felt to promote learning and change in this way? a. What would you have done differently, if anything, in terms of SBC messages around Covid-19? b. What worked particularly well in this area? 21. What changes by the school and teachers did you observe that were done to minimize the negative effects of Covid-19 a. Please describe them and gauge their effectiveness as a whole. b. To what degree were you and your children able to capitalize on new technologies? (like WhatsApp, audio or e-books, etc.) i. What affected a student’s learning under these new conditions, most? (Please be specific – limited internet, electricity, devices, learning curve with new platforms, for teachers or students, etc.) Closing 22. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We really appreciate your perspectives on all of this! Mercy Corps Program Staff IDI Tool Hello, my name is ________________. I work with a company called Causal Design that Mercy Corps has contracted with to evaluate the McGovern-Dole Food for Education and Childhood Nutrition program. Informed Consent: With your permission, I will be recording this interview in order to make sure I understand your words exactly. Your respected voice and perspective is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom and support. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by asking you to state your name and professional position? a. What has been your involvement with the Food for Education program? i. (Probe: ) What are your areas of responsibility in terms of implementing the program? Which components do you work most closely on? Relevance 2. If you were re-designing the Food for Education program, to maximize its impact and sustainability, what elements would you change? (be specific). 116 a. What would you keep the same? i. Why (for any of the above)? 3. How well, if at all, did the Food for Education program complement the goals of the Ministry of Education and Science and the Ministry of Health, do you think? a. Why do you say that? Effectiveness 4. What, if anything, did you observe, as a result of the Food for Education program, that you did not expect - - affecting either school staff, parents, or students? a. [If yes:] Can you please explain? Efficiency 5. What, if any, changes have occurred in how this program was run during the past few years? Please be specific. a. I’m particularly interested in changes that may have occurred in terms of gender inclusion around the WASH and SBC components. Can you please talk a bit about these areas? b. Also, I understand that infrastructure and winterization components of the program have increased significantly since the MTE. Can you please tell me a bit more about how these shifts happened? 6. How well do you feel this initiative responded to and incorporated comments, as appropriate, from the MTE? a. What about the point around the potential programmatic overlap around USAID’s Time to Read (TTR) program? Do you feel like the LOU and revisions to this program resolved the issue? 7. How, if at all, has this program led to any official changes in government support for literacy activities in recent years? a. [If changes named:] Please explain what led to these changes. b. What about in your support for school feeding programs in recent years? i. [If changes named:] Please explain what led to these changes. Sustainability 8. When this program ends, what plans or resources are in place to continue providing food for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 9. When this program ends, what plans or resources are in place to continue improved literacy instruction practices for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? Covid 10. Please describe the direct effects of Covid-19 on students with respect to the health and literacy goals of the Food for Education program. a. How did families and children respond? 11. What changes did you enact to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. i. (Probe: ) How well would you say Mercy Corps responded to the conditions and restrictions brought about by COVID? What, if anything, could have been done better? 12. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? 117 Thank you so much for your time! We greatly appreciate your perspective on all of this! National Government Official(s) IDI Tool Hello, my name is ________________. I work with a company called Causal Design in partnership with Mercy Corps. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools across the country by providing over 2,000 metric tons of supplementary commodities (enriched flour, lentils, rice, split peas, and vegetable oil), benefiting over 90,000 primary grade students. In addition, this program has promoted literacy by providing reading materials to public schools and libraries, promoting positive discipline training for primary grade teachers, and organized community￾based reading activities. It has also supported infrastructure rehabilitation for schools, improved sanitation and hygiene facilities, built capacity with school administrators on project management and budgeting, improved knowledge of nutrition and hygiene in schools, and promoted social behavior change through a network of trained community-based change agents. Goal of this interview: According to the current plan, this phase of the McGovern-Dole Food for Education and Child Nutrition program will end in September 2021. We have invited you to this interview because we would like to learn more about how we can sustain the outcomes achieved thus far. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon by Mercy Corps, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared back with the public after the analysis has been completed. Informed Consent: With your permission, I will be recording this interview in order to make sure I understand your words exactly. Your respected voice and perspective is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom and support. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by confirming your name and position in government? 2. How would you generally rate your: a. familiarity with Food for Education, and b. engagement with Mercy Corps and/or USDA Relevance 3. If you were re-designing the Food for Education program, to maximize its impact and sustainability where you live and work, what elements would you change? (be specific). a. What would you keep the same? i. Why (for any of the above)? b. What did not work well, from your point of view, if anything? 118 4. How well, if at all, did the Food for Education program complement the goals of the Ministry of Education and Science and the Ministry of Health? a. Why do you say that? b. Did the Food for Education program make your job, and affect your Mission, easier or harder? i. Why do you say that? (please be specific and provide any examples) Effectiveness 5. With the goal of maximizing the literacy and health of students in the Kyrgyz Republic, would you have used Food for Education resources differently or the same? a. [If differently, ask:] How would you have used the resources differently? i. Could you please provide examples? ii. Why are these changes needed in your opinion? iii. Were any of these changes enacted by you, or communicated to Mercy Corps as necessary? 6. Was there anything you observed, as a result of the Food for Education program, that you did not expect - - affecting either school staff, parents, or students? a. [If yes:] Can you please explain? Efficiency 7. What, if any, changes have you seen in how this program was run during the past few years? Please be specific. 8. Were you ever provided information regarding previous evaluations or reviews of the Food for Education program? a. [If yes:] If you provided feedback on these previous evaluations, did you see any evidence of changes, or program adaptations, that aligned with your feedback, or otherwise? 9. How, if at all, has this program led to any changes in your office's support for literacy activities in recent years? a. [If changes named:] Please explain what led to these changes. b. What about your support for school feeding programs in recent years? i. [If changes named:] Please explain what led to these changes. Sustainability 10. When this program ends, what plans or resources are in place to continue providing food for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? 11. When this program ends, what plans or resources are in place to continue improved literacy instruction practices for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? Covid 12. Describe the direct effects of Covid-19 on students with respect to the health and literacy goals of the Food for Education program. a. How did families and children respond? 13. What changes did you observe to minimize the negative effects of Covid-19? a. Please describe them and gauge their effectiveness as a whole. 14. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? 119 Thank you so much for your time! We greatly appreciate your perspective on all of this! USDA IDI Tool Hello, my name is ________________. I work with a company called Causal Design that Mercy Corps has contracted with to evaluate the McGovern-Dole Food for Education and Childhood Nutrition program. Informed Consent: With your permission, I will be recording this interview in order to make sure I understand your words exactly. Your respected voice and perspective is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom and support. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by asking you to state your name and professional position? a. What has been your involvement with the Food for Education program? i. (Probes: ) What are your areas of responsibility in terms of overseeing the program? Which components do you work most closely on? Which components are you most interested in? Relevance 2. If you were re-designing the Food for Education program, to maximize its impact and sustainability, what elements, if any, would you change? (be specific). a. What would you keep the same? i. Why (for any of the above)? 3. How well, if at all, did the Food for Education program complement the goals of the USDA, particularly the International Food Assistance Division, do you think? a. Why do you say that? Effectiveness 4. What are you most proud of about the Food for Education program in Kyrgyzstan, over all? a. What has been the most challenging component for you? Efficiency 5. What, if any, changes have occurred in how this program was run during the past few years? Please be specific. a. I’m particularly interested in changes that may have occurred in terms of gender inclusion around the WASH and SBC components. Can you please talk a bit about these areas? b. Also, I understand that infrastructure and winterization components of the program have increased significantly since the MTE. Can you please tell me a bit more about how these shifts happened? 6. How well do you feel this initiative responded to and incorporated comments, as appropriate, from the MTE? a. What about the point around the potential programmatic overlap around USAID’s Time to Read (TTR) program? How well do you feel like that point was resolved? 7. How aware, if at all, are you of this program leading to any official changes in the Kyrgyzstan government's support for literacy activities in recent years? 120 a. [If changes named:] Please explain what led to these changes. b. What about in the Kyrgyzstan government’s support for school feeding programs in recent years i. [If changes named:] Please explain what led to these changes. Sustainability 8. When this program ends, how likely do you feel the Kyrgyzstan government is to continue providing food for primary school students? 9. When this program ends, how likely do you feel the Kyrgyzstan government is to continue improved literacy instruction practices for primary school students? Covid 10. How pleased are you, if at all, with the program adaptations that Mercy Corps made to minimize the negative effects of Covid-19? a. (Probe: ) What, if anything, could have been done better? 11. What else should we know about the Food for Education program as it has been implemented in Kyrgyzstan, from your perspective? Thank you so much for your time! We greatly appreciate your perspective on all of this! USAID IDI Tool Hello, my name is ________________. I work with a company called Causal Design that Mercy Corps has contracted with to evaluate the McGovern-Dole Food for Education and Childhood Nutrition program. Informed Consent: With your permission, I will be recording this interview in order to make sure I understand your words exactly. Your respected voice and perspective is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom and support. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by asking you to state your name and professional position? Relevance 2. I understand that your office is funding a Time to Read (TTR) program in Kyrgyzstan. Can you tell me a bit about the goals and activities of that program, and how it did or did not overlap with the Food for Education program’s literacy promotion activities? a. I understand that Chemonics and Mercy Corps staff have signed two LOUs on the coordination between these USAID and USDA-funded projects. How well has that coordination transpired between the two initiatives? i. What, if anything, has been challenging in this coordination? ii. What, if anything, has gone particularly well? Sustainability 3. When both these programs end, how likely do you feel the Kyrgyzstan government is to continue improved literacy instruction practices for primary school students? 4. What else should we know about either your Time to Read program or the USDA’s Food for Education program as they have been implemented in Kyrgyzstan, from your perspective? 121 Thank you so much for your time! We greatly appreciate your perspective on all of this! (Former) Save the Children Staff IDI Tool Hello, my name is ________________. I work with a company called Causal Design that Mercy Corps has contracted with to evaluate the McGovern-Dole Food for Education and Childhood Nutrition program Informed Consent: With your permission, I will be recording this interview in order to make sure I understand your words exactly. Your respected voice and perspective is very important to help us learn about the successes and challenges of this program, so that it can be improved in the future, both here in Kyrgyzstan and in other countries. We cannot sustain these results without your wisdom and support. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Introduction 1. Can I start by asking you to state your name and current professional position? a. What has been your involvement with the Food for Education program? i. (Probe: ) What are your areas of responsibility in terms of implementing the program? Which components did you work most closely on? Relevance 2. If you were re-designing the Food for Education’s literacy component, to maximize its impact and sustainability, what elements would you change? (be specific). a. What would you keep the same? i. Why (for any of the above)? 3. How well, if at all, did the Food for Education program complement the goals of the Ministry of Education and Science and the Ministry of Health, do you think? a. Why do you say that? Effectiveness 4. What, if anything, did you observe, as a result of the Food for Education program, that you did not expect - - affecting either school staff, parents, or students? a. [If yes:] Can you please explain? Efficiency 5. How well do you feel this initiative responded to and incorporated comments, as appropriate, from the MTE? a. What about the point around the potential programmatic overlap around USAID’s Time to Read (TTR) program? Do you feel like the LOUs and revisions to this program resolved the issue? i. What, if anything, about the coordination of these two programs was challenging? ii. What, if anything, about the coordination of these two programs went well? 6. How, if at all, has this program led to any official changes in government support for literacy activities in recent years? a. [If changes named:] Please explain what led to these changes. Sustainability 7. When this program ends, what plans or resources do you feel are in place to continue improved literacy instruction practices for primary school students? a. [If plans and resources present]: Are these plans feasible in your estimation? 122 i. [If yes:] What makes them feasible? ii. [If no:] What makes them unlikely to sustain? Closing 8. What else should we know about the Food for Education program, from your perspective? a. What, if any, additional suggestions do you have for how it could be improved going forward? Thank you so much for your time! We greatly appreciate your perspective on all of this! School Observation Checklist Visit Date [YYYY-MM-DD] Name of Observer School Name Kitchen and Dining Hall Equipment provided by Mercy Corps (select all appropriate) In some cases, the stove and oven are together (one piece of equipment); then one set of logos and instructions applies to both. Stove Oven Fridge Dishwashing sink Doughmaker Describe _______________________________________________ 123 Stove: Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby Oven: Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby Fridge: Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby Dishwashing Sink: Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby Doughmaker: Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby Other : 124 Is present in the kitchen during the visit Is functioning during the visit Has clearly visible MC/USDA logos Has exploitation manual/instructions on it or nearby If any of the Mercy Corps-provided equipment does not work, please explain the reason: Does all new donated equipment have inventory numbers to indicate that it was registered with the local government? If no, what does not have inventory numbers denoting registration? Is there a constant water supply in the kitchen? Bottled water or water that had to be delivered/purchased on a regular basis do not count as constant water supply (unreliable). Yes No Is there a separate hand washing sink for the cook and other kitchen staff? Not necessarily provided by MC. Yes No Does the school provide full hot meals five days per week? Yes No Is there a filled out and clearly visible kitchen cleaning schedule? Yes No During your visit, did you notice any of the following violations? 125 Cook/kitchen staff or other people walk into the kitchen in street shoes (without covers) Cook/kitchen staff or other people walk into the kitchen in street clothes (without covers) Unauthorized personnel or children in the kitchen No, none Is there a ventilation system in the kitchen? Not necessarily installed by MC. Yes No If so, is it in working condition? Yes No Are there enough chairs for all students in each group that eats at the same time? Benches can be included. Estimated during the feeding process observation. Yes No Are Mercy Corps visibility and program materials present? Yes No Are there handwashing facilities (sinks) in or near the dining hall? Yes No At the time of your visit, was there running water in these sinks? Yes No You can take up to 3 pictures of anything that was particularly interesting or important during the visit. For example, a diverse school menu, food storage conditions in the kitchen, quality issues with equipment or infrastructure, etc. [Picture upload functionality] 126 School storage In the school storage room, are there: A functioning ventilation system Commodities stored less than 15 cm from the wall Commodities stored less than 15 cm from the oor Signs of rodents inside the room Cracks in walls/floor/ceiling large enough for birds or rodents to get through Working locks Signs of birds inside the room Signs of insects inside the room (ants, moth, etc.) Bad smell (chemicals, spoiled food, mold, etc.) Visible water damage on the walls/floor Shelves are unstable / commodities can fall off N/A On the shelves, are there: Damaged containers or bags (torn, leaky, etc.) Spoiled produce or commodities Chemicals, cleaning supplies N/A [Picture upload functionality] Latrines General questions in this section apply to all latrines regardless of who built them. For MC-built latrines there is an additional question in the end (if the relevant grant was selected in the beginning of the questionnaire). What type of latrines do students use? If the school has both latrine types used by students, then select both options. Inside the building Outside How many student latrines are there in total? 127 How many students can use latrines at the same time (total number of stalls)? Are there gender separated latrines at the school? Yes No Are all student latrines in working order at the time of your visit? Yes No Are there handwashing facilities with soap and water within 2-3 meters from the latrine inside the building? Yes No If not, are there inside handwashing stations accessible to students near the latrines, but further than 2-3 meters, in addition to those in the cafeteria/kitchen area? Yes No If the latrine was repaired or built by Mercy Corps, are there: Doors in each stall Locks on each door Functioning ventilation system Accessibility provisions, for people with disabilities, for outdoor latrines built by MC [Picture upload functionality] Winterization and other infrastructure grants Infrastructure grant received from Mercy Corps Select all that applies; if the school received a water supply improvement grant, describe in detail under "other". 128 Window replacement Door replacement Heating system installation or repair Roof repair Floor repair Other ->ADD: Kitchen ventilation? Water filters? Handwashing facilities? Describe (if other) ___________________________________________________ Were the new windows installed correctly? Evenly, neatly, etc. Yes No Are all windows in working order at the time of the visit? Can open/close, no cracks, etc. Yes No Were the doors installed correctly? Evenly, neatly, etc. Yes No Are all doors in working order at the time of your visit? Can open/close, no cracks, etc. Yes No If the heating system was installed or repaired with a grant from Mercy Corps, is it in working order at the time of the visit? Yes No If the roof was repaired, have there been any issues with it since? 129 Yes No If yes, how were the issues addressed? If the floor was repaired, have there been any issues with it since? Yes No If yes, how were they addressed? [Picture upload functionality] Handwashing facilities Are the handwashing facilities accessible to primary grade students? Yes No Are the handwashing facilities accessible to students with physical disabilities? Yes No Comments _______________________________________________ Reading Corner Is the Reading Corner accessible to primary grade students? Yes No If not, why? 130 Books are too high for little kids to reach Classroom/library was closed Other Explain, please Are there clearly visible logos and instructions around the Reading Corner? Check all that apply. Instructions MC/USDA logo Save the Children logo Sign up sheet None of the above Are there MC/USDA provided books in the Reading Corner? Yes No If yes, how many books? Are additional books given to students to borrow and take home? Yes No Comments_____________________________________________________ [Picture upload functionality] Water filters Is a water filtration system present? Yes No If so, is it working? 131 Yes No If not, why not? No power in school Pump broke Filter(s) broke (full/overloaded) Pipe(s) broke Other Describe If it is functioning, Is the water filtration system actively used? Yes No If not, why not? No habit Just installed, have not tried yet Use only occasionally to save filter cartridges Don't know how to use (was not explained or forgot) Other Describe [Picture Upload functionality] 132 Annex C: Terms of Reference McGovern-Dole Food for Education and Child Nutrition 2017 Final Evaluation These Terms of Reference (TOR) are a plan for preparing for and conducting an end-of-program evaluation for the McGovern-Dole Food for Education and Child Nutrition program funded by the US Department of Agriculture (USDA) and implemented by Mercy Corps Kyrgyzstan from 2017 to 2021. This Final Evaluation (FE) is commissioned by Mercy Corps Kyrgyzstan. Date: Feb 1, 2021 1) Program to be Evaluated McGovern-Dole Food for Education and Child Nutrition Program Funded by the US Department of Agriculture (USDA) Implemented October 2017 – September 2021 2) Purpose of the Evaluation This Final Evaluation has several key objectives: ● Assess and provide evidence on the performance and results of the program against set objectives and targets; ● Highlight program achievements and successes supported by evidence gathered during this evaluation; ● Evaluate the degree of school feeding sustainability in target schools after several continuous years of program implementation; ● Identify key lessons learned and provide specific, actionable and evidence-based recommendations for the ongoing and future McGovern-Dole programs. This Final Evaluation will also assess how the achievements are viewed through the perspectives of program stakeholders: MoES officials, school staff, local community, parents of primary grade students, and local governments. 3) Background Mercy Corps has been working in Kyrgyzstan since 1994. For over fifteen years, Mercy Corps has been providing nutritional and technical support to education institutions in the Republic of Kyrgyzstan, including public schools, kindergartens, and vocational schools. Since 2003, Mercy Corps has served as the implementing partner of the United States Department of Agriculture (USDA) McGovern-Dole Food for Education and Child Nutrition program (MGD). This program is implemented in support of the National School Feeding Program developed by the Ministry of Education and Science (MoES) of the Kyrgyz Republic, which is a key stakeholder for this program. In 2017-2021, Mercy Corps supported 440 rural schools across the country by providing over 2,000 metric tons of supplementary commodities (enriched flour, lentils, rice, split peas, and vegetable oil). More than 90,000 primary grade students benefited from nutritious hot meals in schools prepared from these commodities. In addition to school feeding, the program includes the following components for a comprehensive approach: 133 ● Literacy promotion: provision of supplementary reading materials to public schools and libraries, Positive Discipline training for primary grade teachers, as well as organization of out-of-school and community-based reading activities; ● Infrastructure rehabilitation to support winterization of schools (repair or replacement of old windows, roofs, heating systems, etc.); ● Sanitation and hygiene: installation of handwashing stations, repair or construction of improved school latrines; ● Capacity building: tailored training for school administration on project management and budgeting skills, improved knowledge on nutrition and hygiene in schools, etc.; ● Social behavior change communication: introducing improved nutrition and hygiene behaviors to the target communities through a network of trained community-based change agents. The total budget for the 2017-2021 McGovern-Dole Food for Education and Child Nutrition program was $15,453,090.39. Full results framework can be found in Annex D. Over the course of the implementation period to be evaluated (2017 – 2021), the program worked with a total of 440 public schools in rural areas: ● 84 schools enrolled in 2017 and continued receiving full programmatic support until May 2020 when they officially graduated from the program and continued school feeding independently; ● 137 schools enrolled in 2018 and will graduate in May 2021; ● 219 schools enrolled in 2019 and will graduate in May 2021. Distribution of schools by province (province) can be found in Annex II. 4) Evaluation Design i. Evaluation Questions This Final Evaluation should answer the following key evaluation questions with the main focus on sustainability of the intervention: Relevance: (1) Was the program’s design an appropriate solution to meet the stated objectives and results? (2) How did stakeholders perceive the program’s design and its activities? (3) To what degree was program implementation aligned with the goals and objectives of the relevant government ministries (MoES, MoH)? Effectiveness: (1) Did the program meet indicator targets? (2) How do program staff and stakeholders perceive the effectiveness of the processes and resources used by the program? (3) Did program activities show signs of creating unintended positive or negative outcomes? If yes, which activities contribute to this? Efficiency: (1) Have any programmatic or financial adjustments been made during the course of the program? Why? If so, were they appropriate to the situation? (2) To what degree was feedback from the previous evaluations and stakeholder input incorporated into the program? Impact: (1) To what degree have there been changes in each of the following that are attributable to the program: (a) health, nutrition and dietary practices at the schools; (b) government financial support for literacy and school feeding activities, including local government; (c) nutrition behaviors of families for children at home. (2) To the degree that attribution is possible, what combination of activities is responsible for the above impacts? 134 Sustainability: (1) To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue primary school feeding beyond the program? (2) To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue improved literacy instruction practices beyond the program? (3) Have the program activities contributed towards improved information sharing between primary schools and local and national MoES officials? (4) What key factors have contributed to the variation in target schools’ degree of feeding program success55 as measured through dietary diversity, continuous community support, and other aspects of program implementation? COVID-19 Impact: (1) To what extent as the COVID-19 pandemic and associated disruptions in education, such as school closure, affected effectiveness and sustainability of school feeding programs established under the program? (2) To what extent has the COVID-19 pandemic and associated disruptions affected program outcomes for other components? (3) To what extent has MC been able to adapt program implementation during the COVID-19 pandemic? ii. Evaluation Methodology The evaluation methodology will be a non-experimental study that will employ several data collection tools to explore program results related to key program indicators, and to answer evaluation questions listed above. The bidding evaluation entity is required to design and propose a mixed methods evaluation methodology including, but not limited to the following data collection methods: ● Quantitative analysis of existing monitoring data collected throughout the program against set indicator targets; ● Quantitative analysis of an endline reading assessment should it be determined that such an assessment is appropriate56; ● Semi-structured interviews with key program staff, stakeholders from MoES and other relevant government bodies at district and community levels; ● Group interviews or FGDs with adult program participants: parents, school staff and administration, local government representatives; ● Structured observation carried out in a sample of program schools to assess state of provided equipment and infrastructure improvements57; ● Document review, including a nationally representative quasi-experimental Baseline Study on Nutritional and Learning Variables in Kyrgyz Households conducted in 2019 by an independent research company;58 55 Taking into account three different stages of school enrolment (2017, 2018, 2019), different graduation status at the time of evaluation, and slight variation in program intervention package received at each stage. 56 A baseline reading assessment was conducted by the implementation partner, Save the Children, in 2018. A follow up endline assessment using the same methodology was planned for the final evaluation, but at the moment its feasibility is unclear due to the ongoing COVID-19 pandemic. This issue will be discussed in more detail with the selected evaluating agency and Dave the Children during the inception phase. 57Appropriate sample size and sampling approach should be suggested by the bidding evaluation entity in the application; it should be taken into consideration that the program is implemented in all 7 provinces (provinces) in the country. 58 Second phase of the study is planned for Spring 2021 and aims to assess program impact on household-level behaviors. 135 ● Qualitative methods such as contribution analysis, to understand the effectiveness of program interventions and how they led to the observed changes. The details of the evaluation approach, including methods and design, will be further discussed and negotiated with the selected evaluation entity. However, based on the recently conducted evaluations for this program on Kyrgyzstan, MC included a suggested design in Annex F. iii. Existing Program Information Sources Mercy Corps will provide the selected evaluation entity with the following documents and data to support carrying out agreed upon evaluation activities: ● MC-USDA McGovern-Dole Agreement and all subsequent modifications reflecting key adjustments made to the program design, including the COVID-19 pandemic; ● Baseline assessment report and dataset, including the baseline reading assessment conducted by Save the Children; ● Infrastructure needs assessment conducted to select schools for infrastructure and WASH repair grants; ● Gender assessment report conducted by an independent researcher to support adaptation of the Social Behavior Change component to include more male participants; ● Biannual program reports submitted to USDA (seven reports at the time of evaluation); ● Midterm program evaluation report conducted by independent evaluators in 2019, available here; ● All tools used for the baseline assessment, midterm evaluation and infrastructure needs assessment; ● Baseline Study on Nutritional and Learning Variables in Kyrgyz Households report and the dataset; ● Yearly program workplans approved by USDA, including narrative description; ● Monitoring, Evaluation and Learning Plan with a summary of adjustments made throughout the program; ● Performance Management Plan containing information on program indicators and results up to and including at the time of evaluation; ● All relevant SOPs, such as commodities distribution, outlining MC operational procedures for the program; ● List of participating program schools, local government bodies, and key stakeholders at MoES, MoH, etc.; ● Program map with all participating schools marked to support sampling; ● Program database with all programmatic information on participating schools, conducted trainings, grants and commodities, and other assistance provided; ● Indicator Performance Tracking Table with biannual and LoP indicator results and comments; ● Indicator-specific databases, such as number of school meals provided and student attendance; ● Routine monitoring reports from the field team on program implementation quality; ● Program success stories, media coverage archives, video and photo reports, infographics, and other media and communication materials; 136 ● Training materials on various topics covered by the program, such as nutrition and social behavior change. 5) Team Composition, Roles and Responsibilities This FE will be led by an independent evaluation entity working in cooperation with the MC MEL Manager, key program staff (including Country and Programs Directors), government officials, and participant institutions and individuals. Suggested roles and responsibilities for all actors can be found in Annex IV. To successfully carry out this FE, the evaluation entity is required to provide a dedicated evaluation team consisting of:59 ● Lead Evaluator/Team Leader with at least 10-15 years of evaluation experience, particularly of long￾term development programs and complex and dynamic contexts; ● An evaluator with experience with evaluating food assistance and/or school feeding programs will be a significant advantage; ● An evaluator with expertise in designing mixed-methods evaluations and data collection tools, including providing training, troubleshooting during data collection, and conducting interviews and/or FGDs; ● 5-10 data collectors/enumerators to conduct interviews, FGDs, and site visits for observations; it is crucial for the field data collection team to have relevant language skills (Russian and/or Kyrgyz) and in-depth knowledge of the local context60. The evaluation entity will be responsible for training and supervising the field data collection team. Note: it is possible for one team member to have more than one area of expertise outlined above, in which case his/her resume should clearly show it. In addition to the technical skills and expertise outlined above, it would be advantageous to have the following: ● Technical expertise in nutrition, child nutrition, or food security; ● Knowledge of and field experience in the Kyrgyz Republic; ● Previous experience with carrying out evaluation assignments for USDA; ● Previous experience with Mercy Corps MEL policies and requirements, as well as organizational culture and programming; ● Experience with programs based in educational institutions, particularly in the post-Soviet space. Selection criteria are outlined in Annex H. Preliminary Schedule The activities listed in the table above with the suggested level of effort should be carried out within the following preliminary timeline: 59It is possible to have one team member fulfill several requirements; in this case, relevant supporting documents must be included in the application package to highlight that team member’s capacity to fill multiple roles. 60MC Kyrgyzstan has a contracted local data collection services supplier that provided enumerators for several program assessments that can be used for this evaluation. 137 Evaluation entity contracting February 2021 Document review, methodology design, tools development, inception report drafting and review March 2021 Inception report submission with the data collection plan and finalized tools Early April 2021 Field data collection in-country, including training on tools (3 weeks) April 2021 Data analysis and report drafting May 2019 Draft report submission to MC End of May 2019 Draft report submission to USDA End of June 2021 Final Evaluation Report submission July 30, 2021 Exact dates for key deliverables and the overall evaluation timeframe will be discussed with the selected evaluation entity and specified in the contract. Report Structure & Content The Final Evaluation Report must be written in English, not exceed 50 pages (excluding annexes) and follow the structure outlined below: ● Cover Page, List of Acronyms ● Table of Contents ● Executive Summary: This section should be a clear and concise stand-alone document that gives readers the essential contents of the evaluation report, including a summary of major findings, lessons learned, and recommendations. ● Methodology: This section should be sufficiently detailed to help the reader judge the accuracy of the report and its findings. ● Limitations: This section should address constraints and limitations of the methodology, and the implications of these limitations for the findings, including whether and why any of the evaluation findings are inconclusive. ● Results: This section should provide a clear assessment of progress with respect to indicators / targets / objectives and evaluation questions. ● Synthesis, Recommendations, and Lessons Learned: This is space for the evaluation team to think about the data and results and make concrete recommendations for current or future program improvements, pull out organization lessons learned, and generally comment on data and results. 138 ● Conflicts of Interest: Disclose any conflicts of interest or the appearance of conflicts of interest, including the interest of program staff in having a successful program. ● Annexes: These should include a complete file of data collection instruments in English and translations; list of stakeholder groups with number and type of interactions but without PII61; TOR, data collection protocols used, etc. Other deliverables include: ● Cleaned datasets and original transcripts of all primary data collected; ● All photographs taken during field visits; ● A 2-3 pages stand-alone brief describing key findings and recommendations for a wider audience (non￾evaluators). 7) Stakeholder Involvement and Plans for Dissemination The key audiences for this FE will be USDA, MC program management and staff, local community and government partners such as the MoES and MoH, whose activities will be addressed in the evaluation, and other actors working to improve child nutrition and educational environment Kyrgyzstan. First, preliminary results of the FE will be presented at the Program Close Out workshop. Second, Mercy Corps will organize a stakeholder workshop to discuss key results and lessons learned with the key audience composed of MoES and MoH officials, international organizations, and participants. A 2-3 pages brief and program lessons learned will be distributed to key Program stakeholders in Russian, English and Kyrgyz. It will also be available on MC Kyrgyzstan website, MC Digital Library and through other sources as relevant. All final versions of international food assistance evaluation reports will be made publicly available. Evaluators shall provide a copy of the evaluation reports that is free of personally identifiable information (PII) and proprietary information. Final versions of evaluation reports ready for publication should be accessible to persons with disabilities. 62 8) COVID-19 Contingency Plan At the time of TOR publication, public schools in Kyrgyzstan follow their district-level public health authorities’ guidance on whether or not they can accept students. In some cases, schools are closed completely, and all classes are held online. In other cases, only first graders are allowed to attend classes in school while all others study remotely. Moreover, the approach in any district can change unpredictably as the second wave is gaining strength across the country. Understandably, these conditions may last until the end of this school year and affect final program evaluation planning. 61 No personally identifiable information will be included in the final version for public dissemination. The consultant will have to follow USDA’s guidance on how to make the final report Section 508 compliant to be posted publicly. 62 For guidance on creating documents accessible to persons with disabilities, please see the following resources: https://www.section508.gov/create/documents https://www.section508.gov/create/pdfs 139 If the COVID-19 pandemic continues to affect school schedules and accessibility by the time of the evaluation fieldwork, MC will work with the selected evaluation entity to adapt the assessment methodology and tools for remote data collection to the extent possible, using guidance for remote monitoring provided by USDA. At the inception stage, MC will coordinate closely with the Ministry of Education and Science to get a full list of preschools that are open at the time of data collection and can be visited, as well as those that implement online education and can be monitored remotely. Based on this, the selected company will adapt its data collection plan. For those schools that implement online education at the time of data collection, MC and the selected evaluation entity will develop a set of adapted data collection tools that are suitable for remote implementation. For example, an adapted observation form will only include questions that are possible to answer through using video and photos, and interviews or group meetings can be conducted through WhatsApp, Skype, Zoom or Google Meet, all of which have been successfully used by the program this year. 140 Annex D: Results Framework 141 142 Annex E: Program Schools Distribution by Region Province 2017 Cohort (84 schools) 2018 Cohort (137 schools) 2019 Cohort (219 schools) Total Chui 13 57 70 140 Talas 19 2 7 28 Issyk-Kul 3 9 11 23 Naryn 2 11 10 23 Osh 17 19 54 90 Jalal-Abad 15 32 52 99 Batken 15 7 15 37 Total # of Primary Grade Students 21,000 35,000 40,000 96,000 Annex F: Suggested Evaluation Methodology ● Sample size: 70 schools distributed proportionally by cohort and province (15% of all program schools) ● 2-3 key interviews per school (school director, school cook, another key staff involved in school feeding) ● 15-20 District Departments of Education interviews (suggest those not covered in the midterm evaluation) ● 15-20 interviews with local government (ayl okmutu) ● Interviews with key stakeholders at MoES, MoH, USDA, USAID (TTR), WFP and Save the Children ● 10 FGDs with parents of primary grade children and/or SBC change agents63 Evaluation Question Suggested Data Sources (not exhaustive) Relevance: KII with USDA KIIs with MC program management 63 In most cases SBC change agents are parents of primary grade children in program schools. 143 1. Was the program’s design an appropriate solution to meet the stated objectives and results? 2. How did stakeholders perceive the program’s design and its activities? 3. To what degree was program implementation aligned with the goals and objectives of the relevant government ministries (MoES, MoH)? KIIs with stakeholders (MoES, MoH, local government, participants) KII with Save the Children, WFP and USAID (TTR) Review of program modifications and design notes Review of the program reports Review of the National School Feeding Law Effectiveness: 1. Did the program meet indicator targets? 2. How do program staff and stakeholders perceive the effectiveness of the processes and resources used by the program? 3. Did program activities show signs of creating unintended positive or negative outcomes? If yes, which activities contribute to this? Indicator Performance Tracking Table (IPTT) KIIs with program staff KIIs with stakeholders and participants FGDs with parents of primary grade students KII with Save the Children Structured school observation (infrastructure, kitchen, etc.) Review of program reports Review of the Baseline Study on Nutritional and Learning Variables in Kyrgyz Households results Efficiency: 1. Have any programmatic or financial adjustments been made during the course of the program? Why? If so, were they appropriate to the situation? 2. To what degree was feedback from the previous evaluations and stakeholder input incorporated into the program? KII with program management KII with USDA KII with Save the Children Review of program reports Review of the Midterm Evaluation report Impact: 1. To what degree have there been changes in each of the following that are attributable to the program: (a) health, nutrition and dietary practices at the schools; (b) government financial support for literacy and school feeding activities, including local government; (c) nutrition behaviors of families for children at home. Review of the Baseline Study on Nutritional and Learning Variables in Kyrgyz Households results Review of the follow-on study results when available (tentatively April 2021) KIIs with program participants (school staff, DDoE, local government) 144 2. To the degree that attribution is possible, what combination of activities is responsible for the above impacts? FGDs with parents of primary grade students and/or SBC change agents Sustainability: 1. To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue primary school feeding beyond the program? 2. To what extent are school administrators, local and national government agencies showing signs of their commitment and ability to continue improved literacy instruction practices beyond the program? 3. Have the program activities contributed towards improved information sharing between primary schools and local and national MoES officials? 4. What key factors have contributed to the variation in target schools’ degree of feeding program success64 as measured through dietary diversity, continuous community support, and other aspects of program implementation? KIIs with program participants (school staff, DDoE, local government) KII with MoES and MoH Review of program reports Review of other program documents (meeting notes with local government and school directors, etc.) Review of the National School Feeding Law and relevant by-laws Structured school observation (infrastructure, kitchen, etc.) COVID-19 Impact: 1. To what extent as the COVID-19 pandemic and associated disruptions in education, such as school closure, affected effectiveness and sustainability of school feeding programs established under the program? 2. To what extent as the COVID-19 pandemic and associated disruptions affected program outcomes for other components? 3. To what extent has MC been able to adapt program implementation during the COVID-19 pandemic? Review of program modifications and design notes Review of program reports and other documents KII with program management KII with USDA KIIs with program participants (school staff, DDoE, local government) 64 Taking into account three different stages of school enrollment (2017, 2018, 2019), different graduation status at the time of evaluation, and slight variation in program intervention package received at each stage. 145 Annex G: Suggested Roles and Responsibilities Evaluation Entity ● Develop detailed evaluation methodology, including sampling and data collection tools ● Recruit local enumerators ● Train the data collection team on the evaluation methodology and data collection tools ● Organize data collection, including vehicles, trainings, accommodation, interviews, translation, etc. ● Collect and analyze data ● Conduct a presentation on preliminary findings for MC Program Management ● Write a report and address all feedback from MC, USDA and other stakeholders ● Produce other deliverables as described in the TOR USDA ● Review the TOR ● Participate in evaluation interviews as key informants ● Review the draft report and provide feedback ● Review and approve the final report MC HQ ● Support selection and contracting of the evaluation entity ● Process payments ● Resolve contractual issues, if any ● Review the final report MC CD ● Select the evaluation entity ● Obtain government permission for data collection ● Participate in evaluation interviews as a key informant ● Review the report and provide feedback on all drafts ● Approve payment requests MC MEL Manager ● Select the evaluation entity ● Oversee the evaluation process ● Provide all required program documents and other resources outlined in the TOR ● Provide technical support, if necessary and requested ● Participate in evaluation interviews as a key informant 146 ● Review the report and provide feedback on all drafts MC Program Management ● Participate in evaluation interviews as key informants ● Review the report and provide feedback on all drafts MC Program Staff ● Participate in evaluation interviews as key informants Stakeholders in Kyrgyzstan ● Participate in evaluation interviews as key informants ● Audience for the final report Save the Children ● Provide technical support with methodology development, if necessary (reading assessment) ● Participate in evaluation interviews as key informants ● Review the report and provide feedback on all drafts Annex H: Evaluation Criteria Mercy Corps Kyrgyzstan Selection Committee will conduct a technical evaluation which will grade technical criteria on a weighted basis (each criterion is given a percentage, all together equaling 100%). Award criteria shall be based on the proposal’s overall “value for money” (quality, cost, etc.) while taking into consideration donor and internal requirements and regulations. Each individual criterion has been assigned a weighting prior to the release of this tender based on its importance to Mercy Corps in this process. When performing the Scoring Evaluation, the Selection Committee will assign points for each criterion based on the following scale: Point Rationale 0 Not acceptable; has not met any part of the specified criteria 1-4 Has met only some minimum requirements and may not be acceptable 5 Acceptable 6-9 Acceptable; has met all requirements and exceeds some 10 Acceptable; has exceeded all requirements 147 Evaluation Criteria Max Points Possible Weight (%) Points X Weight (A) Price/Cost 10 20% 20 Resources: Labor, individual capacities, and the firm's ability to provide key personnel with the necessary qualifications 10 20% 20 Past Performance: demonstration of previous relevant experience, work completed in relevant countries, ability to produce necessary deliverables, ensured quality and management, and ability to deliver on time/scope/budget as evaluated by past performance 10 30% 30 Corporate Capabilities: organizational ability and capacity to organize work in-country 10 30% 30 TOTAL POSSIBLE SCORE: 40 100% 100 148 Annex I: Literacy Results The key research questions explored in this report include: 1) What literacy skills do children in grade 2 possess? What skills do they struggle with? 2) Do we observe any differences in children’s literacy skills by gender or class language? The focus of the Literacy Boost assessment is to measure children’s literacy skills and related factors in a holistic manner. First, a battery of subtasks measures the component skills of literacy complemented by information on the children’s background, including their socio-economic situation, home literacy environment, and other factors related to literacy development. This iteration of the Literacy Boost assessment tested children’s ability to read in both official languages of the Kyrgyz Republic: Kyrgyz and Russian. Children read a story in these languages aloud and their scores are analyzed to understand their reading fluency and accuracy. Additionally, the survey tool asks oral comprehension questions about the passage to gauge reading comprehension. In addition to this bilingual component the Literacy Boost assessment evaluates the subskills of reading in the children’s language of instruction as shown in Table 14 below. The subtasks include the children’s abilities to identify letters, read common words from their textbooks, decode invented words, comprehend an oral story, and respond to a picture vocabulary test. Children in Kyrgyz-language classes completed these tasks in Kyrgyz, and children in Russian-language classes completed these tasks in Russian. Table 14. Literacy Boost Literacy Subtasks Section Language Description Letter ID Language of instruction Children identify the letters of the alphabet. Most-used words Language of instruction Children attempt to read 20 common words from their textbooks Decodable words Language of instruction Children attempt to read 20 “invented” nonsense words based on the phonology of the language Listening comprehension Language of instruction Children listen to a story and respond to 4 comprehension questions Vocabulary Language of instruction Children identify images with vocabulary items pictured Reading passage 1 Kyrgyz Children read a short passage in Kyrgyz language and respond to 10 questions if they are able to read it Reading passage 2 Russian Children read a short passage in Russian language and respond to 10 questions if they are able to read it 149 The enumerators also asked children about their background characteristics (age, language, household possessions, reading materials, etc.), how they experience reading outside of school and their support from family members. Table 15 provides an overview to this section of the questionnaire. Table 15. Literacy Boost Background Questions Section Description Education Status Questions about their school attendance, educational background Home Environment Questions about the types of possessions children have at home (used to assess the socio-economic status) Print Material Questions about the types of reading materials children have access to Home Learning Activities Children list household members and the learning activities they engage in Community Learning Activities Children list out-of-school learning activities engaged in In order to understand the results, it is firstly important to understand the underlying statistics and makeup of the sample. The children surveyed make up a sample of 46 schools in 7 provinces. The children were around 8 years old and roughly evenly divided across gender. Table 16. Geographic Distribution of Sample Province Schools Children Batken 5 80 Chui 11 158 Issyk-Kul 1 16 Jalal-Abad 14 224 Naryn 1 16 Osh 10 160 Talas 4 64 TOTAL 46 718 150 Figure 6. Child Age by Gender The primary language of instruction was Kyrgyz with around 617 children in the sample relative to students who learn in Russian at 101 of the sample. This may present some difficulty when presenting statistics across the primary language of instruction. Figure 7. Child’s Language of Instruction 151 Results The results of the literacy test at the endline are as follows. First, we focus on the individual sub-skills that children were tested on using the language of instruction. As we can see in Figure 8, Grade 2 children demonstrate strong decoding skills (Letter Identification, Most-used Words, and Decodable Words) and reasonably strong language comprehension skills (Vocabulary and Listening Comprehension). These figures did not substantially change from the baseline to the endline, except for a 3 and 4 percentage point decrease in Decodable Words and Vocabulary respectively and a 7-percentage point gain in listening comprehension. Figure 8. Subtask Average Percentage Correct (n=718) While children demonstrated reasonably strong results in the subskills of reading, when it came to measuring reading comprehension, the story is mixed. As mentioned earlier, children read reading passages in both Kyrgyz and Russian. Children that were able to read at least five words of the passage within 30 seconds are considered “readers” (if they could not, they were deemed “non-readers”). After reading the passage, if children could successfully answer at least eight out of ten comprehension questions correctly, they were considered “readers with comprehension.” Figure 9 below presents a breakdown of students by these three statuses in the child’s class language. 152 Figure 9. Reading Status in the Language of Instruction (Baseline n=768, Endline n=718) The percentage of children unable to read at all is encouragingly low, with just 5% of students unable to read in the language of their class. In fact, a large majority (87%) of students were “readers” in both Kyrgyz and Russian. Despite most children being able to read the passages, relatively few could adequately comprehend the text. Where “reading with comprehension” is defined as the ability to read a grade-appropriate text and correctly answer at least 80% of 10 questions about the text. Fewer than four in ten students could read with comprehension in the language of instruction at the baseline, however this increased slightly by the endline, as seen in Figure 10 below where around 337 students (47%) were above the 80% threshold, while 381 students (53%) had scores beneath the threshold in reading comprehension, of these 5% did not make it through the first 5 words. For those students above the 80% threshold, most scored a perfect score of 100% (around 180) while the scores for children under the threshold decreased consistently. Figure 10. Reading Comprehension in Class Language (n=718) 153 Gender The results across gender show that on average, female students were 2.7 percentage points higher in most used words at the 99 percent level of confidence, and 3.1 percentage points higher in decodable words/invented words at the 90 percent level of confidence. No other changes were measured. Statistical analysis based on a two-tailed difference in means test. Table 17. Literacy Test Results by Gender Variable Male Female Diff Letter Identification 0.96 0.95 -0.006 Most Used Words 0.93 0.96 0.027** Invented Words 0.83 0.87 0.031* Listening Comprehension 0.59 0.58 -0.012 Vocab 0.93 0.94 0.003 N 332 386 718 Significance levels: * < 10% ** < 5% *** < 1% In literacy accuracy and reading comprehension, there were no statistically significant differences across gender in either language of instruction. Table 18. Literacy Accuracy and Reading Comprehension by Gender – Kyrgyz Male Female Diff Accuracy 0.94 0.95 0.009 Reading Comprehension 0.68 0.67 -0.011 N 310 370 680 Significance levels: * < 10% ** < 5% *** < 1% Table 19. Literacy Accuracy and Reading Comprehension by Gender – Russian Male Female Diff Accuracy 0.9 0.91 0.008 Reading Comprehension 0.47 0.48 0.008 N 282 348 630 Significance levels: * < 10% ** < 5% *** < 1% In the number of reading materials owned there were no statistically significant differences across student gender. 154 Table 20. Reading Materials by Gender Variable Male Female Diff Number of Reading Materials at Home 2.96 3.04 0.084 N 332 386 718 Significance levels: * < 10% ** < 5% *** < 1% Language of School Instruction In the language of school instruction, those learning in Russian had scores in letter identification that were 1.8 percentage points higher than those who learned in Kyrgyz. However, students who learn in Kyrgyz were nearly 10 percentage points higher in listening comprehension and 3.8 percentage points higher in Vocabulary relative to those who learn in Russian, statistically significant at the 99 percent level of confidence for both. Table 21. Literacy Test Results by Language of School Instruction Variable Kyrgyz Russian Diff Letter Identification 0.95 0.97 0.018* Most Used Words 0.95 0.96 0.019 Invented Words 0.85 0.86 0.011 Listening Comprehension 0.6 0.5 -0.098*** Vocabulary 0.94 0.91 -0.028*** N 617 101 718 Significance levels: * < 10% ** < 5% *** < 1% In literacy accuracy and reading comprehension there were some dramatically different and statistically significant differences between children who speak Kyrgyz and Russian in their schools. For instance, in reading comprehension in the Kyrgyz language, children who learn in classrooms that are taught in Kyrgyz had scores that were 24 percentage points higher in Kyrgyz reading comprehension than those who learn in Russian speaking classrooms. However, this was not the same for Russian language reading comprehension where they were roughly the same regardless of the language of school instruction. Table 22. Literacy Accuracy and Reading Comprehension in Language and Language of School Instruction Kyrgyz Language Accuracy and Reading Scores Russian Language Accuracy and Reading Scores Language Spoken in School Kyrgyz School Instruction Russian School Instruction Diff Kyrgyz School Instruction Russian School Instruction Diff 155 Accuracy 0.95 0.92 -0.020** 0.9 0.94 0.043*** Reading Comprehension 0.71 0.47 -0.241*** 0.47 0.51 0.045 N 580 100 680 530 100 630 Significance levels: * < 10% ** < 5% *** < 1% In reading materials there were not statistically differences between those who learn in Kyrgyz and Russian at school. Table 23. Reading Materials by Language of School Instruction Variable Kyrgyz Russian Diff Number of Reading Materials at Home 2.96 3.04 0.084 N 332 386 718 Significance levels: * < 10% ** < 5% *** < 1% Means by Provinces When observing the difference between the means (without the statistical difference in means) there are several small differences between provinces that deserve to be brought out in the final analysis. These include the near 5 percentage point difference between Osh and Talas in the letter identification task. The 3-percentage point difference between Osh and Talas, Batken and Naryn in the “Most Used Words” Category. In “Invented Words,” children in Talas scored an 80% at the lowest compared to children in Chui who scored about 88%. Table 24. Average Literacy Test Results by Province Batken Chui Issyk Kul Jalal Abad Naryn Osh Talas Letter Identification 0.949 0.954 0.960 0.963 0.939 0.928 0.974 Most Used Words 0.955 0.944 0.944 0.970 0.953 0.920 0.952 Invented Words 0.854 0.877 0.841 0.858 0.825 0.836 0.802 Listening Comprehension 0.475 0.554 0.453 0.761 0.500 0.439 0.563 Vocab 0.925 0.973 0.912 0.950 0.881 0.892 0.941 N 80 158 16 224 16 160 64 Significance levels: * < 10% ** < 5% *** < 1% 156 The highest reading comprehension in Kyrgyz among the surveyed provinces was in Jalal Abad with an average student score of 81.3 percent compared to the lowest scores in Osh at only 52.6 percent. When compared to Russian, the comprehension goes down dramatically. The highest scoring province was again Jalal Abad, but the lowest was Issyk Kul with only 25 percent and Osh at 27 percent with reading comprehension in Russian. The same dynamic is also the case in reading accuracy, although in Russian, the figures for reading accuracy are consistent and more tightly distributed between around 85 to 95 percent. Table 25. Literacy Accuracy and Reading Comprehension by province – Kyrgyz Batken Chui Issyk Kul Jalal Abad Naryn Osh Talas Reading Comprehension 0.621 0.688 0.600 0.813 0.573 0.526 0.654 Accuracy 0.944 0.952 0.952 0.938 0.964 0.942 0.923 N 78 147 14 218 15 149 59 Significance levels: * < 10% ** < 5% *** < 1% Table 26. Literary Accuracy and Reading Comprehension by Province – Russian Batken Chui Issyk Kul Jalal Abad Naryn Osh Talas Reading Comprehension 0.468 0.538 0.254 0.637 0.307 0.267 0.300 Accuracy 0.926 0.914 0.925 0.891 0.946 0.924 0.847 N 76 128 13 212 14 142 45 Significance levels: * < 10% ** < 5% *** < 1% However, children in Chui, Batken, and Naryn reported having 0.35 more reading materials in their home relative to Jalal Abad who reported only 2.7 reading materials in their home. Table 27. Average Number of Reading Materials by province Batken Chui Issyk Kul Jalal Abad Naryn Osh Talas Number of Reading Materials at Home 3.237 3.316 3.500 2.670 3.375 2.994 2.906 N 80 158 16 224 16 160 64 157 Annex J: Observation Assessment This section presents the results of the on-site observation assessment. The unit for all the following data are schools. Table 28. Sample Regions Region % No. of Schools Issyk-Kul 6% 4 Naryn 7% 5 Jalal-Abad 21% 15 Osh 20% 14 Chui 31% 22 Talas 7% 5 Batken 7% 5 Total 100% 70 Interventions Table 29. What Mercy Corps Interventions Took Place at the School? % No. of Schools Winterization (window or doors) 19% 13 Other Repairs (roof, floors, etc.) 16% 11 Water Filters 27% 19 Kitchen Rehabilitation 44% 31 Latrine Building or Repair 26% 18 Literacy Component 100% 70 Total 100% 70 158 Infrastructure Table 30. Schools that Received an Infrastructure Grant from Mercy Corps and Type of Grant % No. of Schools Window Replacement 19% 13 Door Replacement 13% 9 Heating System Installation or Repair 6% 4 Roof Repair 4% 3 Floor Repair 0% 0 Kitchen Ventilation 26% 18 Water Filters 24% 17 Handwashing Facilities 49% 34 Total 100% 70 Table 31. Are all windows in working order at the time of the visit? % (Total Sample) No. (Total Sample) % (Of schools that received window replacement grant) No. (Of schools that received window replacement grant) Yes 49% 34 100% 13 No 51% 36 0% 0 Total 100% 70 100% 13 Table 32. Were the doors installed (via grant from Mercy Corps) correctly? % No. of Schools Yes 87.5% 21 No 12.5% 3 Total 100% 24 159 Table 33. Are all doors in working order at the time of the visit? % (Total Sample) No. (Total Sample) % (Of schools that received door replacement grant) No. (Of schools that received door replacement grant) Yes 43% 30 100% 9 No 57% 40 0% 0 Total 100% 70 100% 9 Table 34. If the heating system was installed or repaired with a grant from Mercy Corps, is it in working order at the time of the visit? % No. of Schools Yes 25% 1 No 75% 3 Total 100% 4 Table 35. If the roof was repaired with a grant from Mercy Corps, have there been any issues with it since? % No. of Schools No 100% 3 Total 100% 3 Table 36. If a kitchen ventilation system was installed with a grant from Mercy Corps, have there been any issues with it? % No. of Schools No. 100% 18 Total 100% 18 Table 37. School Storage Room Conditions Condition % Yes No. of Schools Commodities stored less than 15 cm from the floor 97% 68 Commodities stored less than 15 cm from the wall 96% 67 160 Functioning ventilation system 83% 58 Working locks 79% 55 Shelves are unstable/commodities can fall off 9% 6 Signs of rodents inside the room 4% 3 Visible water damage on the walls/floor 3% 2 Damaged containers or bags (torn, leaky, etc.) on the shelves 1% 1 Signs of birds inside the room 1% 1 Bad smell (chemicals, spoiled food, mold, etc.) 1% 1 Signs of insects inside the room (ants, moth, etc.) 0% 0 Cracks in walls/floor/ceiling large enough for birds 0% 0 Spoiled produce or commodities on the shelves 0% 0 Chemicals, cleaning supplies on the shelves 0% 0 Equipment Table 38. Equipment Present at Sampled Schools Stove Oven Fridge Dishwashing Sink Dough Maker Present in the kitchen during the visit 100% 100% 93% 100% 80% Functioning during the visit 99% 97% 90% 100% 79% Has clearly visible MC/USDA logos 97% 96% 93% 100% 79% Has exploitation manual/instructions on it or nearby 99% 94% 90% 97% 80% Table 39. Are there enough chairs for all students in each group that eats at the same time? % No. of Schools Yes 97% 68 No 3% 2 Total 100% 70 161 Table 40. Are Mercy Corps visibility and program materials present? % No. of Schools Yes 97% 68 No 3% 2 Total 100% 70 Table 41.Handwashing Facilities in Dining Hall % No. of Schools Handwashing facilities (sinks) present in or near the dining hall 100% 70 Handwashing facilities had running water at the time of the visit 91% 64 Total 100% 70 Maintenance and Cook Behavior Table 42. Does all new donated equipment have inventory numbers to indicate that it was registered? % No. of Schools Yes 70% 49 No 30% 21 Total 100% 70 Additional context for the ‘no’ responses include responses that suggest that there were partial registration numbers on some of the equipment (5 responses) or a lack of registration numbers altogether for various reasons such as inventory numbers being erased (17 responses). Table 43. Does the school provide full hot meals five days per week? % No. of Schools Yes 89% 62 No 11% 8 Total 100% 70 162 Table 44. Kitchen Maintenance: Ventilation, Water Supply, and Cleaning Schedule % Yes No. of Schools Kitchen ventilation system present (not necessarily installed by MC) 89% 62 Kitchen ventilation system in working condition 86% 60 Constant water supply in the kitchen present 93% 65 Separate hand washing sink for the cook and other kitchen staff present 89% 62 A filled out and clearly visible kitchen cleaning schedule present 90% 63 Total 100% 70 Table 45. Kitchen Hygiene Behaviors During the observation visit did the following occur: % Yes No. of Schools Cook/kitchen staff or other people walk into the kitchen in street shoes (without shoe covers) 31% 22 Cook/kitchen staff or other people walk into the kitchen in street clothes (without aprons) 11% 8 Unauthorized personnel or children in the kitchen 4% 3 Total 100% 70 Latrines Table 46. Types of Latrines Used by Students % No. of Schools Latrines Inside the Building 31% 22 Outside Latrines 97% 68 Total 100% 70 163 Table 47. Student Latrine Conditions % No. of Schools Gender-segregated 99% 69 In working order at the time of the visit 90% 63 Within 2-3 meters of handwashing facilities with soap and water 50% 35 If not, there are indoor handwashing stations accessible to students near the latrines65 10% 7 Total 100% 70 Table 48. Conditions in Latrines Repaired or Built by Mercy Corps (at the time of observation) % No. of Schools Doors in each stall 78% 14 Locks on each door 61% 11 Functioning ventilation system 83% 15 People with disabilities accessibility provisions for outdoor latrines 22% 4 Total 100% 18 Table 49. Handwashing Facility Conditions % No. of Schools Accessible to primary grade students 99% 69 Accessible to students with physical disabilities 41% 29 Total 100% 70 65 Note that 27 schools did not have indoor latrines. 164 Table 50. Water Filtration % No. of Schools N Water filtration system present 27% 19 70 Water filtration functioning 95% 18 19 Water filtration system actively used 100% 19 19 Literacy / Reading Component Table 53. Reading Corner % No. of Schools N Is there a Reading Corner in the school? 99% 69 70 MC/USDA provided books in the Reading Corner* 94% 66 70 Reading Corner accessible to primary grade students 91% 64 70 Reading corner inaccessible because books are too high for little kids to reach 50% 3 6 Reading corner inaccessible because classroom/library was closed 67% 4 6 Note: *3 schools were n/a 165 Figure 11. Number of MC-Provided Books in the Reading Corner Table 51. Are additional books given to students to borrow and take home? % No. of Schools Yes 94% 63 No 6% 4 Total 100% 67 166 Bibliography McGovern-Dole International Food for Education and Child Nutrition Program – Kyrgyzstan 2017-2021 Midterm Evaluation Report, November 2019 Mercy Corps, Biannual Report 7 October 2020 - March 2021 Mercy Corps, Biannual Report 6 April 2020 - September 2020 Mercy Corps, Biannual Report 5 October 2019 – April 2020 Mercy Corps, Biannual Report 4 April 2019 – September 2019 Saldaña, J. (2016). The Coding Manual for Qualitative Researchers (3rd ed.). Thousand Oaks, CA: SAGE