IMPAQ International, LLC USDA McGovern-Dole International Food for Education and Child Nutrition Program Phase III in Mali Endline Evaluation Report August 2020 USDA McGovern-Dole International Food for Education and Child Nutrition Program Phase III in Mali Endline Evaluation Report September 2020 IMPAQ International, LLC Page i McGovern-Dole III Endline Evaluation Report SUBMITTED TO Catholic Relief Services 228 W Lexington St Baltimore, MD 21201 www.crs.org ATTENTION Edouard Nonguierma Mali McGovern-Dole III Chief of Party Edouard.nonguierma@crs.org Catholic Relief Services SUBMITTED BY IMPAQ International, LLC 10420 Little Patuxent Parkway Suite 300 Columbia, MD 21044 (443)256-5500 www.impaqint.com PROJECT McGovern-Dole International Food for Education and Child Nutrition III in Mali Agreement Number: FFE-688-2015/014-00 Funding Year: 2015 Project Duration: 2015-2020 DELIVERABLE Endline Evaluation Report AUTHORS Elnaz Safarha Vanessa Hoffman Daniel Zaas Michaela Gulemetova Adam Taube Tamarah Moss Jessica Li Kajal Gulati DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this report can be made available upon request. Please email FAS.Monitoring.Evaluaiton@usda.gov. IMPAQ International, LLC Page ii McGovern-Dole III Endline Evaluation Report Acknowledgements We thank the U.S. Department of Agriculture (USDA) and Catholic Relief Services (CRS) for their financial support. We extend special thanks to Edouard Nonguierma, Sylvain Guindo, Aly Kanoute, and their colleagues at CRS for their support and advice. Special thanks go to Kany Roseline Sidibe from CRS for all her support. Without her support and dedication, we would have not been able to make our contingency plan for this final evaluation work during the COVID-19 pandemic. We also thank Almougaraita Maiga from Education Development Center (EDC) for his technical insights and support in clarifying EDC data and reports. We also thank Bado Sissoko and Edoxi Kindané for their excellent work in supporting qualitative team at IMPAQ, conducting remote interviews, and collecting such high-quality data. Finally, we thank all the local stakeholders and beneficiaries who freely shared their perspectives and enabled us to have an in-depth analysis for the report. The findings, interpretations, and conclusions expressed in this report are entirely those of the authors. They do not necessarily represent the views of IMPAQ International, CRS, or USDA. IMPAQ International, LLC Page iii McGovern-Dole III Endline Evaluation Report Table of Contents Acknowledgements ............................................................................................................................. ii Table of Contents................................................................................................................................ iii Table of Exhibits .................................................................................................................................. v List of Acronyms ............................................................................................................................... viii Executive Summary ............................................................................................................................ ix Methods................................................................................................................................................... ix Key Findings .............................................................................................................................................. x Recommendations................................................................................................................................... xi Section 1. Introduction......................................................................................................................... 1 1.1 Overview of Program Implementation ...............................................................................................1 1.1.1 Sustainability................................................................................................................................2 1.2 Overview of Evaluation Background...................................................................................................3 1.2.1 Baseline and Midline Evaluations ................................................................................................3 1.2.2 Endline Evaluation........................................................................................................................4 Section 2. Evaluation Questions........................................................................................................... 6 Section 3. Evaluation Approach............................................................................................................ 7 3.1 Document Review...............................................................................................................................7 3.2 Remote Qualitative Data Collection ...................................................................................................9 3.2.1 Evaluation Sampling and Design..................................................................................................9 3.2.2 Ethical Considerations and Confidentiality ................................................................................10 3.2.3 Data Collection...........................................................................................................................10 3.2.4 Analysis ......................................................................................................................................12 3.3 Analysis of Secondary Quantitative Data..........................................................................................12 3.3.1 Data Sources ..............................................................................................................................13 3.3.2 Data Quality ...............................................................................................................................13 3.3.3 Analysis ......................................................................................................................................15 3.4 Triangulation of data.........................................................................................................................20 Section 4. Evaluation Results ............................................................................................................. 22 4.1 Relevance ..........................................................................................................................................23 4.1.1 Project Goals..............................................................................................................................23 4.1.2 Project Design ............................................................................................................................24 4.1.3 Fit with Community and Government Priorities........................................................................24 4.1.4 Participant Satisfaction ..............................................................................................................25 4.2 Efficiency ...........................................................................................................................................25 4.2.1 CRS Collaboration with Other Stakeholders..............................................................................25 4.2.2 Timeliness of Project Activities..................................................................................................26 4.2.3 Cost Effectiveness, Project Resource Management, and Food Supply Losses..........................26 4.3 Effectiveness.....................................................................................................................................26 4.3.1 Successes and Challenges of Project Implementation...............................................................26 4.3.2 Coordination and Management.................................................................................................29 4.4 Impact ...............................................................................................................................................29 4.4.1 Performance Evaluation.............................................................................................................30 4.4.2 Exploratory Impact Evaluation...................................................................................................77 IMPAQ International, LLC Page iv McGovern-Dole III Endline Evaluation Report 4.4.3 Perceived Impact .......................................................................................................................79 4.5 Sustainability.....................................................................................................................................81 4.5.1 Sustainability of Core Project Components...............................................................................81 4.5.2 Critical Factors for Sustainability ...............................................................................................84 Section 5. COVID Limitations on Implementation............................................................................... 88 Section 6. Conclusion ......................................................................................................................... 90 6.1 Limitations of the study ....................................................................................................................90 6.1.1 Qualitative Study Limitations.....................................................................................................90 6.1.2 Quantitative Study Limitations ..................................................................................................90 6.2 Key Outcomes and Implications for McGovern-Dole III results framwork .......................................91 6.3 Recommendations............................................................................................................................92 6.3.1 Best Practices with Respect to Current Activities......................................................................93 6.3.1 Sustainability and Future Programming ....................................................................................93 References......................................................................................................................................... 96 Appendices........................................................................................................................................ 98 Appendix A: McGovern-Dole III Mali Results Framework ...................................................................A1 Appendix B: McGovern-Dole III Activities...........................................................................................B1 Appendix C: Key Evaluation Questions............................................................................................... C1 Appendix D: Unreportable Outcomes ................................................................................................D1 Appendix E: Additional Tables............................................................................................................ E1 Appendix F: PMP McGovern-Dole III Indicators.................................................................................. F1 Appendix G: Qualitative Protocols .....................................................................................................G1 Appendix H: Terms of Reference........................................................................................................H1 IMPAQ International, LLC Page v McGovern-Dole III Endline Evaluation Report Table of Exhibits Exhibit 1. Summary of Main Findings ........................................................................................................... x Exhibit 2. Map of Targeted Region in Mali....................................................................................................1 Exhibit 3. Research Criteria and Data Collection Sources.............................................................................6 Exhibit 4. List of Documents Reviewed and Used for the Evaluation...........................................................8 Exhibit 5. Key Informant Interview Stakeholder Types.................................................................................9 Exhibit 6. Secondary Data Sources..............................................................................................................13 Exhibit 7. Data Quality Assessment ............................................................................................................14 Exhibit 8. Performance Indicators Measured and Reported in the Endline Evaluation .............................16 Exhibit 9. BLA Intervention and EGRA Data Collection Timing ...................................................................18 Exhibit 10. Data Triangulation Loop............................................................................................................20 Exhibit 11. Summary of Main Findings .......................................................................................................22 Exhibit 12. Data Sources .............................................................................................................................31 Exhibit 13. Teacher Sample by Region and Curriculum ..............................................................................31 Exhibit 14. SMC Sample by Region and Curriculum....................................................................................32 Exhibit 15. Average Number of SMC Members per School, by Region and Gender ..................................33 Exhibit 16. Classroom Observation Sample by Region and Curriculum......................................................34 Exhibit 17. SMC Members Who Say Teachers Are Often Absent ...............................................................35 Exhibit 18. Classroom Use of BLA Materials, by Year .................................................................................36 Exhibit 19. Percentage of Teachers Receiving Pre-Service Training by Year..............................................37 Exhibit 20. Teachers’ Opinions on Student Reading and Writing ...............................................................39 Exhibit 21. Teachers’ Opinions on Pedagogy ..............................................................................................40 Exhibit 22. School Principals Opinions on Pedagogy ..................................................................................40 Exhibit 23. Average Number of Hot Meals Distributed in December per School.......................................42 Exhibit 24. Average Number of Hot Meals Distributed in December per School, by Region.....................42 Exhibit 25. Average Number of Hot Meals Distributed in December per School, by Gender....................43 Exhibit 26. Average Number of School Days Without an Operational Canteen per School, by Region.....44 Exhibit 27. Data Sources .............................................................................................................................45 Exhibit 28. Average Number of Students who Received Take-Home Rations per School, by Region........46 Exhibit 29. Average Number of Students who Received Take-Home Rations per School, by Gender.......46 Exhibit 30. SILC Participation by Year .........................................................................................................47 Exhibit 31. Average Community Commodities Contributed in March per School, by Commodity............48 Exhibit 32. Average Percentage of Schools Received Community Commodities in March per School, by Region .........................................................................................................................................................48 Exhibit 33. Average Cash Contributed to Schools in January to March per School....................................49 Exhibit 34. Average Cash Contributed to Schools in January to March per School, by Region..................49 Exhibit 35. Average Annual Amount of Grain Contributed to Schools per School .....................................50 Exhibit 36. Average Annual Amount of Grain Contributed to Schools per School, by Region ...................50 Exhibit 37. Average Grain Contributed per Enrolled Student per School...................................................51 IMPAQ International, LLC Page vi McGovern-Dole III Endline Evaluation Report Exhibit 38. Average Grain Contributed per Enrolled Student per School, by Region.................................51 Exhibit 39. SMC Members Who Believe That Knowing How to Read is More Important for Boys than for Girls.............................................................................................................................................................53 Exhibit 40. Average School Enrollment, by Year by Region ........................................................................54 Exhibit 41. Association of Hot Meals Distributed at Schools and Enrollment by Gender, 2019 ................55 Exhibit 42. Association of Students who Received Take-Home Rations and Enrollment by Gender, 201855 Exhibit 43. Association of Deworming Pills Distributed at Schools and Enrollment by Gender, 2019.......56 Exhibit 44. Association of Vitamin A Pills Distributed at Schools by Enrollment and Gender, 2017..........56 Exhibit 45. Schools in the Student Attendance Sample..............................................................................56 Exhibit 46. Student Attendance Rate by Year by Region ............................................................................57 Exhibit 47. Proportion of Students Who Attended Regularly, by Year by Region ......................................58 Exhibit 48. Association of Hot Meals Distributed at School and Regular Attendance by Gender, January 2018 ............................................................................................................................................................59 Exhibit 49. Association of Hot Meals Distributed at Schools and Regular Attendance by Gender, January 2019 ............................................................................................................................................................59 Exhibit 50. Association of Students who Received Take-Home Rations and Regular Attendance by Gender, January 2018...............................................................................................................................................60 Exhibit 51. Association of Vitamin A Pills Distributed at School and Regular Attendance by Gender, October – December 2017........................................................................................................................................61 Exhibit 52. Association of Deworming Pills Distributed at School and Regular Attendance by Gender, January 2019...............................................................................................................................................61 Exhibit 53. EGRA Sample by Region and Curriculum ..................................................................................62 Exhibit 54. Student Literacy Sample by Grade and Gender........................................................................63 Exhibit 55. Decoding Proficiency by Grade and Region..............................................................................64 Exhibit 56. Decoding Proficiency by Gender...............................................................................................64 Exhibit 57. Decoding Proficiency by Curriculum.........................................................................................65 Exhibit 58. Number of Words Read from Passage by 1st Graders..............................................................66 Exhibit 59. Number of Words Read from Passage by 2nd Graders............................................................67 Exhibit 60. Students Meeting the Reading Proficiency Standard, by Region and Grade............................68 Exhibit 61. Reading Proficiency by Curriculum ...........................................................................................68 Exhibit 62. Reading Proficiency with Comprehension by Region and Grade .............................................69 Exhibit 63. Reading Proficiency with Comprehension by Curriculum ........................................................70 Exhibit 64. Grade 1 Performance on Foundational Reading Skills..............................................................72 Exhibit 65. Grade 2 Performance on Foundational Reading Skills..............................................................72 Exhibit 66. Average Number of Deworming Pills Distributed at Schools in October to December per School ....................................................................................................................................................................75 Exhibit 67. Average Number of Deworming Pills Distributed at Schools in October to December per School, by Gender....................................................................................................................................................75 Exhibit 68. Average Number of Vitamin A Pills Distributed at School in October to December per School ....................................................................................................................................................................76 Exhibit 69. Average Number of Vitamin A Doses Distributed at School in October to December per School, by Gender....................................................................................................................................................76 IMPAQ International, LLC Page vii McGovern-Dole III Endline Evaluation Report Exhibit 70. Average Number of Vitamin A Doses Distributed at School in October to December per School, by Region ....................................................................................................................................................77 Exhibit 71. Program Effects on Foundational Reading Skills for Grade 1 ...................................................78 Exhibit 72. Program Effects on Reading Proficiency for Grade 1................................................................79 Exhibit 73. McGovern-Dole Activities.........................................................................................................B1 Exhibit 74. McGovern-Dole III Activity’s Status ..........................................................................................B1 Exhibit 75. Unreportable Outcomes...........................................................................................................D1 Exhibit 77. Grade 1 Foundational Reading Skills Effects (𝜷), Same Schools (Overall, by Region, and by Gender) ....................................................................................................................................................... E1 Exhibit 78. Grade 1 Reading Proficiency Effects (𝜷), Same Schools (Overall, by Region, and by Gender). E1 Exhibit 76. McGovern-Dole III Indicators.................................................................................................... F1 IMPAQ International, LLC Page viii McGovern-Dole III Endline Evaluation Report List of Acronyms AE Learning Academy (Académie d’Enseignement) ASER Annual Status of Education Report BLA Balanced literacy approach CAP Education Animation Center (Centre d'Animation Pédagogique) CNCS National Center for School Canteens (Centre National des Cantines Scolaires) COVID-19 Coronavirus Disease 2019 CRS Catholic Relief Services EDC Education Development Center, Inc. ECOM Ecole Communautaire EGRA Early Grade Reading Assessment IFM Institut de Formation des Maîtres IPEG Institut Pédagogique d'Enseignement KII Key informant interview M&E Monitoring and evaluation McGovern-Dole McGovern-Dole International Food for Education and Child Nutrition MONE Ministry of National Education PDSEC Social, Economic, and Cultural Development Program (Plans de Développement Social, Economique, et Culturel) SARPE Stratégie Alternative de Recruitment du Personnel Enseignant SILC Savings and internal lending community SMC School management committee (Comité de gestion scolaire - CGS) SO Strategic objective THR Take-home ration USDA United States Department of Agriculture WASH Water, sanitation, and hygiene IMPAQ International, LLC Page ix McGovern-Dole III Endline Evaluation Report Executive Summary McGovern-Dole International Food for Education and Child Nutrition (McGovern-Dole) III is a five-year project in northern Mali (FY2016-FY2020), implemented by Catholic Relief Services (CRS) and funded by the United States Department of Agriculture (USDA). Launched in 2015, the McGovern-Dole III project aims to improve the literacy outcomes, as well as health and hygiene attitudes and practices, of 74,006 children in 291 schools in the regions of Mopti and Koulikoro through a variety of activities. CRS selected IMPAQ to conduct the endline evaluation of the McGovern-Dole III project using an evaluation design that was modified from the design for the baseline and midline evaluations due to the spread of the Coronavirus Disease 2019 (COVID-19) pandemic in spring 2020. This endline evaluation report assesses the relevance, efficiency, effectiveness, and impact of the project over the past five years in achieving its intended results. This report also provides recommendations on sustainable exit strategies and lessons learned for future implementation of any new McGovern-Dole phases. METHODS In March 2020, the spread of the COVID-19 pandemic and the resulting restrictions led to school closures. As a result, in consultation with Catholic Relief Services (CRS) and after USDA’s approval, IMPAQ designed an alternate plan to rely on secondary sources of quantitative data and remote qualitative data collection to implement the endline evaluation. Specifically, to address as many research questions (Appendix C provides details) as possible with the alternate plan within the five evaluation criteria (1) relevance, (2) efficiency, (3) effectiveness, (4) impact, and (5) sustainability, IMPAQ used the following approaches:  Document review. IMPAQ conducted an extensive review of all relevant project documents provided by the CRS and Education Development Center (EDC) teams.  Remote qualitative data collection. IMPAQ conducted key informant interviews (KIIs) remotely with USDA personnel, national stakeholders, project staff, and implementing partnerstaff, as well as with government administration and education officials. The remote KIIs extended as well to local-level stakeholders, including principals, teachers, school management committee (SMC) members, and savings and internal lending community (SILC) members. Under ordinary circumstances, we would have gathered information from local stakeholders, except for principals, in focus group discussions.  Quantitative analysis of secondary data. IMPAQ assessed the quality and relevance of existing data, identified the list of indicators that we could examine, and used the relevant data to analyze the performance of activities in relation to expected results.  Triangulation of quantitative and qualitative findings. IMPAQ integrated findings from the document review, remote interviews, and secondary data analysis to mitigate the limitations of each approach by providing contextual understanding and interpretation of the results. IMPAQ International, LLC Page x McGovern-Dole III Endline Evaluation Report KEY FINDINGS Using a mixed methods approach, we analyzed the data and presented them in five categories: relevance, efficiency, effectiveness, impact, and sustainability. The graphic below summarizes outcomes in those research areas to highlight the main takeaways. Section 4 provides details. Exhibit 1. Summary of Main Findings Relevance  Local and national stakeholders agreed that the project aligned with the priorities of their communities and government. They noted that the school gardens supported canteens and was appropriate for the culture of the community. In addition, the project matched government priorities to improve teaching quality, student attendance, and student literacy.  Local education officials and community stakeholders were satisfied with project trainings and responsiveness, in particular with the provision of take-home ration (THR) amidst school closures due to COVID restrictions, teachers’ strikes, and security concerns.  The project addressed barriers to education for girls, for example by providing sensitization to parents on the importance of education and organizing a reading competition for girls.  The project was adaptive and responsive to community environments, given external factors affecting project design such as school closures and security concerns. For example, students brought books home for continued learning, and the project started a tutoring activity with secondary students supporting primary students to improve literacy skills. Efficiency  CRS management of activities has been adaptive and responsive to community needs.  External factors such as school closures due to teacher strikes, COVID-19 and security concerns, created challenges for activity implementation and delivery of project materials. CRS adapted by repurposing commodities for school meals in storage at closed schools for distribution as THR.  CRS worked closely with local implementing partners, School management committees (SMCs), and local governments for timely implementation of project activities. Effectiveness  Parents are more invested in their children’s education as now they are able to monitor their progress in school using colored report cards.  Communities contribute to school canteens and enhance meals with school garden harvests.  Local education officials and community stakeholders recommended additional training for teachers because of teacher relocation and turnover to ensure all teachers have the same level of training. Impact Performance evaluation  Teachers’ literacy instruction knowledge has improved including their attitudes towards students’ learning and their use of balanced literacy approach (BLA) resources. Education officials and community stakeholders also found that BLA had a positive impact on improving literacy for school-age children.  Overall, we found gains among Grade 1 students, but did not for Grade 2 students.  External factors such as security concerns, teacher strikes, and COVID-19 presented challenges to consistent school attendance, teacher attendance, and students’ literacy outcomes.  Local and national stakeholders believed that school meals and THRs have a positive impact addressing student attendance, enrollment, retention, and hunger. The quantitative data also confirmed a positive association between such incentives and student enrollment and attendance rates.  Safe food preparation and storage training for SMC members have a positive effect on the meals produced by school canteens.  There is a persistent gap between the regions with Koulikoro outperforming Mopti in all literacy skills.  Project staff and SILC members said that SILC groups had a positive impact on communities in supporting household income generating activities and school canteens. Impact evaluation IMPAQ International, LLC Page xi McGovern-Dole III Endline Evaluation Report  One year of exposure to BLA is associated with significant increase in alphabet knowledge, decoding ability, and reading comprehension for Grade 1 students. Sustainability  All stakeholders were concerned about the ability of communities to assume ownership of project activities given the available resources. Both the continuation of BLA and the canteens may rely on the strength of community engagement. Stakeholders expected SILC groups to continue after the project ends but differed on whether SMCs would be functional and retain information from project trainings or have diminished capacity.  The project addressed IMPAQ’s recommendations toward sustainability of BLA.  Despite noting how CRS and partners prepared for transferring ownership to local community and government representatives, almost all stakeholders agreed that the ability to continue project activities rests upon how well communities can mobilize resources.  Strikes, insecurity, climate change, and the lack of water at schools all complicate lasting effectiveness and sustained operations.  Due to COVID-19, workshops and meetings intended to ease the handover of project activities to relevant community and government stakeholders were postponed or cancelled, which may affect sustainability. However, slightly more stakeholders at the community level felt that COVID has had limited effect on lasting project impact compared to respondents at the national and municipal levels as well as project and partner staff. Please refer to Section 5 for a discussion of COVID and any impacts of the pandemic on sustainability. RECOMMENDATIONS Based on our triangulation of findings, analysis, and lessons learned, IMPAQ developed the following recommendations to focus on the main drivers of project success, as well as on any changes required for future projects. We group our recommendations into two different categories: best practices and sustainability. Best Practices with Respect to Current Activities Continue collaboration and engagement with communities and local governments. Across different types of stakeholders, respondents praised the level of engagement and collaboration between CRS and the entities involved in coordinating and executing project activities. CRS worked with individuals at the local governance level to train them on monitoring school performance and taking greater initiative and direction in supporting children’s education. SMC members shared that they collaborated with local governments to promote project activities. Stakeholders recommended encouraging communities to include continuing support for project activities in their planning, such as in their Social, Economic, and Cultural Development Program (PDSEC). Increase sensitization on SILC support for schools. Stakeholders provided positive feedback on the establishment and operation of SILCs. They saw the SILC groups as an example of a good practice to continue moving forward. SILC groups empowered women to contribute to children’s needs and carry out projects for the community. Most SILC groups provided funds for members’ income-generating activities. However, there is room for more sensitization to motivate SILC groups to increase contributions to schools. Although SILC group members said that they provided funds or in-kind donations to canteens, other stakeholders said that only about half of SILC groups contributed to local school canteens. IMPAQ International, LLC Page xii McGovern-Dole III Endline Evaluation Report Recommendations for Sustainability and Future Programming Support the government to expand ownership of canteens and apply the school feeding law. Stakeholders identified the school feeding component as an impactful intervention that was critical in encouraging school attendance. Stakeholders believed that this successful strategy, which supports both food security and educational goals,should continue. CRS has successfully built buy-in and capacity among key actors for operation and sustainability including the state, local authorities, and school management committees. Given the strategy’s importance, future program design should consider continuing what CRS started to promote its sustainability after this project ends through tracking execution of the school feeding law. 1 Future program implementers should focus on scaling up and assessing and strengthening capacity for the state, local authorities and communities to open and operate school canteens in new regions across Mali. This could also include exchanges visits between project areas successfully operating canteens to share lessons learned and best practices with new areas in other regions planning to open canteens. Engage with communities to mobilize resources for school canteens. Respondents noted that community support to provide funds and food for the canteen was critical for sustainability. Although communities were able to mobilize for short periods in the face of shortages or commodity delivery delays, there were challenges with communities providing adequate quantities of food to canteens. Stakeholders reported success with school gardens to improve the quality of hot meals, but they also noted difficulty with cultivating collective fields to provide food for canteens. We suggest strengthening agricultural capacities in the communities so that households can produce and contribute more. This support could include assistance with water points and irrigation to promote successful school gardens to enhance school meals. Similar to the second phase of the project, this could include assistance with construction or maintenance of water points and irrigation for schools. Communities could also continue selling crops harvested during the summer to generate supplemental income for schools. In addition to promoting food production and capacity to contribute to canteens, we suggest sensitization to raise awareness about the benefits of the school canteen, such as how it provided students with nutritional meals at school and increased student attendance, to motivate community members to provide support. Responses from national government stakeholders emphasized the need for increased community capacity to provide food and resources for the canteen. 1 The School Feeding Law (Loi N˚2019-013) states that “the State, territorial collectivities, and communities support subsidies intended for school feeding” (Article 26, Title III, Chapter I). Additionally, “targeting, construction, and equipping canteens are the responsibility of the National Center for School Canteens” (Article 28, Title III, Chapter I). However, “territorial collectivities and communities must contribute to the facilitation and sustainability of school canteens” (Article 25, Title II, Chapter V of the Law). IMPAQ International, LLC Page xiii McGovern-Dole III Endline Evaluation Report Flexible program design and tailored activities to meet regional differences. Consistent with the findings from the midline evaluation, we found persistent regional differences in most of the outcomes during endline. We observed that changes in Mopti were limited compared to the Koulikoro region. For example, students in Koulikoro reported much higher levels in reading proficiency over time. This finding was triangulated and confirmed by interviews with local stakeholders. Political instability in Mopti caused multiple school closures over years during the project which affected the project’s outcomes. However, McGovern-Dole III has made multiple efforts to mitigate these challenges which should be either scaled up and/or continued in future program design and implementation. For example, the project added a tutoring program in 27 secondary schools in Mopti in December 2018. The tutoring program selected the best students to tutor younger students in first-cycle schools, and provided them with THRs in return. The project also started training principals and community-paid teachers to address teacher turnover in public schools. In addition, future programs could consider setting separate targets tailored to each region to deliver activities adapted to the regional context or consider a reallocation of resources to ensure both regions improve (e.g., more training in Mopti compared to Koulikoro). Although certain external factors that affect outcomes are more prevalent in Mopti, such as political instability and terrorist activities, future program design and implementation should be more flexible and tailored to deliver activities adapted to the regional context to ensure both regions improve. Promote BLA at the national government level and collaborate with other partners. Education officials reported having the capacity to provide training on BLA, and principals and teachers indicated willingness to continue applying BLA techniques. However, stakeholders emphasized that the government must support BLA by incorporating it in a dedicated section of its training module for teachers. This could include working with IFM and IPEG to include BLA in teacher training curriculums. This would allow for teachers to build skills and knowledge to use best practices from BLA techniques. Additionally, while BLA is not nationally mandated, along with this project, other schools across the country are using BLA as part of the Selective Integrated Reading Activity (SIRA) project funded by the U.S. Agency for International Development. Implementing partners for future programs should consider coordinating and collaborating with other US-funded education projects to exchange BLA lessons learned and enhance its reach. Promote regular teacher attendance in school. Preventing prolonged teacher strikes was beyond the scope of the current project. However, the endline results suggest that school closures over long periods of time lead to substantial reduction in instructional time (about 64 days during the 2018 – 2019 school year) which adversely affects student learning, even in the presence of promising programs such as BLA. The midline evaluation showed that students who were exposed for two or three years to teachers trained in BLA had significantly higher reading proficiency than students taught by teachers not trained in BLA. The endline evaluation, however, showed that Grade 2 students during 2018 – 2019 showed significantly lower reading skills than Grade 2 students two years earlier. Further interventions and incentives aimed at maintaining regular instructional time by encouraging high teacher attendance or promoting alternative instruction (e.g. via radio) will be beneficial for sustainability of the training efforts and for lasting effects on student learning. IMPAQ International, LLC Page xiv McGovern-Dole III Endline Evaluation Report Promote sustainability of activities within COVID restrictions. CRS faced school closures due to COVID, teacher strikes, or security concerns and adapted programming effectively. For school feeding, adaptation included continuing to distribute commodities as THRs rather than as hot meals. Regarding student literacy, implementing partners can continue to explore options for remote lessons, such as mobile libraries and providing funding for printed learning materials and USB drives (in case of having access to technology) for students and teachers to use at home. Future implementers could continue working with the Ministry of National Education (MONE) to ensure students have access to radios even in remote areas and providing radio-based lessons to mitigate the education loss caused by COVID. Moreover, if collecting data in person or remotely becomes feasible, future programs could conduct an assessment to understand how the applied mitigations strategies by the project affected various outcomes and use the findings to refine future design, given that no data have been collected after COVID. IMPAQ International, LLC Page 1 McGovern-Dole III Endline Evaluation Report Section 1. Introduction McGovern-Dole International Food for Education and Child Nutrition (McGovern-Dole) III is a five-year project in northern Mali, implemented by CRS and funded by the United States Department of Agriculture (USDA). Launched in 2015, the McGovern-Dole III project aims to improve the literacy outcomes, as well as health and hygiene attitudes and practices, of 74,006 children in 291 schools2 in the regions of Mopti and Koulikoro through a variety of activities. This report assesses the relevance, efficiency, effectiveness, and impact of the project in the past five years. Additionally, we provide recommendations on sustainable exitstrategies, as well as lessons learned for future implementation of any new McGovern-Dole phases. This section provides a brief overview of the program context for the McGovern-Dole III evaluation. Thereafter, Section 2 outlines the evaluation questions. Section 3 describes the endline evaluation approach. Section 4 presents the endline evaluation findings from our mixed-method approach. Section 5 explains in detail the limitations on the project implementation imposed by COVID-19. Finally, Section 6 concludes with lessons learned, the implications of the outcomes for the McGovern-Dole III results framework, the limitation of our study, and recommendations. 1.1 OVERVIEW OF PROGRAM IMPLEMENTATION In response to low school attendance and the recurrent food crisis in Mali, CRS has implemented USDA-funded McGovern-Dole school feeding projects with local partners and the Ministry of National Education (MONE) since 2007 (phase I, 2007 – 2010; phase II, 2011 – 2015; and phase III, 2015 – present). McGovern-Dole II reached 76,411 primary and secondary students in 311 schools and achieved positive results in enrollment and attendance rates, especially for girls, through provision of school meals and THRs. In addition, McGovern-Dole I and II focused on implementing activities such as establishing water points and school gardens; constructing latrines; and improving communities’ 2 This number includes 264 primary schools and 27 secondary schools, which were added to the project in December 2018 for tutoring program. In tutoring activity, the best students in secondary schools were selected to support younger students in primary schools in Mopti region. Exhibit 2. Map of Targeted Region in Mali IMPAQ International, LLC Page 2 McGovern-Dole III Endline Evaluation Report water, sanitation, and hygiene (WASH) knowledge and practices. Although enrollment and attendance improved during the first two phases of the project, the quality of education, particularly literacy in primary grades, still lagged. Therefore, in September 2015, USDA awarded CRS US$29.9 million to implement the third phase of the McGovern-Dole program, a five-year project (FY2016 – FY2020) to build on previous successes by improving the literacy, health, and nutrition of 74,006 school-aged children in 291 primary schools in the regions of Mopti and Koulikoro (Exhibit 2). CRS is leading the implementation of the third phase in partnership with Amprode Sahel, Caritas Bamako, Caritas Mopti, Education Development Center (EDC), and Guamina. CRS is working with EDC on educational quality and with the other four partners to implement activities in areas where they have a local community presence. Based on the McGovern-Dole III results framework (see Appendix A), the project aims to achieve two key strategic objectives (SOs) by the following theories of change.  SO1. Achieve higher literacy rates for school-aged children by improving literacy instruction in an environment in which students are enabled to participate actively in class and attend school consistently.  SO2. Improve use of health and nutrition practices with a focus on three key areas: safe food preparation knowledge and storage practices, access to preventive health interventions, and reduction of health-related absences. To achieve these objectives, McGovern-Dole III included elements of previous phases, including provision of school meals and THRs, distribution of vitamin A and deworming medication, and formation of savings and internal lending community (SILC) groups. In addition, the third phase expanded the scope of previous phases by including various activities such as capacity building of SMC members, expansion of illustrated report cards, provision of literacy materials to schools, and training for teachers and administrators on the balanced literacy approach (BLA). (See the full list of McGovern-Dole activities in all three phases in Appendix B.) In spring 2020, with the emergence of COVID-19 and its consequent restrictions, CRS suspended or modified some of the McGovern-Dole III activities for the last six months of project implementation. Appendix B shows the status of those activities at the end of the project during COVID-19: April to September 2020. 1.1.1 Sustainability Given that McGovern-Dole III represents the third phase of the project, CRS built into the project design several elements to promote sustainability. Per the sustainability plan and other CRS project documents, these activities include overarching goals to ensure the continuity of canteens by devolution to communities and local authorities and to make quality a priority at the community level. Specifically, one area of focus has been strengthening the capacities of SMCs and local municipal officials to manage canteens, enroll and retain children in school, and feel better equipped to engage with communities to mobilize support. In working with the SMCs, CRS has trained members on their roles and responsibilities, informed them of texts regulating the state and local authority funds allocated to schools, and enhanced their skills to design school-specific action plans and budgets. To encourage graduation and community IMPAQ International, LLC Page 3 McGovern-Dole III Endline Evaluation Report ownership, the project has offered a grant to fund action plans for SMCs attaining 70 percent of the graduation criteria, which the local mayor’s office will match. 3 Local municipal officials have also received training from school administrators on modules such as Democratic Governance; Participatory School Management; Monitoring and Evaluation; and refreshers in Food Preparation and Storage Practices and Commodity Management. The package of trainings provided to SMCs and local mayors build a common understanding of roles, help encourage collaboration to address community-level barriers, and ensure the application of funds for school action plans. In terms of quality at the community level, CRS has implemented activities such as advocating with local authorities, the Education Animation Center (CAP), and Learning Academies to include teacher training in the PDSEC and setting up a core of certified BLA trainers, which will improve early grade reading-writing scores. At the national level, efforts include transferring skills to school administrators and government officials to monitor the application of BLA, monitor progress via the Early Grade Reading Assessment (EGRA), and strengthen tool development. Lastly, a key component of the sustainability approach for McGovern-Dole III has been the continuation of SILC groups. Typically comprised of women, these SILC groups work together to save useful lump sums, take small loans, and access emergency social funds. CRS has requested members to set targets for canteen contributions to encourage SILCs to improve their capacity to manage and support school feeding. SILC groups are encouraged to focus on two to three highly nutritive crops to increase dietary diversity in school meals. 1.2 OVERVIEW OF EVALUATION BACKGROUND 1.2.1 Baseline and Midline Evaluations CRS selected IMPAQ International, LLC (IMPAQ) to design and conduct the impact and performance evaluations of McGovern-Dole III at baseline in 2016 and at midline in 2018. IMPAQ used quantitative and qualitative methods to establish baseline and midline values for outcome indicators required by USDA; refine targets for performance indicators; generate data for comparative analysis; and validate project strategies and assumptions at baseline and midline. IMPAQ designed the McGovern-Dole III evaluation with two components: 1. Performance evaluation. The performance evaluation aimed to measure changes in outcomes over the life of the project with regard to USDA’s two strategic objectives: SO1, improved 3 According to CRS’ sustainability plan: “By the end of the project, SMCs will graduate, which includes: defining and attaining community grain contribution (CRS estimates 45% of SMCs will attain 2 months or more and 55% will attain one month by Y5); providing compensation for cooks and recognition for storekeepers and ‘lunchroom’ attendants; providing condiments for sauce, cooking fuel, water, and utensils; storing commodities properly; organizing annual school enrollment campaigns; and obtaining and managing government resources.” IMPAQ International, LLC Page 4 McGovern-Dole III Endline Evaluation Report literacy of school-aged children, and SO2, increased use of health, nutrition, and dietary practices. 2. Impact evaluation. IMPAQ designed the impact evaluation to identify the causal effect of the BLA teacher training intervention on the literacy of primary school children. To conduct these two evaluation components at midline, IMPAQ collected survey data from students, parents or caregivers, teachers, principals, and SMC members, using the same instruments designed at baseline. We also assessed students reading proficiency level using the Annual Status of Education Report (ASER). In addition, IMPAQ collected qualitative key informant interview (KII) and focus group discussion data from national stakeholders, the project staff and implementing partners, local administrative and education officials, parents, and SMC members. At midline, local stakeholders (parents, SMC members, and local administration officials) agreed that the project aligned with the priorities of their communities. The quantitative program performance findings at midline also suggested that the project was moving toward the intended objectives with regard to the McGovern-Dole results framework (Appendix A), especially SO1, improving the literacy of school-aged children. Local stakeholders’ perceptions of the impact and effectiveness of McGovern-Dole activities confirmed the quantitative outcomes. However, project staff and local stakeholders pointed to similar challenges, including teacher turnover, political instability in the country, and stakeholders’ lack of capacity to take over after the project ends. More importantly, the lack of income-generating activities for parents was a critical issue for sustainability. Alongside performance evaluation findings, the impact evaluation provided strong evidence of causal impact of the BLA training on students’ literacy outcomes using a quasi-experimental methodology. Specifically, we used a cohort-comparison method to compare the midline outcomes of second- and third￾grade students with the baseline outcomes of peers when teachers had not received any training. The midline report showed significant positive impacts on students’ literacy outcomes after exposure to BLA￾trained teachers for two and three years. 1.2.2 Endline Evaluation In February 2020, CRS selected IMPAQ to conduct the endline evaluation of the McGovern-Dole III project. IMPAQ designed the endline evaluation with the following objectives:  Assess whether the project achieved its expected results with consideration of the theory of change, fidelity of implementation, and program management.  Build on baseline and midline studies to enhance learning and understanding of the project results, with an emphasis on project efficiency, effectiveness, impact, and sustainability.  Identify lessons learned, draw conclusions, and provide recommendations for future early grade literacy and food assistance programs. IMPAQ had initially planned to use an endline evaluation design similar to the baseline and midline evaluation designs. However, in March 2020, the spread of the COVID-19 pandemic and the resulting IMPAQ International, LLC Page 5 McGovern-Dole III Endline Evaluation Report health and safety restrictions led to school and border closures. As a result, collecting primary data to conduct the evaluation as originally planned was no longer a viable option. Thus, in consultation with CRS and with USDA’s approval, IMPAQ designed an endline evaluation that relied on secondary quantitative data and remote qualitative data collection. As a first step in designing the evaluation, IMPAQ conducted a short evaluability assessment. CRS shared all available data, reports, and other relevant documents with us. Upon close examination of the data and after a series of close consultations with the CRS team (including EDC) to address any gaps in our understanding of the information, we designed the endline evaluation. Section 3 describes our evaluation approach in detail. IMPAQ International, LLC Page 6 McGovern-Dole III Endline Evaluation Report Section 2. Evaluation Questions Following the McGovern-Dole III evaluation plan, IMPAQ assessed evaluation questions with the following evaluation criteria at endline: (1) relevance, (2) efficiency, (3) effectiveness, (4) impact, and (5) sustainability. The evaluation used existing secondary sources of quantitative data in addition to qualitative data collected remotely to address the evaluation questions listed in full in Appendix C. Exhibit 3 also lists their analysis methods within the evaluation criteria. Exhibit 3. Research Criteria and Data Collection Sources Inability to collect primary quantitative data and to conduct remote focus group discussions during the pandemic4 yielded challengesto fully answering some of the endline evaluation questions. To addressthis limitation, IMPAQ created a matrix to map available sources of data to each evaluation question and explored all possible avenues for data triangulation. We employed existing data creatively and relied on interviews with project staff, partners, and local education officials as a proxy for community-level focus groups and primary survey and assessment data. We also added a few more questions to capture the extent to which the COVID-19 pandemic affected the endline evaluation results and program implementation in the last six months of the project. Appendix C includes our evaluation questions matrix and designates whether we answered the evaluation question fully, addressed it partially with alternate options, or did not address it at all. 4 Unfortunately, poor connectivity via phone or online platforms prevented reaching stakeholders for in-depth focus groups. Criterion Data Sources Relevance Qualitative: KIIs with project and partner staff, national government representatives, education officials, mayors, principals, SMC and SILC members, and USDA officials Effectiveness Qualitative: KIIs with project and partner staff, national government representatives, education officials, mayors, principals, SMC and SILC members, and USDA staff Efficiency Qualitative: KIIs with project and partner staff, national government representatives, education officials, mayors, principals, SMC and SILC members, and USDA staff Impact Quantitative: Secondary data from CRS and EDC: early grade reading assessment, teacher survey, principal survey, classroom observations, SMC survey, and school attendance registers Qualitative: KIIs with project and partner staff, national government representatives, education officials, mayors, principals, SMC and SILC members, and USDA staff Sustainability Qualitative: KIIs with project and partner staff, national government representatives, education officials, mayors, principals, SMC and SILC members, and USDA staff IMPAQ International, LLC Page 7 McGovern-Dole III Endline Evaluation Report Section 3. Evaluation Approach IMPAQ conducted a short evaluability assessment using all available data, reports, and other relevant documents shared by CRS. Upon close examination of the data and after a series of close consultations with the CRS team (including EDC) to address any gaps in our understanding of the information, IMPAQ designed the endline evaluation. Although our approach differed from the approach used for the baseline and midline evaluations, we used the following alternate strategies to maintain rigor in addressing the evaluation questions:  Document review. IMPAQ conducted an extensive review of all relevant project documents provided by the CRS teams.  Remote qualitative data collection. IMPAQ conducted KIIs remotely with USDA staff, national stakeholders, project and implementing partner staff, and government administration and education officials. The remote KIIs extended as well to local-level stakeholders, including principals, teachers, SMC members, and SILC members. Under normal circumstances, we would have collected data from local stakeholdersincluding parents, except for principals, in focus group discussions.  Quantitative analysis of secondary data. We assessed the quality and relevance of existing data, identified the list of indicators that we could examine, and used relevant data to analyze the performance of activities in relation to expected results.  Triangulation of quantitative and qualitative findings. IMPAQ integrated findings from the document review, remote interviews, and secondary data analysis to mitigate the limitations of each approach by providing contextual understanding and interpretation of the results. This section describes each of these approaches in detail. Throughout this evaluation, we employed a rigorous mixed-method approach — from filling the gaps in quantitative data with qualitative data to triangulating quantitative and qualitative findings. This section describes each of these approaches in detail. 3.1 DOCUMENT REVIEW As a first step, the evaluation team reviewed the documents provided by the CRS team before conducting remote interviews and secondary data analysis. We reviewed these documents to see whether they were relevant to the evaluation questions and whether any of their information could support the qualitative and quantitative data analysis. During the review process, IMPAQ worked closely with the CRS and other project partners, particularly EDC, to understand the background of the existing data and reports. Exhibit 4 lists all the documents we used for our endline evaluation among those that we have received and reviewed. IMPAQ International, LLC Page 8 McGovern-Dole III Endline Evaluation Report Exhibit 4. List of Documents Reviewed and Used for the Evaluation5 Documents Year Published A. Reports 1. IMPAQ Baseline Evaluation Report September 2016 2. IMPAQ Midline Evaluation Report July 2018 3. CRS Performance Reports  CRS Mali McGovern-Dole Performance Report from FAIS 2015; Oct. 2017 - March 2018; April - Sept. 2018; Oct. 2019 - March 2020 4. Project Semi Annual Reports  Mali Semi-Annual Report 2018 April - Sept. 2018  Mali Semi-Annual Report 2019 April - Sept. 2019 5. Project Quarterly Reports  McGovern-Dole Quarterly Report April - Sept. 2016; Oct. 16 - March 17  Quarterly Activity Report (Region: Mopti) Oct. 2018 - Dec. 2018  Annual Activity Report (Region: Koulikoro) Oct. 2018 - Sept. 2019; April 2019 - June 2019; Jan. 2019 - June 2019  Quarterly Report (Quarter 3) McGovern-Dole 3 for the Region of Koulikoro and Mopti 2019 6. SILC Contribution to the Production of the USDA Half-Year Report 7. Stakeholders Workshop Report Feb. 2020 8. Annual EDC EGRA reports 2015-2019 B. Project Documents List of Key Informants for Endline Evaluation June 2020 C. Other Documents 9. List of Indicators and Performance Monitoring Plan 2015 10. SILC Reference Quarterly Reports 2016-2020 11. Sustainability Reference Documents 2016-2020 The document review served as the initial step in the qualitative analysis to help the evaluation team understand implementation, any changes in activities, and any reported challenges and obstacles faced during implementation. This information not only helped in the design of the interview guides, but also contributed to contextualizing the findings from interviews and filling information gaps where remote interviews lacked sufficient information. For quantitative data analysis, the documents formed the backbone of the performance and impact evaluations in the absence of new primary survey data from students, teachers and principals, parents, and SMC and SILC members. We examined the relevance of the secondary data to answer the evaluation 5 Although CRS team had provided other documents not mentioned in Exhibit 2, at this stage, the evaluation team plans on using the documents listed in Exhibit 2. If relevant, the evaluation would use other documents at a later stage. IMPAQ International, LLC Page 9 McGovern-Dole III Endline Evaluation Report questions related to performance and impact criteria, as well as to construct the performance indicators and other outcome indicators. In addition, we used CRS and EDC data reports to understand the timing and frequency of data collection and the data collection approach, including the selection of respondents, grades, and schools. During data analysis, we also used these documents to complement the qualitative and quantitative findings. During the document review phase, IMPAQ worked closely with the CRS and other project partners, particularly EDC, to understand the background of the existing data and reports. 3.2 REMOTE QUALITATIVE DATA COLLECTION 3.2.1 Evaluation Sampling and Design Due to COVID-19 pandemic restrictions, IMPAQ conducted KIIs remotely both online and by phone. These KIIs served to discuss the project’s accomplishments, design, sustainability, and participant satisfaction, as well as the working relationship between project staff and their partners, where appropriate. CRS facilitated coordination based on stakeholder types by providing their contact information. The evaluation team conducted KIIs with 37 stakeholders including relevant USDA staff members and representatives of CRS, implementing partner staff, government officials, and project stakeholders, as shown in Exhibit 5 below. Remote interviews occurred with stakeholders at national and local levels in Mopti and Koulikoro. The evaluation team attempted to interview an equal distribution of male and female respondents; however, while gender distribution was possible for SMC and SILC stakeholders, nearly all other stakeholders were male due to challenges in including female education officials, principals, and teachers with the remote approach. IMPAQ determined the remote fieldwork itinerary based on scheduling and the availability of KII participants. To facilitate remote interviews, a local consultant in Mali, with IMPAQ supervision and training, conducted in-country KIIs with local- and national-level stakeholders either by phone or on online platforms. Exhibit 5. Key Informant Interview Stakeholder Types Stakeholder Type Sample Size Potential Respondents U.S. government 2 USDA officials Project and implementing partner staff 8 CRS headquarters and in-country staff, EDC, Caritas Bamako, Guamina, Amprode, Caritas Mopti National government 2 MONE, National Center for School Canteens (CNCS) Mayors 2 Mayors in Mopti and Koulikoro Education officials 8 Director of Learning Academies (AE), canteen officer at AE, director of the Education Animation Center (CAP), education advisor Principals 4 Principals in two schools in Mopti and two schools in Koulikoro Teachers 4 Teachers in two schools in Mopti and two schools in Koulikoro IMPAQ International, LLC Page 10 McGovern-Dole III Endline Evaluation Report Stakeholder Type Sample Size Potential Respondents SMC members 4 SMC president and member in two schools in Mopti and two schools in Koulikoro SILC members 4 SILC members in two schools in Mopti and two schools in Koulikoro 3.2.2 Ethical Considerations and Confidentiality The evaluation team observed utmost confidentiality related to sensitive information and feedback elicited during the KIIs. To mitigate bias during the data collection process and give informants maximum freedom of expression, only the local consultant and limited evaluation team members were present during remote phone or online KIIs. The evaluation team respected the rights of participants in this evaluation. During this study, the evaluation team took several precautions to ensure the protection of respondents’ rights. 1. The evaluation team conducted remote KIIs by phone or online from a private area where no one else could hear. When scheduling interviews, the evaluation team advised the participant to be in a quiet place alone with good cell phone or internet connection. 2. No interview conducted without receipt of informed consent from the respondent. 3. The evaluation team talked with respondents to assess their ability to make autonomous decisions and their understanding of informed consent. Participants understood that they had the right to skip any question with which they were not comfortable or to stop at any time. 4. While CRS provided introductions before the interviews, when necessary, the evaluation team conducted KIIs with participants and stakeholders without the participation of any project staff. 5. Whenever possible and with the permission of the informants, we audio recorded the interviews only for the purpose of the study; the recordings will be destroyed once the analysis is completed. These recordings will be for the evaluation team only. We will not be share them with CRS or anyone else. 6. The evaluation team controlled written notes at all times by transmitting data electronically using secure file transfer protocols and stored notes in a secure server. 3.2.3 Data Collection Data Collection Preparation Prior to data collection, the IMPAQ qualitative lead held training and discussion sessions with the local consultant and qualitative team members. The IMPAQ qualitative lead worked with the local consultant to ensure the cultural appropriateness of data collection instruments and of the interviews themselves. The qualitative team drafted interview guides in English, translated the guides to French, provided cultural context review, and submitted to CRS for final approval. After the first week of interviews, the team met to discuss challenges, such as questions that confused respondents or not having time to cover all the IMPAQ International, LLC Page 11 McGovern-Dole III Endline Evaluation Report questions. These meetings continued each week throughout data collection to allow the team to adjust the instruments and strengthen team members’ interviewing and summarizing skills. Data Collection The qualitative team adhered to the following data collection protocols throughout the endline evaluation:  Interviews incorporated a degree of flexibility, allowing follow-up questions to capture any information relevant to the evaluation questions and domains.  The evaluation team followed a consistent data collection approach with each respondent, while allowing for limited variation according to cultural practices in each locality. For example, the local consultant translated interview guide questions from French to Bamanankan as needed for interviews. With oversight from IMPAQ and logistical support from CRS, the local consultant led the qualitative data collection effort focusing on four schools in Mopti and Koulikoro. IMPAQ worked closely with CRS to identify appropriate individuals for KIIs and organize the data collection. The qualitative researchers documented their progress daily — for example, the number of KIIs conducted and with whom — and noted any challenges encountered. The interviews followed a cascading method from national to local level, starting with interviews with USDA staff, then project and partner staff, then MONE officials and mayors, next education officials, then principals and teachers, and finally SMC and SILC members. The local consultant and other qualitative team members recorded KIIs, took detailed notes, and transcribed the recordings shortly afterward. In addition, the IMPAQ qualitative team communicated with the local consultant to check on progress, determine whether any adaptations to the schedule would be needed, and discuss whether to probe certain topics further. After receiving KII transcriptions, the qualitative team met to discuss insights and questions. Challenges Because of delays in the project evaluation kickoff between IMPAQ and CRS, data collection began in July. The interviews followed a cascading method from national to local level, starting with interviews with USDA staff, then project and partner staff, then MONE officials and mayors, next education officials, then principals and teachers, and finally SMC and SILC members. There were delays in scheduling interviews at the national government level given busy schedules, but after multiple efforts, we reached all stakeholders by the end of the data collection period. The remote approach also brought some other challenges. In a few cases, calls dropped during an interview because of poor connectivity. When this occurred, the local consultant could follow up to complete the interview. There was concern shared by one education official in Mopti with hesitation and initial reluctance to participate due to security concerns in the area, and not being able to build the same trust and rapport with a remote interview as opposed to one in-person. However, after further communication with CRS, the participant agreed to the interview without recording and at the end of the interview. IMPAQ International, LLC Page 12 McGovern-Dole III Endline Evaluation Report 3.2.4 Analysis Throughout the data collection, the IMPAQ qualitative team started translating the notes, received in French, into English. After receiving the first set of notes, the team and data collectors debriefed to:  Clarify any questions where respondents were confused  Discuss any challenges with remote interviewing, including any unanticipated issues or barriers related to COVID-19  Identify what topics and issues needed further probing  Determine how to adapt the guides in real time, if needed, to obtain more meaningful data  Ensure that the research team shared a common understanding and interpretation of the main points and themes Following this initial meeting, the qualitative team met on a weekly basis to continue discussing these points. After the data collection, the team reviewed and analyzed the debrief notes, supplemented by interview recordings, to identify recurring patterns pertaining to the five research domains (relevance, effectiveness, efficiency, impact, sustainability). The evaluation team used a thematic networking approach6 to capture salient themes for each research domain and noted any important similarities and key differences that may inform the quantitative results. 3.3 ANALYSIS OF SECONDARY QUANTITATIVE DATA To track project performance, IMPAQ worked with CRS to secure prompt access to secondary data consisting of available monitoring data and data from EDC related to the literacy component of the project. After gaining access to the data, the evaluation team conducted an evaluability assessment to examine the data quality and relevance in answering the evaluation questions. As part of the evaluability assessment, we also developed a list of relevant indicators that we could construct for the analysis. Using the information from this assessment, we designed our performance and impact evaluations at endline, which we describe in detail in Section 3.3.3. Next, we conducted a thorough data cleaning and quality checks, and finally analyzed the data for potential usage in both the performance and impact evaluations, in triangulation with qualitative analysis. 6 We conducted analysis using thematic networks: web-like illustrations (networks) that summarize the main themes in qualitative data. Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative Research. 1(3):385-405. IMPAQ International, LLC Page 13 McGovern-Dole III Endline Evaluation Report 3.3.1 Data Sources To conduct our secondary quantitative data analysis, we worked with the monitoring and literacy data outlined in Exhibit 6. Each dataset had a different frequency of collection across different years as we also show in Exhibit 6. Exhibit 6. Secondary Data Sources7 Name of Dataset Frequency of Data Collection Years of data available (end of school year) EDC Literacy Data Student EGRA dataset Yearly 2016-2019 Teacher dataset Yearly 2016-2019 School administrator dataset Yearly 2016-2019 SMC dataset Yearly 2016-2019 CRS Monitoring Data Attendance Monthlya 2017-2020 Deworming dataset Quarterly 2017-2020 Vitamin A dataset Biannual 2017-2018 Hot Meals dataset Monthly 2017-2020 Canteen dataset Monthly 2017-2020 Take-home Rations dataset Quarterly 2017-2019 Community Commodity Contributions dataset Monthly 2017-2020 Cash community contribution dataset 9 times per year 2017-2020 Grain community contribution dataset Monthly 2017-2020 Colored report card dataset Monthly 2017-2020 Source: CRS monitoring data a Attendance data was collected for October-June in 2016-2017, October-April in 2017-2018, October-July in 2018-2019, and October – January in 2019-2020. 3.3.2 Data Quality Before starting the data analysis, IMPAQ conducted an in-depth quality assessment on any data that we mapped to evaluation questions. Exhibit 7 describes some issues we encountered in the CRS monitoring data for project activities and EDC data and how we addressed them for the analysis. 7 While the frequency of data collection for CRS monitoring data is shown above, the datasets were presented to the evaluation team as follows: the Vitamin A data was shown 3 times per year, the Hot Meals data was shown 10 times per year, the Canteen data was shown yearly, the Take-home Rations data was shown 9 times per year, the Community Commodity Contributions data was shown 9 times per year, the Colored report card data was shown 11 times per year. IMPAQ International, LLC Page 14 McGovern-Dole III Endline Evaluation Report Exhibit 7. Data Quality Assessment Data Issue Remedy Duplicate entries The above listed CRS monitoring data included several duplicate school entries across the monitoring datasets that we used for the analysis. For each dataset, we first investigated cases when the same school appeared more than once in a particular year. Then, we merged each monitoring dataset with a master list, containing a full list of targeted school communities and regions information to resolve the duplicate schools. When merging the master list with each monitoring dataset, we excluded any duplicate schools whose community and region names from the dataset sample did not matched the master list, assuming they are out of the targeted sample. If the community and region of a duplicate school matched the master list of schools, then we considered it to be a valid school and looked further in the data to resolve the issue. Next, we checked the duplicate entriesto see if they are complementary (i.e., two records together provide complete information for one school). In such cases, we combined them into one school entry. Otherwise, we excluded both records from the analysis because we could not determine the true information for that school in that year. For example, the community commodity contributions had 23 duplicate cases, which did not have unique school identification numbers, and each had a single duplicate for a given year. Following the approach explained above, we dropped nine duplicates after merging the dataset with the master list based on mismatched school name and geographic information. However, for the remaining cases, the duplicate records were each combined into a single entry as they had identical information for the school’s name and geography, as well as complimentary information for the volumes of community contributions. EDC used multiple observers in each classroom, so there were several observations for each class. We randomly selected one observation to keep. Missing data The CRS monitoring data had missing information for some schools across the years. In other words, not every school was in each monitoring dataset for all years. There did not appear to be a pattern associated with the missing data, nor could they be linked to particular events. EDC aimed to survey the same 80 schools each year, however, their staff were unable to reach every school every year. In such cases, they surveyed replacement schools. This process led to an inconsistent sample of schools over time. For the performance evaluation, we kept only schools for which we had data at the beginning and the end of the project. For the exploratory impact evaluation, we conducted the regression analysis on the full sample. As a robustness check, we also ran the regression analysis on restricted samples including only schools with data for both time periods. Coverage over years of collected data CRS collected their monitoring data with highly varying frequency and consistency over time. Some monitoring datasets presented monthly data for every year with as many as 11 months out of the year while others had only 9 months or fewer. Other monitoring datasets had only annual data but no monthly data. Two monitoring datasets featured quarterly data only. Depending on the quality and consistency of the school data, we aggregated school information into quarterly or yearly totals or averages to allow for meaningful comparison over time. EDC collected data at the beginning of the project in December 2015 and then consistently at the end of each school year after that. IMPAQ International, LLC Page 15 McGovern-Dole III Endline Evaluation Report Data Issue Remedy Data consistency for each instrument over years The CRS monitoring data included several schools that had only information for a single year across the provided datasets. These were not necessarily the same schools across different monitoring datasets. We excluded these schools from the analysis, as we could not measure the changes in their performance indicator over time. For most of the monitoring datasets, fewer than 3% of observations were excluded; for the cash contributions dataset, we had to drop 15% of the sample. The EDC data had similar inconsistencies in the survey instrument over time. In some cases, they measured variables differently from year to year. To the extent possible, we reconciled these discrepancies to report on the outcomes. For example, teachers’ beliefs about students’ learning was recorded on a scale of 1-5 (1 being strongly agree, 2 somewhat agree, 3 neutral, 4 somewhat disagree, and 5 strongly is agree) for the first few years of data collection but later changed to a simple “agree or disagree” statement. IMPAQ reconciled this inconsistency by aggregating the five categories into to agreement (somewhat agree or strongly agree) or disagreement (neutral, somewhat disagree, or strongly disagree). Additionally, where relevant, we compiled multiple data sources into a master file for analysis. We adjusted the data for consistency across different datasets and examined the frequency distributions for each variable to ensure that all data were within a valid range. As the last step, we further cleaned the data, applied acceptable techniques to address missing observations (e.g., imputation, deletion), and created additional outcome variables to address the evaluation questions listed in Appendix C. 3.3.3 Analysis McGovern-Dole III was not able to collect the final monitoring data for all indicators in year 2019 – 2020 due to COVID-19. Thus, the endline evaluation analysis is limited to understanding the project’s achievements using the latest data available. For the performance evaluation, we constructed a reduced set of performance indicators based on available secondary data, performed descriptive analysis by comparing averages and percentages between different years, conducted t-tests, and ran simple regression analysis to analyze the correlation of different project interventions in the achievement of key project outcomes. For the impact evaluation, we focused on literacy outcomes and performed exploratory regression analyses. Importantly, although this design varied from the impact design used at baseline and midline and we were not able to assign causality to the observed changes in literacy, we still examined the influence of BLA training in improving Grade 1 literacy. Performance Evaluation Design To understand the key project achievements, we performed descriptive analysis by computing averages and percentages between different yearsfor a reduced set of performance indicators related to outcomes IMPAQ International, LLC Page 16 McGovern-Dole III Endline Evaluation Report ofstudents, teachers, principals, and SMCs and other school-related activities.8 We used this methodology relying on secondary data to assess and quantify the project’s performance by tracking changes in outcomes, measured in the same manner over the course of the project. We used clustered t-tests to examine whether there were systematic differences between sub-groups, such as by region, gender, or school curriculum. In addition to descriptive analysis and t-tests, we also used simple regression models to explore how grade-level attendance and enrollment were associated with distribution of hot meals, THRs, vitamin A supplements, and deworming pills. The descripting analysis can only suggest correlations between the observed changes in outcomes and the project’sinterventions. Since we did not collect the project data from a random representative sample of beneficiaries and we could not select a new sample of individuals at each monitoring data collection point, we cannot conclusively determine whether the interventions caused the changes. For the endline performance evaluation, we looked into two set of indicators and outcomes. First, we examined all possible avenues to rigorously address the McGovern-Dole evaluation indicators required by the approved performance monitoring plan. However, due to the limitations of the secondary data, we were able to report only on a limited set of evaluation indicators. For indicators we were not able to address quantitatively, we relied on qualitative data to provide contextual information. Exhibit 8 lists the evaluation indicators that we measured and reported with available data sources at endline. Those indicators that we were not able to collect using secondary data were marked N/A. Exhibit 8. Performance Indicators Measured and Reported in the Endline Evaluation Indicator Revised Indicator Data Source Disaggregated By Percentage of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text No revision EDC EGRA data Gender, region, grade, curriculum Percentage of students who demonstrate decoding abilities No revision EDC EGRA data Gender, region, grade, curriculum Percentage of students who reach the national reading standards by the end of the year No revision EDC EGRA data, KIIs with education officials Gender, region, grade, curriculum Average number of days missed per student per school year due to student health issues No revision CRS monitoring attendance data, KIIs with all stakeholders Gender, region, grade, curriculum Percentage of school-aged children enrolled in school No revision CRS monitoring attendance data, KIIs with all stakeholders Gender, region, grade, curriculum 8 This analysis was limited as we lacked data to understand the effects on some outcomes, as mentioned in Appendix E. IMPAQ International, LLC Page 17 McGovern-Dole III Endline Evaluation Report Indicator Revised Indicator Data Source Disaggregated By Percentage of community members demonstrating knowledge of educational benefits N/A KIIs with SMC and SILC members N/A In addition to the key indicators, to understand the main project achievements, we looked at all the reported outcomes at midline within the McGovern-Dole results framework. We reported on those outcomes that we were able to address using secondary data; where we could not, we constructed an alternate indicator as a proxy that could serve as the second-best option. Where possible, we also filled quantitative data gaps with qualitative data analysis. However, there were still some outcomes and indicators measured at baseline and midline that we were not able to construct. For example, no monitoring data were available to answer evaluation questions related to parents and SILC members. Even with the available secondary data or qualitative notes, we lacked information on hunger, and observations on school infrastructure such as food storage and handwashing stations; see Appendix D for this list. Relevance to Baseline and Midline Evaluation Design During baseline and midline, we designed the performance evaluation with a pre-determined sample size relevant to the context of monitoring activities implemented by the CRS team and project partners. For the endline performance evaluation, we had to rely on monitoring data collected for monitoring and management purposes and literacy data collected for internal evaluation not for final evaluation. We list two key caveats about use of secondary data for the performance evaluation: 1. Different sets of data were collected for different schools, and therefore our comparison did not necessarily use the same set of schools. For example, EDC literacy-related data are available for approximately 80 schools, and the monitoring data are available for nearly all schools. 2. Because we used monitoring data, our analysis focused on understanding differences in implementation by region and by gender. However, we were limited in constructing outcomes related to the effects of implementation of all activities. For example, the data allowed us to examine participation of SILC members over the years. However, it did not allow us to analyze how their participation affected their livelihood, as we did for the midline evaluation. Exploratory Impact Evaluation Design The design for the exploratory impact evaluation relies on the secondary data collected by EDC during 2016 – 2019 related to students, their teachers, and schools. EDC collected EGRA data from randomly selected students from approximately 80 schools each year during this period. We combined these data with the schedule of the BLA training to understand whether exposure to BLA package (i.e., teacher professional development training and ongoing support to teachers from pedagogical counselors) improved students’ literacy outcomes. Specifically, we exploited the variation in students’ exposure to BLA package within a school to measure the program effect on students’ literacy outcomes. We used this design in accordance with the BLA training schedule and the timing of EGRA data collection over the years. IMPAQ International, LLC Page 18 McGovern-Dole III Endline Evaluation Report We focused on EGRA in Grades 1 and 2 to allow us to use at least two years of literacy data for each grade, measured at about the same time: in June, corresponding to the end of the school year. This consistency is essential to provide with meaningful comparisons over time of literacy levels at the same point in the school year. Specifically, according to the BLA implementation plan, in the first year, 2015 – 2016, the first cohort of Grade 1 teachers received initial BLA training in December 2015 and ongoing support from pedagogical counselors for the rest of the school year. Reading scores measured by EGRA were collected from Grade 1 students at the end of the same school year, in June 2016, reflecting half a year of exposure to the BLA techniques. In the second year, 2016 – 2017, these Grade 1 teachers became Grade 2 teachers and received additional refresher training in December 2016. At the end of that school year, EDC collected EGRA data from the Grade 2 students, reflecting 1.5 years of cumulative exposure to BLA. In the third year, 2017 – 2018, a new cohort of Grade 1 teachers received BLA package (an initial training in September and a follow-up training December together with a full year of pedagogical support). EDC collected EGRA data from Grade 1 students at the end of the school year, in June 2018, reflecting a full year of exposure to the BLA intervention. In the fourth school year, 2018 – 2019, the Grade 1 teachers followed their students into Grade 2 and received additional refresher training in December 2018. In June 2019, they collected EGRA data from Grade 2 students who had two full years of exposure to the BLA intervention. Exhibit 9 shows the timing of the BLA intervention and EGRA data collection each year as a visual representation of the exploratory impact evaluation design. We are depicting the initial BLA trainings as indication of when the intervention began for each cohort of teachers. Blue bubbles correspond to Grade 1 and green bubbles correspond to Grade 2. Exhibit 9. BLA Intervention and EGRA Data Collection Timing While we did not have access to baseline data from the same schools prior to the BLA roll out or access to comparison group of similar schools who did not receive the program to construct a valid counterfactual, we exploited the variation in the length of BLA exposure of two separate cohorts of teachers and their students. In other words, by comparing EGRA assessments for Grade 1 students in June 2016 with EGRA assessments for Grade 1 students in June 2018 (the blue bubbles), we tested in an exploratory regression analysis whether the additional exposure to the BLA intervention is associated with IMPAQ International, LLC Page 19 McGovern-Dole III Endline Evaluation Report higher Grade 1 reading outcomes. This exploratory analysis assumes that the sample of students with EGRA scores was representative of all beneficiary students in each year and that there were no major changes in school performance such as curriculum reforms or other national education policies during these years.9 Specifically, we estimated the following regression equation: 𝑌𝑖𝑠𝑡 = 𝛽𝒓𝑡 + 𝛼𝑿𝑖𝑠𝑡 + 𝛾𝒁𝑠𝑡 + 𝜇𝑠 + 𝜀𝑖𝑠𝑡 In the equation above, 𝑌𝑖𝑠𝑡 refers to literacy outcome of Grade 1 student i attending schools in year t. 𝒓𝑡 measures the duration of exposure to BLA in year t. 𝑿𝑖𝑠𝑡 refers to student gender and age, and 𝒁𝑠𝑡 captures school characteristics such as region and language of curriculum. 𝜇𝑠 captures fixed observed and unobserved school determinants of performance. The coefficient of interest is 𝛽, which captures the change in Grade 1 students’ literacy outcomes due to one year of exposure to BLA-trained teachers.10 In addition to the full sample of Grade 1 data, we implemented a sub-group analysis for sub-samples by gender and region to examine any potential effects.11 This exploratory impact evaluation approach seems to be best suited to identifying the impact of exposure to BLA among Grade 1 students given the available data. However, our data may be limited in identifying any effects if sufficient variation does not exist within in a school or the sample is too small. As some schools were replaced during the years, we also ran the exploratory regression analyses, as robustness checks, on a restricted sample of schools that provided at least two years of data. Relevance to Baseline and Midline Evaluation Design At baseline and midline, the impact evaluation used a cohort-comparison quasi-experimental design. At midline, the design allowed us to measure two- and three-year impacts on children of exposure to BLA￾trained teachers. The analysis showed that children’s literacy outcomes improved by one level on ASER reading assessment due to two years of exposure to a BLA-trained teacher and these outcomes increased by more than one level and a half after three years of exposure to BLA-trained teachers.12 9 We also compared Grade 2 EGRA assessments in 2017 with Grade 2 EGRA assessments in 2019 (the green bubbles), however, this analysis was problematic. The 2018 – 2019 teacher strikes led to extensive school disruptions which reduced to total length of real exposure to the BLA intervention of the second cohort of Grade 2 students. We are, therefore, unable with this exploratory impact analysis to disentangle the BLA effect in boosting literacy from the negative effect of school disruptions. 10 We clustered our standard errors at the school level to account for any correlation across the error terms. 11 We also performed sub-group analysis by language of curriculum to explore differential program effect. 12 One level improvement on ASER reading assessment can be interpreted as an average student moving from comfortably reading simple sounds (level B) to reading complex sounds (level C) and one level and a half can be understood as a student moving to decoding simple 1-2 syllabus words (level D). For more information, see IMPAQ Midline Evaluation Report. IMPAQ International, LLC Page 20 McGovern-Dole III Endline Evaluation Report This evaluation design is different from the original cohort-comparison design because we are unable to measure the two-, three-, and four-year impacts of exposure to BLA-trained teachers on students. However, we can measure the change in Grade 1 students’ literacy outcomes due to one year of exposure to BLA. A key limitation of this exploratory design is that the estimated program effects here may be confounded by time effects, either exaggerating it or understating it. In the presence of adverse time effects such as prolonged national teacher strikes leading to school disruptions, the estimated overall effect can be null or even negative. This limitation does not suggest that BLA training was ineffective. In fact, the findings at midline, which showed that exposure to BLA for two and three years led to improvements in children’s literacy outcomes, are still valid. We want to emphasize that this endline exploratory impact evaluation approach serves only to complement the findings at midline within the context of its limitations. That is, the design can only contribute to confirming the positive impacts of BLA training found at midline, as it relies on different samples from the same beneficiary population. Moreover, this evaluation relies on EGRA data while the midline evaluation relied on ASER data, making the comparison of effects difficult. However, the endline evaluation bolsters our findings at midline about the program’s impact and offers the second-best proof of program impacts, in the absence of primary data collection. 3.4 TRIANGULATION OF DATA To enhance the rigor of the endline evaluation and mitigate the limitations of our contingency plan in using each of the approaches described in this section, we integrated the data and our findings throughout the evaluation in a learning loop. Exhibit 10 shows the cycle of data triangulation. Exhibit 10. Data Triangulation Loop IMPAQ International, LLC Page 21 McGovern-Dole III Endline Evaluation Report The evaluation team held regular meetings to share findings from the document review, secondary data analysis, and KIIs to design the endline evaluation, plan for analysis, and compare the findings from different data sources and the three approaches. For example, document review and initial secondary data analysis findings helped inform interview guide development; we included questions in the guides to address limitations in secondary data analysis to the extent possible. Similarly, document review helped us modify the endline analysis plan, and enabled use of the secondary data to the fullest extent possible. During the analysis phase, document review and initial findings from KIIs provided context for secondary data analysis findings and vice versa. Moreover, we used both quantitative and qualitative insights to ensure that our recommendations are robust and reflect the evidence we found in the performance and impact evaluations, especially in light of the limitations that the COVID-19 pandemic imposed on the endline evaluation. IMPAQ International, LLC Page 22 McGovern-Dole III Endline Evaluation Report Section 4. Evaluation Results Following the organization of evaluation questions in Appendix C and with respect to the evaluation criteria, we have broken down the presentation of results into the categories of relevance, efficiency, effectiveness, impact, and sustainability within the McGovern-Dole results framework. Exhibit 11 summarizes the salient themesin each category. The rest of this section provides details for each category. Exhibit 11. Summary of Main Findings Relevance  Local and national stakeholders agreed that the project aligned with the priorities of their communities and government. They noted that the school gardens supported canteens and was appropriate for the culture of the community. In addition, the project matched government priorities to improve teaching quality, student attendance, and student literacy.  Local education officials and community stakeholders were satisfied with project trainings and responsiveness, in particular with the provision of THR amidst school closures due to COVID restrictions, teachers’ strikes, and security concerns.  The project addressed barriers to education for girls, for example by providing sensitization to parents on the importance of education.  The project was reported as adaptive and responsive to community environments, given external factors affecting project design such as school closures and security concerns. For example, students brought books home for continued learning. Efficiency  CRS management of activities has been adaptive and responsive to community needs.  External factors such as school closures due to teacher strikes and security concerns, created challenges for activity implementation and delivery of project materials. CRS adapted by repurposing commodities for school meals in storage at closed schools for distribution as THR.  CRS worked closely with local implementing partners, SMCs, and local governments for timely implementation of project activities. Effectiveness  Parents are more invested in their children’s education as now they are able to monitor their progress in school using colored report cards.  Communities contribute to school canteens and enhance meals with school garden harvests.  Local education officials and community stakeholders recommended additional training for teachers because of teacher relocation and turnover to ensure all teachers have the same level of training. Impact Performance evaluation  Qualitative findings show that teachers’ literacy instruction knowledge has improved including their attitudes towards students’ learning and their use of BLA resources. Education officials and community stakeholders also perceived that BLA had a positive impact on improving literacy for school-age children.  Overall, we found gains among Grade 1 students, but did not for Grade 2 students. External factors such as security concerns, teacher strikes, and COVID-19 presented challenges to consistent school attendance, teacher attendance, and students’ literacy outcomes.  Local and national stakeholders believed that school meals and THRs have a positive impact addressing student attendance, enrollment, retention, and hunger. The quantitative data also confirmed a positive association between such incentives and student enrollment and attendance rates.  Safe food preparation and storage training for SMC members have a positive effect on the meals produced by school canteens.  There is a persistent gap between the regions with Koulikoro outperforming Mopti in all literacy skills. IMPAQ International, LLC Page 23 McGovern-Dole III Endline Evaluation Report  Project staff and SILC members said that SILC groups had a positive impact on communities in supporting household income generating activities and school canteens. Impact evaluation  One year of exposure to a BLA-trained teacher is associated with significant increase in alphabet knowledge, decoding ability, and reading comprehension for Grade 1 students and even larger significant increase in reading comprehension for Grade 2 students. However, negative significant findings in oral reading fluency and in listening comprehension among Grade 2 students suggest that students lost foundational reading skills likely due to extensive school disruptions during the 2018 – 2019 teacher strikes. Sustainability  All stakeholders were concerned about the ability of communities to assume ownership of project activities given the available resources. Both the continuation of BLA and the canteens may rely on the strength of community engagement. Stakeholders expected SILC groups to continue after the project ends but differed on whether SMCs would be functional and retain information from project trainings or have diminished capacity.  The project addressed IMPAQ’s recommendations toward sustainability of BLA.  Despite noting how CRS and partners prepared for transferring ownership to local community and government representatives, almost all stakeholders agreed that the ability to continue project activities rests upon how well communities can mobilize resources.  Generally, project stakeholders felt confident that the government has sufficient technical capacity to lead project activities and applauded CRS for support provided to MONE and CNCS in joint missions and stakeholder workshops. A few project staff indicated that local municipalities should have greater involvement not only in managing and mobilizing resources for canteens but also, broadly, in monitoring school activities. Some project stakeholders noted that political change and high turnover of municipal and national staff mean that established relationships may be disrupted, which effectively “turns back the clock” on the impact of capacity-building trainings.  Strikes, insecurity, climate change, and the lack of water at schools all complicate lasting effectiveness and sustained operations.  The postponement or cancellation of workshops and meetings intended to ease the handover of project activities to relevant community and government stakeholders may affect sustainability. 4.1 RELEVANCE Interview topics focused on stakeholders’ perspectives on the strengths and weaknesses of the project design and the extent to which the project considered economic, cultural, and political contexts. Stakeholders shared their perceptions on the extent to which the intervention met the needs of beneficiaries and aligned with Mali’s national and local education and development strategies. 4.1.1 Project Goals The perceptions of project goals were consistent across respondent stakeholders, regardless of region. Overall, respondents considered project goals reasonable given that the findings of the midline evaluation guided the way forward at this phase. Project staff and national and local government stakeholders shared the emphasis on girls’ education despite barriers for girls such as their roles and responsibility at home. In the region of Koulikoro, issues related to equity and the project’s ability to address the key context of poverty was important and seen as part of the project’s goals, according to a local government stakeholder. In addition, respondents raised the ability of parents to afford supplies as an issue in achieving project goals from mayors. IMPAQ International, LLC Page 24 McGovern-Dole III Endline Evaluation Report 4.1.2 Project Design In terms of project design, respondents raised issues related to strengths, challenges, and unintended outputs that enhanced the project and achievement of stated goals. Strengths included community involvement, project relationships, and student attendance. Mobilization of local resources from the community, local partners, and government officials led to sound execution and delivery of the project activities. In addition, the relationship between project staff and local school staff, along with partners, was fluid. The project implementation followed an adaptive approach that participants considered reasonable to the realities of the region. In addition, there was a stakeholder perception that the project improved attendance of girls in school. (See more information on findings from monitoring data in Section 4.4). Respondents considered project goals timely and achievable. At the same time, participants did raise some challenges that the project faced in relation to local staff engagement, security, and teacher strikes. Some implementing partners described misalignment in types of positions between the CRS and the implementing partners. We discuss this in detail in Section 4.3.2. One participant suggested that to prevent overload for supervisors at the local partner level and to improve coordination, it would be helpful to have a local partner counterpart position for each CRS program component position. Beyond the initial goals, the role of school gardens in supporting canteens was important to project design. In addition, respondents described the activities of SILCs as creating better financial stability for parents. In Mopti, in particular, it was reported that project design included monitoring and alert mechanisms related to food availability that were successfully carried out by agents from the community. 4.1.3 Fit with Community and Government Priorities In terms of project fit with the community and government priorities, respondents pointed to strengths and challenges. The strengths included that school gardens were appropriate, given the agricultural context of communities. Innovation to continue the work despite emerging challenges was also important. For example, CRS collaborated with USDA to distribute THRs in communities with closed schools due to security concerns. The project also fits with the goals and priorities of the MONE and the government Decennial Program focused on education. Parents understood the importance of education for girls, and the colored report cards helped parents’ understanding. Challenges emerged in the project’s attempt to carry out a mobile literacy program to counter security issues through a radio outreach program. Unfortunately, it was later determined that not all children had radios in their homes. The fit of the project led education officials interviewed in Mopti and Koulikoro to plan to continue providing support for BLA application after this project ends with authorization from MONE. The project “The MONE objective is to improve the quality of teaching, increase school attendance, increase children’s learning level, and allow children to succeed in their lives. The implementation of school feeding is in line with achieving these objectives.” –National government stakeholder IMPAQ International, LLC Page 25 McGovern-Dole III Endline Evaluation Report also fit with the Social, Economic, and Cultural Development Program (PDSEC) five-year plans developed by local governments. Respondents perceived that the project prevented student dropouts from school due to rural migration of boys and early marriage of girls. In Koulikoro, respondents said that teachers plan to continue using BLA, once authorized by the government when the project is done. Local project staff provided sensitization on the importance of education and held meetings with parents to allow girls to continue with school after early marriage. 4.1.4 Participant Satisfaction Respondents were generally satisfied to very satisfied with the project. Respondents had favorable opinions of the level of project responsiveness and continued engagement despite challenges. For example, they described the project as being responsive to the MONE’s request to distribute hot meals to students taking exams during their exam week. Because this distribution was not part of the project, it required an amendment, and CRS determined that there was sufficient food available to meet this request. In addition, despite the COVID-19 pandemic, CRS provided two months of THRsto students. Also, the EDC training was described as having developed participants’ skills in pedagogical approaches. One mayor described receiving many trainings on good governance and citizenship and finding participating in trainings for SMCs useful to understand their expected competencies. In interviews at the national government level, trainings, joint missions, and brainstorming workshops on school feeding were described as appropriate and responsive to government stakeholder needs. In Mopti, an example of a positive social experience in a SILC group was described as happening when two group members died and the other members forgave their debts and gave donations to their families. In Koulikoro, certificates of merit motivated teachers. Generally, in both regions, respondents stressed the importance of BLA in teacher training and student education. 4.2 EFFICIENCY Interview topics focused on efficiency in CRS’s collaboration with other stakeholders, the timeliness of project activities, cost effectiveness, food supply losses, and project resource management. 4.2.1 CRS Collaboration with Other Stakeholders In terms of CRS’s collaboration efforts, the remote interview findings raised areas of strength and considerations for improvement. The strengths included respondents’ description of the communication between USDA and CRS as fluid. BLA was a strong methodology developed in collaboration with the U.S. Agency for International Development (USAID). In addition, the partnership with EDC helped the project anticipate the consequences of the teacher strikes. One example is that students were allowed to take books home, so that they could continue their education, despite the strikes. CRS worked with implementing partners to carry out project activities despite security concerns. The “hire to do” approach of CRS — having a plan and then hiring local expertise to carry out the work — was well received. In Mopti, the project achieved buy-in as project staff and SMCs worked closely with local governments. In Koulikoro, CRS entrusted local partners with leading activities, a model that supported community buy-in. One implementing partner staff member described collaboration between CRS and USDA to distribute commodities of closed schools as THRs to avoid spoilage and to ensure that children would receive food. IMPAQ International, LLC Page 26 McGovern-Dole III Endline Evaluation Report Collaboration with the local government technical services were perceived as favorable, as SMCs and municipality management relied on these mechanisms for training. While CNCS appreciated training provided, given plans to scale up school feeding across the country, there were requests to expand capacity building for regions beyond this project. 4.2.2 Timeliness of Project Activities Respondents described the mobilization of resources as efficient; they said that all resources were used. In addition, both Mopti and Koulikoro respondents said that the M&E facilitators were dynamic and from the community, a factor that seemed to enhance project efficiency. 4.2.3 Cost Effectiveness, Project Resource Management, and Food Supply Losses Remote interview respondents described resource expenditure in compliance with USDA guidelines. Visits to review account records, signed contracts with CRS outlining procedures, and audit missions were critical to project resource management. Though project resource management was compliant with USDA guidelines, there were opportunities for modification and flexibility to respond to project progress, not to exceed US$100,000. Project and partner staff noted challenges with commodity losses during shipment to Mali and from regional warehouses to schools. In one instance, religious extremists stole commodities. CRS notified USDA regarding all commodity losses. Additionally, there was an issue when CRS was informed that the vegetable oil contained genetically modified organisms, and USDA agreed to replace it with sunflower oil. In another case, some of the oil expired. The project followed USDA rules: CRS sold approximately 11 metric tons of spoiled vegetable oil to the soap industry and generated an amount of $4,024, to be used for program activities. To avoid having commodities expire again, the project used them for THRs when schools closed. The teacher strikes affected distribution of commodities. Over a period of about three months, CRS did not receive oil commodities from the U.S. During that time, CRS asked communities to contribute oil to continue canteen operations. In regards to losses and damages from local warehouses, one implementing partner mentioned specifically that building bigger warehouses to hold about six months of food would decrease the number of shipments needed per year, hence the loss of food related to shipments and transport. 4.3 EFFECTIVENESS Interview topics focused on the successes and challenges of program implementation and on the effectiveness of management arrangements. 4.3.1 Successes and Challenges of Project Implementation According to stakeholders, key successes of project implementation included BLA provision, school canteen function and THR provision, cultivation of school gardens, proper food storage and preparation, community contributions to schools, establishment and operation of SMCs and Savings and internal lending community, community engagement, and colored report cards. IMPAQ International, LLC Page 27 McGovern-Dole III Endline Evaluation Report Successes. In terms of the instructional training, stakeholders reported that EDC delivered a complete sequence of initial BLA training, monitoring, and follow￾up. Principals in Mopti noted that the learning environment and student reading abilities improved; project and partner staff described improvement in teachers’ confidence and mastery in applying BLA techniques. Education officials, teachers, and principals described the regular monitoring and feedback as useful. In addition, instructional materials, such as radios, books, games, alphabet boards, albums (small brochures), and USB drives were described as helpful. According to local government and community stakeholders, the project seemed to establish successfully resources and services within schools and engaged parents and community members. Implementing partners and other stakeholders reported that canteens were well managed and stocked and that they followed safe food preparation and storage standards. Principals, education officials and national government stakeholders said that vegetables from school gardens enhanced the quality of daily meals for students, with project staff clarifying that this was the case as long as there was adequate water available. School gardens also seemed reduce canteen expenses related to purchasing vegetables and could generate income for the school from harvests sold during summer months. Supporting data from progress reports included that any excess crops not consumed in the canteen were sold for additional income for the canteen. Community food contributions to schools was both a success and a challenge. For example, when there was a shortage of oil and delivery was delayed, the SMCs led communities to contribute enough oil for the canteens for three months. Additionally, project and partner staff noted that, in villages far from big roads and cities, community contributions to school canteens were very successful following sensitization by SMCs. In addition to sensitizing and mobilizing support from the community, SMCs have operated successfully, collaborated well with local municipalities, and received grants based on their performance. To support and engage parents and community members, the project successfully established SILC groups and provided colored report cards to parents. According to SILC group members and project and partner staff, the SILCs received training, were sustainable and autonomous, and were able to generate funds for members and donations for community needs. The project provided colored report cards to parents, many of whom had low or no literacy skills, to track their children’s school performance. Principals and teachers said that colored report cards improved parents’ involvement in their children’s education. “The colored report card is also an effective tool for parents to monitor the performance of kids at school; its implementation has succeeded because parents that are illiterate know whether their child has performed well or not.” –Principal, Mopti “Students of teachers trained in BLA are better off in reading and writing than those not trained with BLA. It is with BLA that we learn the difference between the sound and the letter. For the moment we are comfortable with the approach.” –Teacher, Mopti IMPAQ International, LLC Page 28 McGovern-Dole III Endline Evaluation Report Challenges. According to stakeholders, key implementation challenges included security concerns, teacher strikes and turnover, inconsistent community food contributions to schools, use of collective fields, SILC contributions to schools, inconsistent community and government ownership of activities, lack of community distribution of vitamin A and deworming medication, and lack of WASH activities. Security concerns were a challenge for project implementation on many fronts. In some cases, parents were reluctant to send children to school and teachers did not attend due to safety concerns, thus reducing attendance rates. Project and partner staff described challenges in transporting project materials and commodities due to a regulation that prohibited vehicles like the ones typically used to transport these materials such as pick-up trucks in areas facing security concerns. Stakeholders noted that turnover –– new teachers entering schools and trained teachers leaving –– led to inconsistent application of the literacy techniques across classrooms and schools. Stakeholders also expressed concerns that prolonged teacher strikes minimized learning time for students and schools’ ability to implement the intervention activities. Although some community food contributions to canteens were successful, principals, SMC members, and project and partner staff shared that there were also challenges. Principals noted that community contributions were small in the beginning but improved after SMC sensitizations. However, partner staff and other stakeholders said that the quantity of community food contributions was inadequate. Project and partner staff said that, in villages next to big roads or cities, community food contributions were inadequate despite sensitization and other project efforts. Additionally, SMC members noted challenges with the collective fields because of a lack of people available to work on the fields. Lastly, while SILCs operated successfully, not all of them contributed to schools. Although SILC members interviewed said that their groups donated cash or food to local schools, project and partner staff said that approximately half of SILC groups contributed to schools and that they needed to do more sensitization of the community. While some principals and education officials interviewed noted regular SILC contributions to the canteen, others noted modest contributions. Progress reports reviewed included many SILCs providing monetary and in-kind donations to schools, with some reporting not being aware of recommendations to contribute to schools. The project initially distributed vitamin A supplements and deworming medication, then partners raised awareness with SMCs on mobilizing financial resources to purchase and distribute these items. The results were mixed. Some communities were able to take on this activity, but many lacked the resources to do so. For example, in progress reports, challenges included not raising sufficient funds or misunderstandings with the community. In an interview, a project partner shared that one community received support from health structures to purchase these items. Additionally, the project did not focus as much on WASH activities as on activities in other areas, given that this was not a focus of the third phase of the project. Project staff advised local government to work with other actors to focus on this component to sustain the efforts from the second phase of the project that did focus on WASH. IMPAQ International, LLC Page 29 McGovern-Dole III Endline Evaluation Report 4.3.2 Coordination and Management CRS and partners reported that CRS is managing the project well, using a consultative and participatory approach, involving partners in all parts of implementation, and creating synergy among partners for successful project implementation. CRS conducts missions to monitor partners to see what is working and where there is room for improvement. Regarding staffing structure and capacity, CRS and partners stated that each partner has its own team. The teams may not be as varied as CRS’ team is, but the variety is often not needed. CRS and partners mentioned the complementarity of the model, with CRS managing the project, EDC specializing in educational quality and training, and the four local field partners bringing knowledge of the area and presence in the community for project implementation. One partner staff member noted that, although CRS had a sustainability lead, the partners did not have an equivalent role; this respondent thought such a role would have been beneficial to facilitate direct communication instead of going through other staff. Although partners generally said that coordination between CRS and partners was going well, some partner staff mentioned a need for more capacity building beyond guidance for project implementation. CRS and partners also reported that program activities were consistently monitored. In addition to holding quarterly meetings to share updates and address issues, CRS conducted regular field visits. One partner staff member described the challenges of monitoring due to security concerns in some locales as an area for improvement. In the beginning of this project phase, CRS started using a digital platform for electronic data collection to improve the quality and timeliness of results from the field. For example, tablets recorded the geographic location of field data collection agents, providing enhanced reliability. CRS and partners shared that field agents needed a lot of training in electronic data collection. Additionally, CRS and partners used phone calls in situations where in-person data collection was not possible due to security concerns, teacher strikes, or later on because of COVID-19. Partners noted as a strength CRS’s quality control system and M&E manual describing all roles and responsibilities. An area for further discussion was the M&E lead for each partner, who had access to a tool with backup data that was separate from the CRS monitoring database. Partner M&E leads had mixed opinions about having access to a separate database and not the central CRS monitoring database. In some cases, partners said that this led to confusion about the M&E lead’s responsibilities indicating that there could be a need for more sensitization and clarity regarding this role. 4.4 IMPACT To assess the overall impact of the McGovern-Dole III on the literacy of school-aged children (SO1) and on health and nutrition practices (SO2), we looked at the outcomes within the project results framework in three ways: 1. Performance evaluation. We used a mixed-method approach to assess changes in outcomes over the life of the project to the extent possible. We relied on multiple data sources with different samples to address as many evaluation questions as rigorously as possible. IMPAQ International, LLC Page 30 McGovern-Dole III Endline Evaluation Report 2. Exploratory impact evaluation. Without making any causal claim, we estimated the influence of BLA package on student literacy using secondary data collected by EDC using EGRA. 3. Perceived impact. To better understand and contextualize the quantitative findings of the performance and exploratory impact evaluations, we gathered and analyzed qualitative data related to the perceived influence of project activities on students’ literacy (SO1) and nutrition and hygiene (SO2) outcomes. During data analysis process, we have identified a few external factors out of project’s control such as teachers’ strike and political instability, which caused school disruptions in Mali, especially after 2018. These events seemed to cause some challenges for the literacy interventions since teachers were not consistently in their classrooms to teach or integrate the received training while teaching. For example, there was a string of teachers’ strikes in 2018-2019 that led to an extensive loss of instructional time for students. EDC calculated this loss equivalent to approximately a total of 64 school days over the academic year 2018-2019, as mentioned in EDC EGRA report in 2019. Another prevalent factor was political instability and terrorist attacks that mainly affected Mopti region (Tracking Conflict Worldwide, International Crisis Group 2020). Teachers were also on strike in 2017-2018, but they were not as extensive as those in 2018-2019 (GardaWorld 2019). Moreover, these factors made assessing project’s progress challenging over years given the evident differences in school environment from one year to another. For example, it could be challenging to compare students’ reading proficiency in a “normal” year, when teachers are regularly attending school to teach with no school disruptions, with their peers who have been through crisis in the same school but another year because of political instability or their teachers’ strikes. Therefore, the results in this section should be interpreted with caution, especially for students’ literacy outcomes. 4.4.1 Performance Evaluation In this section, we present the findings with respect to the McGovern-Dole results framework depicted in Appendix A. For each strategic objective, first, we introduce their corresponding McGovern-Dole outcomes with a brief reference to the rationale behind their theory of change. Then, we triangulate qualitative and quantitative findings to describe the observed changes and provide contextual information. For secondary quantitative analysis, we examined all the relevant data by gender, region, grade, and school’s curriculum type, where applicable. We only highlight them in the report when the difference was significant. Finally, we conclude how the changes in these outcomes affected the main strategic objectives of McGovern-Dole III. MGD 1.1 Improved Quality of Literacy Instruction McGovern-Dole III posits that if teachers receive improved pedagogical materials (Fullan & Landworthy, 2014); if teachers attend and teach at school more regularly (Mattioli et al., 2016); and if teachers and educational leaders’ capacity to deliver and support high-quality literacy is enhanced (Leithwood et al., 2004), then the quality of literacy instruction in Mali would improve. To assess each of these intermediate outcomes, we used multiple data sources, as shown in Exhibit 12. IMPAQ International, LLC Page 31 McGovern-Dole III Endline Evaluation Report Exhibit 12. Data Sources Outcome Qualitative Sources Quantitative Sources Teacher attendance KIIs with education officials, principals, teachers, SMC members EDC Survey of SMC members Literacy instructional materials KIIs with principals and teachers EDC Classroom observations Skills and knowledge of teachers KIIs with education officials, principals, teachers EDC Surveys of teachers, SMC members, and school administrators Skills and knowledge of administrators KIIs with education officials, project staff and partners, principals, teachers N/A To clarify the sample characteristics of each quantitative dataset for these outcomes, we describe below teacher, SMC, and classroom observation data collected by EDC. Exhibit 13 shows the breakdown of the teachers surveyed by EDC by school year, region, and curriculum. In December for the 2015 – 2016 school year and in June for 2017 – 2018, EDC surveyed two teachers in each school; in other years, it surveyed just one teacher per school, in June. EDC administered the survey twice for 2015 – 2016, once in December, at the beginning of the project, and once at the end of the year in June. We split the samples evenly between regions. However, classic schools, where French is the language of instruction, outnumber bilingual curriculum schools in the sample, where Bamanankan is the predominant language of instruction for early grades. Each year, EDC surveyed teachers in various grades from Grade 1 to Grade 3, as well as teachers that taught more than one grade in a classroom, referred thereafter as multi-grade. Given the inconsistency of teacher grades in different years, we do not disaggregate outcomes related to teachers by grade in this report. 13 Exhibit 13. Teacher Sample by Region and Curriculum Characteristic 2015 – 2016 (Dec.) 2015 – 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Region Mopti Schools 19 17 17 18 19 Teachers 46 17 17 33 19 Koulikoro Schools 21 17 20 19 21 Teachers 45 17 20 39 21 Curriculum Bilingual Schools 4 4 4 3 4 13 In December 2015, EDC surveyed teachers in Grades 1, 2, and 3, as well as, multigrade classrooms. In June 2016, it surveyed Grade 1 and multigrade teachers. Then in 2017, EDC surveyed Grade 2 and multigrade classrooms. In 2018, it surveyed Grade 1, 3, and multigrade classrooms. Finally, in 2019 EDC surveyed Grade 2 and multigrade classrooms. IMPAQ International, LLC Page 32 McGovern-Dole III Endline Evaluation Report Characteristic 2015 – 2016 (Dec.) 2015 – 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Teachers 12 4 4 8 4 Classic Schools 36 30 32 33 36 Teachers 79 30 32 62 36 Total Schools 42 42 41 40 42 Teachers 91 34 37 72 40 Grades Surveyed G1, G2, G3, & multi-grade G1 & multi-grade G2 & multi-grade G1, G3, & multi-grade G2 & multi-grade Source: teacher survey; authors’ calculations. Note: EDC surveyed schools twice in the 2015 – 2016 school year in December and June, and once in June every other year. In the data, we are missing the curriculum for one school. Exhibit 14 shows the distribution of the SMC sample by region, curriculum, and year. We included only SMC members whose schools had data available at the beginning and end of the program. It seems that one SMC member from each school was surveyed. Exhibit 14. SMC Sample by Region and Curriculum Characteristic 2015 – 2016 (Dec.) 2015 – 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Region Mopti 18 17 18 17 18 Koulikoro 24 23 22 22 24 Curriculum Bilingual 5 4 5 4 5 Classic 37 36 35 35 37 Total 42 40 40 39 42 Source: SMC survey; authors’ calculations. EDC asked selected SMC members about the number of SMC members affiliated with their school. At the end of the 2015 – 2016 school year, SMCs reported an average of 10.2 SMC members in each school. The overall number remained steady, landing at 9.8 SMC members per school at the end of the 2018 – 2019 school year. As shown in Exhibit 15, Mopti saw its SMC membership drop from 15.8 to 7.3 on average while in Koulikoro the number doubled from 6.1 to 12.0 SMC members per school. The drop in SMC membership in Mopti might have been associated with a shift in the community priorities given the frequent terrorist attacks in that region. SMCs were composed primarily of men, a finding that remained consistent over time. IMPAQ International, LLC Page 33 McGovern-Dole III Endline Evaluation Report Exhibit 15. Average Number of SMC Members per School, by Region and Gender Source: SMC survey; authors’ calculations. N = 23 in Mopti and 17 in Koulikoro in June 2016; 22 in Mopti and 18 in Koulikoro in 2016 – 2017; 22 in Mopti and 17 in Koulikoro in 2017 – 2018; and 23 in Mopti and 19 in Koulikoro in 2018 – 2019. Exhibit 16 shows the sample of schools in the classroom observation sample. Although EDC collected observational data from 65-82 schools each year, we analyzed data only from the 43 schools that had observational data consistently at the beginning and the end of the project. The number of schools in the sample is fairly even between regions with a small tilt toward schools in Koulikoro. EDC observed multiple classrooms in each school at the beginning of the project and one per school for the remaining years. It observed Grades 1, 2, and 3 and multigrade classrooms in the first year and select grades thereafter.14 On average, EDC staff observed 47 students per classroom in December 2015. In the remaining years, visiting in June, they observed between 32 and 41 students per classroom. The initial number is likely higher because it was measured earlier in the school year, when we found attendance rates to be at their highest and before students could drop out. 14 In 2017 – 2018, EDC observed Grade 1 and Grade 3 classrooms; in 2018 – 2019, it observed Grade 2 and multigrade classrooms. There are no data on the grades observed for the June 2016 or the 2016 – 2017 data collections. 14 11 12 5 10 9 10 6 6 10 3 2 4 1 3 2 2 2 1 2 17 13 16 6 13 11 12 8 7 12 0 2 4 6 8 10 12 14 16 18 20 Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro 2015-2016 (Dec) 2015-2016 (June) 2016-2017 (June) 2017-2018 (June) 2018-2019 (June) Male Female IMPAQ International, LLC Page 34 McGovern-Dole III Endline Evaluation Report Exhibit 16. Classroom Observation Sample by Region and Curriculum Characteristic 2015 – 2016 (Dec.) 2015 – 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Region Mopti 19 19 17 16 19 Koulikoro 24 23 22 16 24 Curriculum Bilingual 5 5 5 5 5 Classic 38 h37 33 26 38 Total 43 42 39 32 43 Source: classroom observation survey; authors’ calculations. Note: We are missing the curriculum for one school. Teacher Attendance (MGD 1.1.1.) According to education officials, using BLA techniques motivates teachers to come to school because the approach does not require as much effort and preparation as other methods. Principals and SMC members also noted perceptions that teacher attendance improved due to greater motivation after seeing improvement in student learning with BLA techniques. However, all of the teachers interviewed said that there was no problem with teacher attendance and that absences were due only to illness or social issues. The teachers interviewed did not share changes regarding motivation. On the other hand, local education officials, community stakeholders (principals, teachers, and SMC members), and project and partner staff observed that teacher attendance increased during the project, except when schools were closed due to teacher strikes. We integrated the findings from qualitative interviews with responses from the SMC survey collected annually by EDC. In that survey, we asked SMCs about teacher absenteeism. As shown in Exhibit 17, the percentage of SMC representatives who believed teachers were often absent fluctuated over the years. However, overall it increased substantially from 30 to 54 percent by the end of the 2018 – 2019 school year. The variation seems to be more drastic in Mopti than in Koulikoro. Though teacher absenteeism appears to have increased in both regions, the increase was largely driven by Mopti, where the percentage of SMC members who said teachers were often absent increased by 52 percentage points over the life of the project. However, SMC responses seem to be consistent with the teacher strikes,15 especially in 2017 – 2018 and 2018 – 2019. (EDC, 2019) (GardaWorld, 2019). 15 Teachers were also on another strike in the 2017 and 2018 school year, however, the length of their strike was not as extensive as in the 2018-2019 school year. IMPAQ International, LLC Page 35 McGovern-Dole III Endline Evaluation Report Exhibit 17. SMC Members Who Say Teachers Are Often Absent Source: SMC survey; authors’ calculations. Note: N = 17 in Mopti and 23 in Koulikoro in June 2016; 18 in Mopti and 19 in Koulikoro in 2016 – 2017; 7 in Mopti and 14 in Koulikoro in 2017 – 2018; and 15 in Mopti and 20 in Koulikoro in 2018 – 2019. Provision of Instructional Materials (MGD 1.1.3) In interviews, teachers, education officials, and project staff and partners shared the impacts of improved instructional materials. Education officials in Koulikoro noted improved teaching quality because of the materials and training; they said that the materials helped teachers feel comfortable with the new techniques and motivated to use them. Similarly, one implementing partner noted that, with the materials and training, new students were able to read within three months. Education officials in Koulikoro and project staff and partners shared that specific materials were beneficial, such as using the radio with teaching, games, brochures, alphabet posters, and books. However, they also pointed to some challenges, for example, one teacher in Mopti mentioned receiving materials but having functionality issues with some such as the radio and battery charger. Exhibit 18 shows the materials used in classes observed by EDC in December 2015, June 2016, and June 2019. At the beginning of the project, teachers mostly used only blackboards, slates, and textbooks. Later in the project, they seemed to use a more diverse selection of materials, including alphabet and text posters, flash cards, and story books. However, this result should be interpreted with caution as observational data only provides a snapshot of the classroom on a single day when the observation was conducted. More observations might be needed throughout the year to provide a better vision of how teachers use learning materials in their planned syllabus. 35% 61% 29% 87% 26% 42% 29% 30% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2015-2016 (June) 2016-2017 2017-2018 2018-2019 Mopti Koulikoro IMPAQ International, LLC Page 36 McGovern-Dole III Endline Evaluation Report Exhibit 18. Classroom Use of BLA Materials, by Year Source: classroom observation survey; authors’ calculations. N = 90 – 109 observations in Dec. 2015; 42 in June 2016; and 40 in June 2019. Skills and Knowledge of Teachers (MGD 1.1.4) We looked at the types of pre-service and in-service trainings received by teachers beyond the BLA training. As shown in Exhibit 19, the plurality of teachers received their pre-service training through the IFM (Institut de Formation de Maîtres), 16 a finding that is consistent with the midline report. This proportion changed very little throughout the project. At baseline and midline, we found that many teachers also received the SARPE (Stratégie Alternative de Recruitment du Personnel Enseignant) 16 IFM is a teacher training school. All teacher training schools have a four-year program for Grade 9 graduates and a two-year program for Grade 12 graduates. The training program includes psychology; pedagogy; and instructional subjects such as science, mathematics, and languages. 98% 58% 1% 49% 2% 2% 4% 2% 0% 0% 100% 45% 88% 7% 19% 50% 21% 5% 14% 0% 100% 75% 60% 40% 35% 28% 25% 25% 0% 0% 0% 20% 40% 60% 80% 100% 120% Blackboard Slates Alphabet posters Manuals Text posters Alphabet cards Flash cards Story books Radio support audio Video or multimedia support Percent of Observed Classrooms 2015-2016 (Dec.) 2015-2016 (June) 2018-2019 IMPAQ International, LLC Page 37 McGovern-Dole III Endline Evaluation Report training; 17 the EDC survey did not ask about this training. Twice the proportion of teachers were trained through ECOM (Ecole Communautaire) and the end of the program (18 percent) compared to the beginning (nine percent). 18 A few teachers at the beginning of the project were trained through IPEG (Institut Pédagogique d'Enseignement) which is a training institute that was replaced by IFM. By the 2018 – 2019 school year, 13 percent of teachers had no formal pre-service training, an increase from 5 percent at the beginning of the project. Exhibit 19. Percentage of Teachers Receiving Pre-Service Training by Year Source: Teacher survey; authors’ calculations. N = 91 teachers in Dec. 2015; 32 in June 2016; 37 in 2016 – 2017; 70 in 2017 – 2018; and 40 in 2018 – 2019. EDC also asked teachers if they had received training to teach the bilingual curriculum, which was not part of the BLA training. Approximately 30 percent of teachers reported receiving training in the bilingual curriculum, including one-third of teachers in bilingual schools. The consistency of teachers’ responses 17 SARPE is “a fast-track training route which involves taking slightly older students—again, with a minimum qualification of the DEF (although many will have received some further education)—and training them over what was 15 days and is now six months. SARPE is organized and taught by the local education authorities, with school advisors taking a prominent role in the training.” (“Mali: Teacher Preparation and Continuing Professional Development in Africa (TPA)”). Center for International Education (CIE). (2016). Brighton, England: University of Sussex. http://www.sussex.ac.uk/cie/projects/completed/tpa/mali. 18 ECOM is a 45-day training program for community school teachers. Those teachers are hired and paid by communities but go through this government-supported training program. The program also includes psychology, pedagogy, and instructional subjects. 42% 9% 1% 5% 20% 41% 19% 6% 9% 0% 38% 24% 3% 16% 0% 41% 21% 0% 9% 4% 40% 18% 0% 13% 8% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% IFM Ecom IPEG None Other 2015-2016 (Dec.) 2015-2016 (June) 2016-2017 2017-2018 2018-2019 IMPAQ International, LLC Page 38 McGovern-Dole III Endline Evaluation Report about bilingual training over the years seems to suggest that teachers’ skills to teach students in local language have not been improved even in bilingual schools. To improve teacher’s quality of teaching, the McGovern-Dole III implemented BLA package, as mentioned before. BLA was developed by USAID/PHARE program with the MONE, and focused on teachers in Grade 1, 2 and 3. This program incorporates interactive radio programs plus seven strategies designed to improve students’ language mechanics, phonics, decoding, comprehension, encoding, and written expression through a system of in-service training for teachers and school administrators. BLA also provided a support mechanism for teachers to improve their teaching methods by consistent coaching and mentoring of trained school administrators. The program trained administrators to make regular school visits, observe teachers in their classroom and provide them with constructive feedback. At baseline and midline, we asked teachers directly whether they received the BLA training and whether they used those techniques. The outcomes at midline suggested that BLA package was implemented as planned. Almost all teachers received the training on BLA. The BLA intervention also appeared to have translated into actual application in the classroom, self-reported by teachers. More importantly, significant improvement in students’ literacy outcomes reflected how well BLA was received by teachers. After midline (2018), although political instability and teachers’ strikes caused challenges for implementing partners to continue providing and improving BLA, they ensured maintaining these improvements and addressing IMPAQ’s recommendations toward sustainability of BLA. Their effort includes:  The educational advisors and trainers supported ten of teacher’s learning communities in 2018- 2019 to strengthen the pedagogical capacities of teachers in targeted grades focusing on guided reading and guided writing;  The school administrators expanded their feedback session from their school visits and pedagogical coaching sessions to all teachers (teachers with and without BLA training). The purpose of these extended sessions was to help all teachers learn from the administrators’ constructive feedback;  The project initiated one model lesson per quarter for each school to allow untrained teachers from the same school observing their trained colleagues while they were teaching in classroom. Teachers were expected to exchange ideas after class and share their lessons learned;  EDC also hosted a two-day orientation session for teachers in higher grades (Grade 4, 5, and 6) in 2019-2020 to learn about BLA techniques; and  The project incorporated trainings on positive disciplines to reinforce the concept of a child￾friendly classroom for classroom management and avoiding corporal punishments in BLA trainings. At endline, we used EDC teacher survey data at to understand the progress on teacher’s skills and knowledge. Unlike IMPAQ survey, EDC teacher survey did not include direct questions from teachers on IMPAQ International, LLC Page 39 McGovern-Dole III Endline Evaluation Report use of different BLA techniques/strategies. Instead, they mainly focused on a breakdown of guided reading and writing techniques by asking teachers their opinion and observing them in classrooms. We used those data at endline to understand the progress on teacher’s skills and knowledge. Teachers were also asked if they had already been trained in interactive radio instruction and BLA. Before the project began, 38 and 40 percent of teachers reported they have already received interactive radio and BLA training (from other sources), respectively. However, by the end of the first year of the program, almost all teachers reported having received both kinds of training. A small dip in 2017 – 2018 might be explained by interruptions caused by teacher strikes. EDC asked teachers a series of questions to gauge their opinions on specific pedagogical topics to understand if they followed BLA techniques. Exhibit 20 compares teachers’ opinions on reading and writing from near the beginning of the project to near the end. At endline, teachers were significantly less likely to say that students must memorize a text to understand it, that writing is about good handwriting and respecting grammar rules, or that students must be able to identify and copy all the letters of the alphabet before learning to read and write (p < 0.01). The proportion of teachers who agreed that students must know the relationship between sounds and names of letters in order to write remained high over years. The slight decline (7 percentage points) from 2015 – 2016 to 2018 – 2019 is not statistically significant. Exhibit 20. Teachers’ Opinions on Student Reading and Writing Opinion 2015 – 2016 (Dec.) 2018 – 2019 Difference (p-value) Students must memorize a text to be able to understand it. 82% 21% –61%*** (0.00) Writing is about good handwriting. 92% 29% –63%*** (0.00) Writing is about respecting grammar rules. 95% 61% –34%*** (0.00) To be able to write, one must know the relationship between sounds and names of letters. 94% 87% –7% (0.20) Students must be able to identify all the letters of the alphabet and copy them before they can read and write a word. 91% 34% -57%*** (0.00) Writing well is above all being able to express your own ideas 88% 95% 7% (0.27) Source: teacher survey; authors’ calculations. N = 56 – 82 teachers in Dec. 2015 and 38 in 2018 – 2019. Note: Teachers of Grades 1 – 4 were surveyed in 2015 – 2016 and teachers of Grade 2 in 2018 – 2019. EDC also asked Teachers whether BLA texts that they received were well adapted to instruction in Grades 1 and 2. In 2018 – 2019, 86 percent of teachers said “yes,” compared to 76 percent at baseline. However, this difference is not statistically significant. Moreover, EDC asked teachers their opinions about how students learn. Exhibit 21 shows their responses. The proportion of teachers who believed that teachers must correct students’ reading and writing errors and that students in Mali find it difficult to write because they are not smart declined by margins significant at least at the 5 percent level. The proportion of teachers who said that teachers should encourage student’s imaginations during writing activities also declined significantly. Additionally, the proportion of teachers who reported that all students who go to school have the ability to learn to read increased from 82 percent at baseline to 95 percent at endline, a IMPAQ International, LLC Page 40 McGovern-Dole III Endline Evaluation Report significant increase at the 10 percent level. However, the proportion of teachers who said that teachers should encourage student’s imaginations during writing activities also declined significantly. Exhibit 21. Teachers’ Opinions on Pedagogy Opinion 2015 – 2016 (Dec.) 2018 – 2019 Difference(p￾value) All students who go to school are able to learn to read. 82% 95% 13%* (0.06) All students who go to school are able to learn to write. 87% 95% 8% (0.21) Students must be able to ask questions to their teachers 95% 97% 2% (0.57) One must encourage students’ imagination during writing activities. 99% 92% –7%** (0.05) One must correct all of the students’ errors. 86% 11% –80%*** (0.00) Teachers must correct all of the students’ writing. 90% 18% –72%*** (0.00) Students in Mali find it difficult to write because they are not intelligent. 35% 8% –27%*** (0.01) Source: teacher survey; authors’ calculations. N = 62 teachers in Dec. 2015 and 37 in 2018 – 2019. Note: Teachers of Grades 1 – 4 were surveyed in 2015 – 2016 and teachers of Grade 2 in 2018 – 2019. Similarly, as shown in Exhibit 22, EDC asked school principals the same pedagogical questions posed to teachers. By endline, almost every principals (99 percent) agreed that all students who attend school are capable of learning to read and write, up from 85 percent and 83 percent respectively. This finding is significant at the 1 percent level. There were substantial declines in the proportion of school principals who believed teachers should correct all students’ reading and writing errors and that students in Mali find it difficult to write because they are not smart. Exhibit 22. School Principals Opinions on Pedagogy Opinion 2015 – 2016 (Dec.) 2018 – 2019 Difference (p-value) All students who go to school are able to learn to read. 85% 99% 14%*** (0.00) All students who go to school are able to learn to write. 83% 99% 16%*** (0.00) One must encourage students’ imagination during writing activities. 97% 100% 3% (0.16) One must correct all of the students’ errors. 76% 15% –61%*** (0.00) Teachers must correct all of the students’ writing. 77% 13% –65%*** (0.00) Students in Mali find it difficult to write because they are not intelligent. 28% 3% –25%*** (0.00) Source: school principal survey; authors’ calculations. N = 71 or 72 teachers in Dec. 2015 and 68 – 71 in 2018 – 2019. In the qualitative interviews, partners and other stakeholders reported that teachers received training on BLA and that teachers had mastered the BLA techniques, as determined by regular monitoring, IMPAQ International, LLC Page 41 McGovern-Dole III Endline Evaluation Report assessments, and feedback sessions. They noted that, with training, teaching quality and practices improved. However, one education official suggested holding additional trainings to promote consistent application of BLA techniques. Interviewed teachers confirmed that they found the training to be useful. Project and partner staff said that the project shifted teachers’ mindset to believe that children could learn to read at an early age. Local education officials and community stakeholders (principals and SMC members) stated that teachers seemed to be more comfortable in the classroom, more motivated, and more engaged with students. They said that teachers even seemed to be having fun when applying BLA techniques. Teachers noted finding the monitoring and feedback from education advisors to be helpful. While creating teacher learning communities was included in the project sustainability plan and as a successful practice in progress reports, one education official in Koulikoro recommended that the project support teacher learning communities, while another shared an example of teachers starting a learning community being supported by local governments to continue capacity building. Skills and Knowledge of Administrators (MGD 1.1.5) In stakeholder interviews, education officials and principals said that they were trained by the project. At the government level, education advisors shared being satisfied with the training they received to build their skills on BLA and EGRA. They described feeling comfortable leading trainings for teachers on their own after receiving project training. The CAP directors also noted that the training was sufficient to build the capacity of education advisors to support and train school principals and teachers. They added that the project training also addressed previous gaps in knowledge so that all education advisors were at the same skill and competency level for monitoring and training teachers. Principals said that they found the training to be useful, but some requested adding more days of initial training or refresher training. Overall, despite previous difficult working conditions, according to an SMC member in Koulikoro, teachers’ pedagogical knowledge appears to have improved over the life of the project. The effect of the BLA training is apparent in the change in teachers’ pedagogical opinions over time. However, there are still challenges, in particular with teacher absenteeism, which appears to be on the rise in Mopti, as confirmed with teacher attendance data shown above. Additionally, many teachers still have no formal pre-service training. MGD 1.2 Improved Attentiveness According to the McGovern-Dole III results framework, if nutrient-rich meals are regularly provided to children at school and access to food is increased (Bradely & Greene et al., 2013; Rasberry et al., 2011), then students’ short-term hunger will be mitigated (SO1.2.1) (Dani, Burrell, & Demmig-Adams, 2005; Glewwe, 2001), so that their attentiveness in the classroom will improve (Ross, 2010). “I am confident in myself; I know how to manage a school, manage the issues with the SMC, manage a school with BLA, supervise a canteen and use tools for food storage management.” –Principal, Mopti IMPAQ International, LLC Page 42 McGovern-Dole III Endline Evaluation Report We looked at this path in the results framework mostly using qualitative findings with support from CRS monitoring data. The CRS monitoring data regarding hot meals contained records for 10 months for each school year, 2016 through 2019. The distribution of hot meals at schools represents the project’s effort to affect students’ diets and access to nutrition, while encouraging them to attend school. Exhibit 23 summarizes findings for the whole sample. Exhibits 24 and 25 show consistent distributions of hot meals in schools in both regions and for both student genders. The overall values of Exhibit 23 represent the weighted average of the values shown by region in Exhibit 24, or the sum of the values shown by gender in Exhibit 25. Exhibit 23. Average Number of Hot Meals Distributed in December per School Source: CRS monitoring data. N = 247 schools in 2016 – 2017; 245 in 2017 – 2018; 270 in 2018 – 2019; and 267 in 2019 – 2020. Note: Showing single-month output for December as representative of typical monthly distribution amounts because there were minimal missing data across the available schools and because differences across years were less apparent in annually aggregated values. The values shown in this graph represent the weighted average of the values shown by region below, or the sum of the values shown by gender below. Exhibit 24. Average Number of Hot Meals Distributed in December per School, by Region Source: CRS monitoring data. N = 119 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 117 in Mopti and 128 in Koulikoro in 2017 – 2018; 140 in Mopti and 130 in Koulikoro in 2018 – 2019; and 128 in Mopti and 139 in Koulikoro in 2019 – 2020. 2,831 2,486 2,960 3,187 0 500 1,000 1,500 2,000 2,500 3,000 3,500 2016-2017 2017-2018 2018-2019 2019-2020 Hot Meals 3,184 2,781 3,081 3,589 2,502 2,234 2,843 2,779 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 2016-2017 2017-2018 2018-2019 2019-2020 Hot Meals Mopti Koulikoro IMPAQ International, LLC Page 43 McGovern-Dole III Endline Evaluation Report Note: Showing single-month output for December as representative of typical monthly distribution amounts because there were minimal missing data across the available schools and because differences across years were less apparent in annually aggregated values. Exhibit 25. Average Number of Hot Meals Distributed in December per School, by Gender Source: CRS monitoring data. N = 247 schools in 2016 – 2017; 245 in 2017 – 2018; 270 in 2018 – 2019; and 267 in 2019 – 2020. Note: Showing single-month output for December as representative of typical monthly distribution amounts because there were minimal missing data across the available schools and because differences across years were less apparent in annually aggregated values. In a sharp decline from the first year of data collection, CRS monitoring data show that the 2017 – 2018 school year had a yearly total average of 4.9 days in Mopti and 8.8 days in Koulikoro on which school was open, but the canteen did not operate. We see an inconsistency between these results and responses from school principals at the midline evaluation report (March 2018). At midline, they reported almost zero days of canteen non-operation. This inconsistency may be explained by the different timing of data collection, with the monitoring data collected at the end of the school year and the principal interviews conducted in early 2018. Overall, the average number of days without canteen operation has declined in both regions since 2016, as shown in Exhibit 26. The increase during the 2018 –2019 school year might be related to security issues that either caused school closures or delayed some of the project activities such as training cooks. It could also be related to the teacher strikes and subsequent school closures which affected public schools across the country. According to CRS performance report in FY2018-2019, only schools with community-paid teachers continued to operate normally in the most secure zones. 1,362 1,205 1,429 1,557 1,423 1,243 1,485 1,571 0 200 400 600 800 1,000 1,200 1,400 1,600 1,800 2016-2017 2017-2018 2018-2019 2019-2020 Hot Meals Boys Girls IMPAQ International, LLC Page 44 McGovern-Dole III Endline Evaluation Report Exhibit 26. Average Number of School Days Without an Operational Canteen per School, by Region Source: CRS monitoring data. N = 119 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 117 in Mopti and 128 in Koulikoro in 2017 – 2018; 134 in Mopti and 129 in Koulikoro in 2018 – 2019; and 132 in Mopti and 126 in Koulikoro in 2019 – 2020. In addition, principals, teachers, and education officials noted increased attentiveness, concentration, focus, and motivation for students with implementation of BLA. Principals added that children were more attentive because they found the classes to be fun and enjoyed the games involved. Related to student attentiveness, SMC members and local education officials noted that the project addressed student hunger, making it easier for students to focus and learn. MGD 1.3 Improved Student Attendance McGovern-Dole III hypothesizes that if nutritious meals are regularly provided to children at school; if parents take a more active role and ownership in the quality of their children’s education (Cornille et al., 2004; Fan & Chen, 2001); and if the school infrastructure19 is improved (Fisher, 2001);if the attitude of the community toward the benefits of education, especially for girls, is improved (Gorard et al., 2012), then, parents will be more motivated to enroll their children in school and encourage them to attend school more consistently. Exhibit 27 shows the data sources we used to explore the theory of change behind improved attendance outcomes. 19 Improved school infrastructure refers to MGD 1.3.3 outcome. In lack of school observations and primary data, we do not have any information to report on this path. 16.6 4.9 8.8 3.8 16.4 6.8 9.3 6.0 0 2 4 6 8 10 12 14 16 18 2016-2017 2017-2018 2018-2019 2019-2020 School Days Mopti Koulikoro IMPAQ International, LLC Page 45 McGovern-Dole III Endline Evaluation Report Exhibit 27. Data Sources Outcome Qualitative Sources Quantitative Sources Increased economic and cultural incentives and decreased disincentives KIIs with all stakeholders THR monitoring data SILC reference quarterly reports Increased community understanding of benefits of education KIIs with all stakeholders Community commodity contribution monitoring data Grain monitoring data Increased enrollment KIIs with principals, teachers, SMC members, and project and partner staff CRS attendance data Hot meals monitoring data THR monitoring data Deworming pills monitoring data Vitamin A pills monitoring data Attendance and reduced health-related absences* KIIs with all stakeholders CRS attendance data Hot meals monitoring data THR monitoring data Deworming pills monitoring data Vitamin A pills monitoring data Community commodity contribution monitoring data Canteen monitoring data *Notes about health-related absences are covered under SO2, increased health and nutrition practices. Increased Economic and Cultural Incentives and Decreased Disincentives (MGD 1.3.1) McGovern-Dole III activities that focus on improving economic and cultural incentives in order to improve attendance include provision of hot meals to all students, provision of THRs conditioned on regular attendance, and formation of SILC groups. See section 4.4.3 increased cultural incentives and decreased disincentives to student school attendance. Provision of hot meals was briefly discussed in the previous sub-section under MGD 1.2 (improved attentiveness). Consistent with earlier findings, CRS monitoring data on THR distribution shows that the average number of students who received rations stayed steady year over year across both regions and gender, as shown in Exhibits 28 and 29. This finding is consistent with how the project activities were designed. Stakeholder interviews cited improved canteen operation related to SMC training on food preparation, storage, and hygiene; mobilization of community support for canteens; and successful use of school gardens to improve the quality of meals provided. IMPAQ International, LLC Page 46 McGovern-Dole III Endline Evaluation Report Exhibit 28. Average Number of Students who Received Take-Home Rations per School, by Region Source: CRS monitoring data. N = 111 schools in Mopti and 126 in Koulikoro in 2016 – 2017; 108 in Mopti and 127 in Koulikoro in 2017 – 2018; and 99 in Mopti and 126 in Koulikoro in 2018 – 2019. Exhibit 29. Average Number of Students who Received Take-Home Rations per School, by Gender Source: CRS monitoring data. N = 237 schools in 2016 – 2017; 235 in 2017 – 2018; and 225 in 2018 – 2019. Regarding formation of SILC groups, stakeholders described the SILC groups as a major success of the project. SILC groups were established with initial training provided by project staff and partners. They implemented savings and social aid components. The savings allowed members to save money to use for income-generating activities. Social aid allowed for community contributions or could become emergency loans members repay without interest. A SILC member in Koulikoro shared that the group received training to include a penalty component to assess fines to SILC members as a penalty for violating group rules. The groups were composed primarily of women; respondents noted that the 149 124 128 130 116 132 0 20 40 60 80 100 120 140 160 2016-2017 2017-2018 2018-2019 Take-Home Rations Mopti Koulikoro 64 57 63 74 63 68 0 10 20 30 40 50 60 70 80 2016-2017 2017-2018 2018-2019 Take-Home Rations Boys Girls “Households not participating in SILC activities are in poverty because they don’t have any financial source to initiate income generating activities, or to take care of social needs. The SILC makes us financially autonomous.” –SILC member IMPAQ International, LLC Page 47 McGovern-Dole III Endline Evaluation Report groups created social cohesion. SILC members and partner staff noted that, after initial training, the groups were able to operate on their own and had good potential for sustainability. Group members noted the benefits of being able to access money for income-generating activities more easily than from a financial institution. SILC and SMC members said that they were not aware of collaboration between the groups, though SILC members said that their SILCs contributed to school operations and canteens by giving cash, grains, vegetables, fish, and meat to schools. However, project and partner staff and community stakeholders said that some SILCs provided modest support for schools while others did not. Project staff noted that approximately half of SILC groups contributed to local schools, so there is a potential need for more sensitization on supporting schools. Exhibit 30 shows the SILC participation counts for each year by gender. As expected, SILC membership improved over the first two years. According to CRS new groups were added in other regions with USDA’s approval, particularly at the end of 2018. In the last years of McGovern-Dole III (2019 and 2020), the project was taking a supervisory role for newly added SILC groups. Exhibit 30. SILC Participation by Year Source: SILC reference quarterly reports Increased Community Understanding of Benefits of Schools (MGD 1.3.5) To increase community awareness of the benefits of education, McGovern-Dole III engaged parents and communities in school-related activities. These activities helped transfer a sense of ownership to the community to support the sustainability of the project. We point out some of these activities in this section, including community contributions to the school canteen, provision of trainings to SMCs, engaging parents in their children’s school performance. Community Contributions. In terms of contributions to the school and its canteen, the data suggest that, over time, the communities moved from commodity contributions (wood, condiments, water, and soap) to cash contributions. Using the month of March as representative of monthly contribution volume, CRS monitoring data on commodity contributions to schools show that the average contributions declined slightly over time. To arrive at the data shown in Exhibit 31, we calculated the percentage of schools that 5,256 16,692 12,269 127 13,294 541 323 325 - 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 March 2017 March 2018 March 2019 March 2020 Number of SILC Members Women Men IMPAQ International, LLC Page 48 McGovern-Dole III Endline Evaluation Report received any wood, condiments, water, or soap from their community by dividing the number of schools that received more than zero kilograms of commodities in March by the total number of schools in that dataset. The resulting information confirms that contributions of commodities declined not only in total volume, but also in terms of the share of schools that received any commodity resources. Exhibit 32 shows the breakdown by region. Exhibit 31. Average Community Commodities Contributed in March per School, by Commodity Source: CRS monitoring data. N = 245 schools in 2016 – 2017; 244 in 2017 – 2018; 270 in 2018 – 2019; and 267 in 2019 – 2020. Note: Showing single-month output for March as representative of typical monthly contribution amounts. This dataset’s variables contain only values that are greater than zero. Exhibit 32. Average Percentage of Schools Received Community Commodities in March per School, by Region Source: CRS monitoring data, author calculations. N = 117 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 116 in Mopti and 128 in Koulikoro in 2017 – 2018; 140 in Mopti and 130 in Koulikoro in 2018 – 2019; and 139 in Mopti and 128 in Koulikoro in 2019 – 2020. Note: Showing single-month output for March as representative of typical monthly contribution amounts. Commodities were wood, condiments, water, or soap. Exhibit 33 shows that community cash contributions, measured during the first quarter of the calendar year, steadily increased over years. As Exhibit 34 shows, average cash contributions per school in 19,068 18,358 16,124 12,403 41,789 50,194 35,428 28,066 11,340 10,522 8,010 6,496 3,567 3,709 2,931 2,387 0 10,000 20,000 30,000 40,000 50,000 60,000 2016-2017 2017-2018 2018-2019 2019-2020 Commodities Contributed in kg Wood Condiments Water Soap 93% 93% 87% 80% 82% 88% 77% 68% 0% 20% 40% 60% 80% 100% 2016-2017 2017-2018 2018-2019 2019-2020 Schools That Received Commodities Mopti Koulikoro IMPAQ International, LLC Page 49 McGovern-Dole III Endline Evaluation Report Koulikoro were steady during the reporting period, while Mopti saw high volatility, with a sharp decline in the 2018 – 2019 school year. The values in Exhibit 33 represent the weighted average of the regional values presented in Exhibit 34, with the more numerous Koulikoro schools driving the overall trend. The volatility in cash contributions in Mopti could be caused by resource insecurity stemming from civil conflict and violence in the region during 2018. Alternatively, data collected in Mopti could have been inconsistent with when funds were truly delivered to schools, raising general data quality concerns. Exhibit 33. Average Cash Contributed to Schools in January to March per School Source: CRS monitoring data. N = 100 schools in 2016 – 2017; 115 in 2017 – 2018; 112 in 2018 – 2019; 98 in 2019 – 2020. Note: Showing single-quarter output for January to March as representative of typical quarterly contribution amounts. The values shown in this graph represent the weighted average of the values shown by region below. Exhibit 34. Average Cash Contributed to Schools in January to March per School, by Region Source: CRS monitoring data. N = 24 schools in Mopti and 76 in Koulikoro in 2016 – 2017; 34 in Mopti and 81 in Koulikoro in 2017 – 2018; 31 in Mopti and 81 in Koulikoro in 2018 – 2019; 33 in Mopti and 65 in Koulikoro in 2019 – 2020. Note: Showing single-quarter output for January to March as representative of typical quarterly contribution amounts. This shift in type of community contributions is consistent with our midline evaluation results. SMC members stated that parents’ contributions of wood and food to school canteens were not impactful and that the fungible quality of monetary contributions might be more useful. The midline evaluation reported that parental contributions of food had the least impact on improving wellbeing. 30,703 41,206 43,560 50,838 0 10,000 20,000 30,000 40,000 50,000 60,000 2016-2017 2017-2018 2018-2019 2019-2020 Cash Contributions in XOF 15,708 48,632 14,003 57,965 35,438 38,089 54,872 47,219 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 2016-2017 2017-2018 2018-2019 2019-2020 Cash Contributions in XOF Mopti Koulikoro IMPAQ International, LLC Page 50 McGovern-Dole III Endline Evaluation Report In addition to the four commodities cited above, communities also contributed grain to schools. Almost all schools received at least some grain contribution from their local communities, though the amount of contribution declined in 2019 – 2020, as shown in Exhibit 35. This could be because the 2020 school year was not yet complete at the time of this evaluation, and most contributions for April and May of 2020 were listed as zero. Exhibit 36 shows the total grain distribution by year for each region. The proportion of schools that received grains from the community was consistently above 94 percent for all years, with minimal differences between regions. Exhibit 35. Average Annual Amount of Grain Contributed to Schools per School Source: CRS monitoring data. N = 161 schools in 2016 – 2017; 192 in 2017 – 2018; 210 in 2018 – 2019; 165 in 2019 – 2020. Note: The values shown in this graph represent the weighted average of the values shown by region below. Exhibit 36. Average Annual Amount of Grain Contributed to Schools per School, by Region Source: CRS monitoring data. N = 78 schools in Mopti and 83 in Koulikoro in 2016 – 2017; 81 in Mopti and 111 in Koulikoro in 2017 – 2018; 97 in Mopti and 113 in Koulikoro in 2018 – 2019; 76 in Mopti and 89 in Koulikoro in 2019 – 2020. To understand whether the reduced contribution of grains from the community was driven by reduced enrollment, we also compared the average total grain contributed to the number of students enrolled in the school. As shown in Exhibits 37 and 38, the data suggest a decline in the most recent year, with 837 805 866 725 650 700 750 800 850 900 2016-2017 2017-2018 2018-2019 2019-2020 Grain Contributions in kg 1,011 565 706 638 673 980 1,004 799 0 200 400 600 800 1,000 1,200 2016-2017 2017-2018 2018-2019 2019-2020 Grain Contributions in kg Mopti Koulikoro IMPAQ International, LLC Page 51 McGovern-Dole III Endline Evaluation Report regional differences following the same general trend. This could be driven by poor crop conditions that year affecting grain yields, or civil conflict pressuring households to be less willing to contribute their grain to local schools. Interviews with implementing partners suggested that communities in villages far from big roads and cities provided larger contributions to canteens and were more engaged than communities in villages near big roads and cities. Exhibit 37. Average Grain Contributed per Enrolled Student per School Source: CRS monitoring data, attendance data, and author calculations. N = 192 schools in 2017 – 2018; 190 in 2018 – 2019; 163 in 2019 – 2020. Note: Enrollment data was not available for 2016 – 2017. The values shown in this graph represent the weighted average of the values shown by region below. Exhibit 38. Average Grain Contributed per Enrolled Student per School, by Region Source: CRS monitoring data, attendance data, and author calculations. N = 81 schools in Mopti and 111 in Koulikoro in 2017 – 2018; 78 in Mopti and 112 in Koulikoro in 2018 – 2019; 74 in Mopti and 89 in Koulikoro in 2019 – 2020. Note: Enrollment data was not available for 2016 – 2017. 3.9 4.2 3.1 0 1 2 3 4 5 2017-2018 2018-2019 2019-2020 Grain Contribution in kg per Enrolled Child 2.6 3.3 2.6 4.7 4.8 3.5 0 1 2 3 4 5 6 2017-2018 2018-2019 2019-2020 Grain Contribution in kg per Enrolled Child Mopti Koulikoro IMPAQ International, LLC Page 52 McGovern-Dole III Endline Evaluation Report CRS monitoring data on “other” community contributions show that the most prominent and costly contributions to schools were white boards, oil, meat, fish, and teacher salaries. Provision of training to SMCs. SMCs received training to provide sensitization to parents and community members on the importance of education. More specifically, SMC members received training on school management, development of an action plan, management of food commodities, food preparation and hygiene, and sensitization for parents on the importance of education. Interviewed SMC members described finding the training to be useful, for example, in operating the canteen, following hygiene rules in the kitchen, and providing good meals. Stakeholders said that the training helped build SMC capacity to, for example, negotiate with local governments for operating funds. SMC training on sensitization allowed SMCs to sensitize parents and community members on the importance of education, according to MONE officials and community stakeholders. Challenges shared included SMC members’ low literacy, which limited their ability to complete administrative forms. SMCs also mobilized communities to provide funds or in-kind support for canteens; in some cases, they were able to renovate school infrastructure. Stakeholders recommended providing follow-up support for SMCs, and implementers noted that providing competitive grants improved performance by providing motivation and awarding SMCs that were functioning well. Parents’ engagement in children’s schooling. One project activity to engage parents in their children’s school was distributing colored report cards. The purpose was to engage all parents, even those who might not have good literacy skills, to keep abreast of their child’s academic performance. The colored report cards tracked performance with the following colors: red for well below average, purple for below average, yellow for average, and green for doing well. Parents could sign and return the colored report cards to the schools to acknowledge receipt. According to interviews with project and partner staff, the distribution of colored report cards led parents to encourage increased school attendance for their children. However, monitoring data did not suggest any strong association between the colored report cards and attendance or enrollment. Any proposed correlation should be interpreted with caution. To understand this association better, data must be collected from a representative sample of parents who received and signed the cards. During baseline and midline evaluations, we asked SMC members and parents directly about their attitudes toward and conceptions of schooling; we then reported on these attitudes as an outcome indicator. In the absence of primary data, we were not able to construct the same indicator. However, we explored SMC and school principal data collected by EDC to understand these stakeholders’ perceptions of community attitudes. We complemented this information by asking SMC and SILC members in KIIs about their perceptions of parent and community attitudes toward education. EDC asked SMC members if they believed knowing how to read was more important for boys than girls. The percentage of SMC members who thought it was more important for boys to know how to read than girls declined from 39 percent at the end of the 2015 – 2016 school year to 5 percent by the end of 2018 “By training the SMC members we give them the capacity to advocate for funding to get the subsidies that the government allocates for schools.” –Project staff member IMPAQ International, LLC Page 53 McGovern-Dole III Endline Evaluation Report – 2019. Exhibit 39 shows that there was a slightly larger decline of 35 percentage points in Koulikoro compared to 29 percentage points in Mopti. Exhibit 39. SMC Members Who Believe That Knowing How to Read is More Important for Boys than for Girls Source: SMC survey; authors’ calculations. 17 in Mopti and 22 in Koulikoro in June 2016; 17 in Mopti and 22 in Koulikoro in 2016 – 2017; 17 in Mopti and 22 in Koulikoro in 2017 – 2018; and 18 in Mopti and 24 in Koulikoro in 2018 – 2019. In addition, to encourage girls and recognize their efforts in the local community, the project organized a pilot competition in Nara’s six urban schools on June 20, 2018. This initiative was introduced to promote girls reading where education barriers are a hindrance to their development, specifically for girls. In interviews, project and partner staff pointed to this initiate as a successful activity that highlighted the importance of education for girls in the community, which raised their awareness on girls’ retention at school. Increased Enrollment (MGD 1.3.4) We examined student enrollment data collected by CRS for the 2016 – 2017 through 2019 – 2020 school years. As shown in Exhibit 40, the average enrollment per school rose slightly during the project, from 257 to 278 in Mopti and from 262 to 275 in Koulikoro. The data show that slightly more boys than girls in Koulikoro and slightly more girls than boys in Mopti were in enrolled in schools. These gender trends are consistent over time. Exhibit 40 also shows that the higher level of enrollment was maintained over years, and the project achieved its planned target. The slight fluctuation, especially in Mopti, could be related to school closures and reopening as mentioned in performance reports. The finding on steady increased enrollment is supported by interviews with education officials and SMC members in Koulikoro. They noted that increased student enrollment in schools resulted in a need to build new classrooms, open a new secondary school, and provide more plates for hot meals at the canteen to serve the additional students. One principal in Mopti said that, after the school opened a canteen, student enrollment increased from 150 to 527 students. Stakeholders and implementers cited the role of school canteens, THRs, sensitization, and improvements in literacy in increasing student enrollment. For example, one education official in 35% 18% 24% 6% 41% 23% 18% 4% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 2015-2016 (June) 2016-2017 (June) 2017-2018 (June) 2018-2019 (June) Mopti Koulikoro IMPAQ International, LLC Page 54 McGovern-Dole III Endline Evaluation Report Koulikoro said that parents, seeing that one child learned to read earlier with BLA, were motivated to enroll their other children in school as well. Exhibit 40. Average School Enrollment, by Year by Region Source: CRS attendance data; authors’ calculations. N = 114 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 111 in Mopti and 128 in Koulikoro in 2017 – 2018; 138 in Mopti and 130 in Koulikoro in 2018 – 2019; and 132 in Mopti and 127 in Koulikoro in 2019 – 2020. Food- and health-related project activities might have played an important role in the decision-calculus of whether or not students were enrolled (Levinger, 2006; Ahmed et al., 2002). To understand the forces driving higher enrollment, we used CRS monitoring data to examine the relationship between student enrollment and project interventions to provide hot meals, THRs, vitamin A, and deworming pills. Exhibits 41 through 44 present enrollment counts for each student’s gender at a given school along the y-axis, and one of the aforementioned four indicators along the x-axis. Each dot represents a single school. A line of best fit is also provided in red to show whether enrollment had a positive, neutral, or negative association with the corresponding x-axis indicator. The data shown in Exhibits 41 and 42 seem to suggest that higher enrollment is associated with more food being distributed at school in the form of hot meals and THRs. These exhibits show along the x-axis the number of hot meals distributed in each school and the number of students who received THRs, respectively. One possible explanation is that parents enrolled their children with the intent of receiving these benefits. When comparing female and male enrollment against the yearly total of hot meals distributed to girls and boys in each school, we observe clearly positive relationships. 112 139 118 144 118 144 130 147 145 123 144 123 148 129 148 129 257 262 261 268 265 273 278 275 0 50 100 150 200 250 300 350 Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro 2016-2017 2017-2018 2018-2019 2019-2020 Enrollment Count Boys Girls IMPAQ International, LLC Page 55 McGovern-Dole III Endline Evaluation Report Exhibit 41. Association of Hot Meals Distributed at Schools and Enrollment by Gender, 2019 Source: CRS attendance data and monitoring data; authors’ calculations. N = 263 schools. Exhibit 42. Association of Students who Received Take-Home Rations and Enrollment by Gender, 2018 Source: CRS attendance data and monitoring data; authors’ calculations. N = 222 schools. Note: Showing the most recent data available in the THR dataset for 2018 – 2019 school year. Similarly, the distribution of vitamin A and deworming pills suggest a positive relationship with enrollment for both genders, as shown in Exhibits 43 and 44. Both vitamin A and deworming pills were positively associated with higher enrollment counts. However, we cannot make any causal conclusion here, as there could be other reasons for higher enrollment. A number of schools had high enrollment without receiving any deworming pills. IMPAQ International, LLC Page 56 McGovern-Dole III Endline Evaluation Report Exhibit 43. Association of Deworming Pills Distributed at Schools and Enrollment by Gender, 2019 Source: CRS attendance data and monitoring data; authors’ calculations. N = 99 schools. Exhibit 44. Association of Vitamin A Pills Distributed at Schools by Enrollment and Gender, 2017 Source: CRS attendance data and monitoring data; authors’ calculations. N = 236 schools. Having discussed intermediate outcomes that could affect student attendance, we now look at attendance monitoring data collected by CRS and discuss potential predictors of changes in the outcomes. Exhibit 45 shows the distribution of schools in the attendance sample by region and year. The schools are nearly evenly split between the two regions. Exhibit 45. Schools in the Student Attendance Sample Region 2016 – 2017 2017 – 2018 2018 – 2019 2019 – 2020 Mopti 114 111 132 132 Koulikoro 128 128 127 127 Total 242 239 268 259 Source: CRS student attendance data; authors’ calculations. We calculated student attendance rates by dividing the number of days that students attended school by the total number of days in the school year. As shown in Exhibit 46, the attendance rate ranged from an IMPAQ International, LLC Page 57 McGovern-Dole III Endline Evaluation Report overall percentage of 87-91 percent each year, with few differences between regions or genders. CRS monitoring data also showed that average number of days missed due to health issues dropped, achieving its project target. Although the attendance rate was steady throughout the years, schools were disrupted and closed during teacher strikes and political instability in the 2017 – 2018 and 2018 – 2019 school years and because of COVID-19 in the last six months of the project. While these closures do not affect the attendance rate, they do affect the amount of schooling received by students each year. For example, due to COVID-19, the 2019 – 2020 school year was shortened; schools were open for an average of only 53 days. Nearly all interview respondents said that they had seen student attendance increase in project schools. Exhibit 46. Student Attendance Rate by Year by Region Source: CRS attendance data; authors’ calculations. N = 114 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 111 in Mopti and 128 in Koulikoro in 2017 – 2018; 138 in Mopti and 130 in Koulikoro in 2018 – 2019; and 132 in Mopti and 127 in Koulikoro in 2019 – 2020. Consistent with steady low absence rate, proportion of students who attended school regularly remained high and steady over years showing the project achieved its attendance target for students. Exhibit 47 shows the available attendance data by month. A few dips are apparent. As reported at midline, the drop in October to December 2016 might be explained by security concerns or by inclement weather that prevented students from traveling to school. It may also have been that canteens were not yet fully operational so that children were not yet incentivized to attend school for a meal. Both December 2017 and April 2019 saw large drops in student attendance, which rebounded the next month, which could be explained by temporary school closures for security reasons. In addition, consistent with what SMC members reported on teacher attendance, these drops are likely a result of teacher absenteeism given that their strikes might have caused school disruption (EDC, 2019) (GardaWorld, 2019). According to 89% 88% 89% 90% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro Mopti Koulikoro Boys Girls Overall 2016-2017 2017-2018 2018-2019 2019-2020 IMPAQ International, LLC Page 58 McGovern-Dole III Endline Evaluation Report performance report (October 2018-Mrach 2019), during strikes, only schools with community-paid teachers continued to operate normally in the most secure zones. Exhibit 47. Proportion of Students Who Attended Regularly, by Year by Region Source: CRS attendance data; authors’ calculations. N = 114 schools in Mopti and 128 in Koulikoro in 2016 – 2017; 111 in Mopti and 128 in Koulikoro in 2017 – 2018; 138 in Mopti and 130 in Koulikoro in 2018 – 2019; and 132 in Mopti and 127 in Koulikoro in 2019– 2020. a Regular attendance is defined as attending at least 80 percent of the time. It is unclear if this takes into account days when the school was officially closed. Note: Data were collected only through April in 2017 – 2018 and through January in 2019 – 2020. Qualitative findings also confirmed the low steady student absence rate. A national government stakeholder noted attendance rate increases, especially for girls. Other stakeholders mentioned that addressing barriers such as household chores and early marriage helped with increasing student attendance. SMC members and project and partner staff also suggested that using attendance registers to track attendance and then following up with parents of children who were absent played a role in increasing student attendance. SMCs also promoted student school attendance through sensitization. As we did for enrollment, we examined the associations between student regular attendance (if students attended 80 percent of school days) and distribution of hot meals, THRs, vitamin A, and deworming medications. As before, each dot represents a single school, with the line of best fit shown in red. Exhibits 48, 49, and 50 demonstrate that high attendance, for both female and male students, is positively 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% October December February April June October December February April June October December February April June October December February April June 2016-2017 2017-2018 2018-2019 2019-2020 Mopti Koulikoro IMPAQ International, LLC Page 59 McGovern-Dole III Endline Evaluation Report associated with the number of hot meals and THRs distributed at schools. This positive relationship was expected given that receipt of hot meals and THRs was contingent on school attendance. This finding was supported in interviews with principals, teachers, and project and partner staff, who said that THRs were a key factor in increasing student attendance, especially because only children with high attendance rates were eligible to receive THRs. One teacher in Mopti emphasized that THRs provided economic value that incentivized parents to make sure their children attended school to receive THRs, which reduced household expenses. Nearly all respondents cited the school canteen as a major factor increasing student attendance. Stakeholders noted that many children live far away from school and typically return home for lunch; they may not return for afternoon classes. With the school canteen open, students can stay at school and not miss any classes. Also, project and partner staff added that a meal is guaranteed at the school canteen, but there may not be enough food at home for lunch. Exhibit 48. Association of Hot Meals Distributed at School and Regular Attendance by Gender, January 2018 Source: CRS attendance data and monitoring data; authors’ calculations. N = 211 schools. Note: Female/Male Attendance refers to the number of students who attended at least 80% of school days. Exhibit 49. Association of Hot Meals Distributed at Schools and Regular Attendance by Gender, January 2019 IMPAQ International, LLC Page 60 McGovern-Dole III Endline Evaluation Report Source: CRS attendance data and monitoring data; authors’ calculations. N = 240 schools. Note: Female/Male Attendance refers to the number of students who attended at least 80% of school days. Exhibit 50. Association of Students who Received Take-Home Rations and Regular Attendance by Gender, January 2018 Source: CRS attendance data and monitoring data; authors’ calculations. N = 170 schools. Note: Female/Male Attendance refers to the number of students who attended at least 80% of school days. Showing the most recent data available in the THR dataset for 2018 – 2019 school year. We observe a similarly positive relationship in Exhibits 51 and 52 between high attendance rates and distribution of vitamin A and deworming pills. The distributions of both types of health resources shared a positive relationship with regular attendance for both female and male students. However, for all of these associations, the simple regressions pictured in scatterplots do not allow for causal interpretation. Other factors could have influenced the associations. IMPAQ International, LLC Page 61 McGovern-Dole III Endline Evaluation Report Exhibit 51. Association of Vitamin A Pills Distributed at School and Regular Attendance by Gender, October – December 2017 Source: CRS attendance data and monitoring data; authors’ calculations. N = 236 schools. Note: Data on vitamin A distribution were available only for the 2016 – 2017 and 2017 – 2018 school years. Female/Male Attendance refers to the number of students who attended at least 80% of school days. Exhibit 52. Association of Deworming Pills Distributed at School and Regular Attendance by Gender, January 2019 Source: CRS attendance data and monitoring data; authors’ calculations. N = 86 schools, where data were available. Note: Female/Male Attendance refers to the number of students who attended at least 80% of school days. In addition to student attendance, stakeholders also discussed student retention and dropout rates in interviews. Education officials and community stakeholders (principals, teachers, and SMC members) shared that girls especially were staying in school and not dropping out. They noted that the project provided sensitization and worked to address barriers to education for girls by, for example, advocating for girls to stay in school even after early marriage. As for student attendance, community stakeholders and local education officials stressed the importance of school canteens and THRs in promoting student retention. IMPAQ International, LLC Page 62 McGovern-Dole III Endline Evaluation Report SO1 Improved Literacy of School-Age Children So far, this section has discussed factors that contribute to the McGovern-Dole SO1 (improved literacy of school-aged children). McGovern-Dole III posits that if the literacy instruction quality is enhanced in addition to an improvement in student enrollment, attendance, and attentiveness, then students will achieve better learning outcomes. In this section, we provide findings from EGRA instrument collected from students in Grade 1 and 2 by EDC, 20 complemented by qualitative interviews with education officials, community stakeholders (teachers, principals, and SMC members), and project and partner staff. The EGRA was administered in French in classic schools and in Bamanankan in bilingual schools. EDC collected EGRA data on Grade 1 students at three different times, including two in the 2015 – 2016 school year: December 2015, June 2016, and June 2018. For Grade 2 students, EDC collected data in June 2017 and June 2019. In order to make meaningful comparisons, we conducted descriptive analysis together with t-tests using June data over the years for both Grade 1 (between 2016 and 2018) and Grade 2 (between 2017 and 2019). For simplicity, we labeled the EGRA that was collected in earlier years (2016 for G1 and 2017 for G2) as “EGRA administration 1” and “EGRA administration 2” for assessments that were conducted later in time (2018 for G1 and 2019 for G2) throughout this section. Exhibit 53 shows the number of schools and students represented by region and curriculum. The numbers are fairly even by region, with a slightly higher proportion of schools and students coming from Koulikoro. In terms of curriculum, almost all of the schools are classic, rather than bilingual, schools. As noted above, while 79-80 schools were surveyed by EDC each year, our analysis only includes the 42 schools that have data at both the beginning and end of the project. Exhibit 53. EGRA Sample by Region and Curriculum Characteristic 2015 – 2016 (Dec.) 2015– 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Grade Grade 1 Grade 1 Grade 2 Grade 1 Grade 2 Region Mopti Schools 18 18 18 17 18 Students 254 250 248 228 259 Koulikoro Schools 24 24 23 23 24 Students 337 334 313 323 326 Curriculum Bilingual Schools 4 4 4 4 4 Students 60 60 60 75 61 20 EDC also collected data from Grade 3 students. However, we had to exclude them from the analysis as these data were collected only once, in June, 2018, and we were not able to make comparisons over time. IMPAQ International, LLC Page 63 McGovern-Dole III Endline Evaluation Report Characteristic 2015 – 2016 (Dec.) 2015– 2016 (June) 2016 – 2017 2017 – 2018 2018 – 2019 Grade Grade 1 Grade 1 Grade 2 Grade 1 Grade 2 Classic Schools 38 38 37 36 38 Students 531 524 501 476 524 Total Schools 42 42 41 40 42 Students 591 584 561 551 585 Source: student survey; authors’ calculations. Exhibit 54 shows the breakdown of the student literacy sample by grade and gender. The number of students in the sample each year ranges from 551 to 591. The proportions of girls and boys were fairly even throughout the years. Exhibit 54. Student Literacy Sample by Grade and Gender Year Male Female Total Percent Number Percent Number Grade 1 2015 – 2016 (Dec.) 51% 303 49% 288 591 2015 – 2016 (June) 51% 296 49% 288 584 2017 – 2018 54% 296 46% 255 551 Grade 2 2016 – 2017 53% 295 47% 266 561 2018 – 2019 50% 290 50% 295 585 Source: EGRA Instrument; authors’ calculations. To measure changes in the literacy outcomes, we followed the three main literacy indicators required by the approved McGovern-Dole III performance monitoring plan: 1. Decoding proficiency – A student’s ability to read at least six (Grade 1) or 12 (Grade 2) invented words out 100 in one minute that were given to students 2. Reading proficiency at the national standard level – A student’s ability to read at least 20 words (Grade 1) or 31 words (Grade 2) aloud from a passage in one minute 3. Reading proficiency with comprehension – A student’s ability to read proficiently at the national standard level and answer at least three (Grade 1) or four (Grade 2) comprehension questions about a passage Decoding Proficiency With support from EDC and grade level standards, we defined students passing the decoding proficiency outcome if they correctly read aloud at least six (Grade 1) or 12 (Grade 2) invented words out of 50. Exhibit 55 shows the progress of Grade 1 and Grade 2 students from 2015 to 2018, disaggregated by region. At the end of the 2015 – 2016 school year, 11 percent of Grade 1 students met the decoding standard. This number increased to 18 percent by the end of the 2017 – 2018 year, a significant difference at the 1 percent level, mostly driven by increase in means of outcomes in Koulikoro. Education officials, community stakeholders (teachers, principals, and SMC members), and project and partner staff IMPAQ International, LLC Page 64 McGovern-Dole III Endline Evaluation Report supported these findings in interviews, noting successes with student decoding capacity and emphasizing that children were learning to read in Grade 1, and learning to read faster, with BLA techniques. For Grade 2 students, the percentage of students who could decode at least 12 words decreased from 26 percent at the end of the 2016 – 2017 school year to 20 percent at the end of 2018 – 2019. The total decrease for both regions was not statistically significant, but the drop from 21 to 8 percent in Mopti was significant at the 1 percent level. The decline in decoding for Grade 2 students may be explained by the string of teacher strikes in 2018 – 2019, which meant the students had less instructional time. Exhibit 55. Decoding Proficiency by Grade and Region Source: student assessment; authors’ calculations. N = 254 schools in Mopti and 337 in Koulikoro in Dec. 2015; 250 in Mopti and 334 in Koulikoro in June 2016; 248 in Mopti and 313 in Koulikoro in 2016 – 2017; 228 in Mopti and 323 in Koulikoro in 2017 – 2018; and 259 in Mopti and 326 in Koulikoro in 2018 – 2019. Exhibit 56 shows that Grade 1 boys and girls saw similar significant increases in reading. While Grade 2 boys passed the decoding component at similar levels in both years of the assessment, the percentage of girls who passed decreased from 29 percent in the first year to 17 percent in the second year, a difference that is significant at the 1 percent level. Exhibit 56. Decoding Proficiency by Gender Gender and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent Number Percent Number (p-value) Boys Grade 1 11% 296 18% 296 6%** (0.03) Grade 2 23% 295 22% 290 –1% (0.78) Girls Grade 1 12% 288 19% 255 7%** (0.02) Grade 2 29% 266 17% 295 –11%*** 0% 17% 9% 21% 8% 0% 7% 25% 30% 29% 0% 5% 10% 15% 20% 25% 30% 35% 40% 2015 - 2016 Dec. 2015-2016 June 2017 - 2018 2016 - 2017 2018 - 2019 Students Meeting the Decoding Standard (%) Mopti Koulikoro Grade 1 Grade 2 IMPAQ International, LLC Page 65 McGovern-Dole III Endline Evaluation Report Gender and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent Number Percent Number (p-value) (0.00) Source: EGRA Instrument; authors’ calculations. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01.; standard errors are clustered at the school level Exhibit 57 shows decoding proficiency results for bilingual and classic schools. Only the Grade 1 students in classic schools saw a gain that is statistically significant at the 1 percent level. Among Grade 2 students in bilingual schools, the percentage of students who met the decoding standard decreased from 60 percent to 21 percent, which is significant at the 1 percent level. The slight decrease in the percentage of students in classic schools who met the decoding proficiency standard is not statistically significant. However, we should interpret these results with caution due to a very small number of bilingual schools in the sample. Exhibit 57. Decoding Proficiency by Curriculum Curriculum and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent Number Percent Number (p-value) Bilingual Grade 1 17% 60 20% 75 3% (0.62) Grade 2 60% 60 21% 61 –39%*** (0.00) Classic Grade 1 11% 531 18% 476 7%*** (0.00) Grade 2 22% 501 20% 524 –2% (0.41) Source: EGRA Instrument; authors’ calculations. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. Reading Proficiency at National Standard Level To measure reading proficiency as defined by MONE, enumerators asked students to read a 50-word passage aloud and timed it. We considered students to be proficient readers by national standards if they can read at least 20 words in Grade 1 or 31 words from a passage in Grade 2 in one minute. Exhibits 58 and 59 show the number of words read by Grade 1 and 2 students from a passage in one minute by region. The passing threshold defined by MONE is represented by a red vertical dotted line. No students could read at grade level at the beginning of the project. There was improvement over time, but most students were below the thresholds for their grade level at the second EGRA administration (2017 for Grade 1, 2019 for Grade 2), indicating that they had limited reading fluency. However, the students were in line with the project targets of 12 and 20 percent at midline and endline, respectively. IMPAQ International, LLC Page 66 McGovern-Dole III Endline Evaluation Report Exhibit 58. Number of Words Read from Passage by 1st Graders Source: student assessment; authors’ calculations. N = 322 in Mopti and 269 in Koulikoro for Dec. 2015; 319 in Mopti and 265 in Koulikoro for June 2016; and 308 in Mopti and 243 in Koulikoro for 2017 – 2018. Grade 2 students saw little change in reading proficiency at national standard level from the 2016 – 2017 to 2018 – 2019 school years. In fact, as shown in Exhibit 59, the percentage of students who could not read any words increased between the two EGRA administrations, particularly in Mopti, where the proportion more than tripled from 19 percent to 65 percent. Although teachers’ strikes and/or political instability might have affected some of these changes, especially in Mopti, it would require further research to understand these significant regional differences for future programming. In addition to comparing the Grade 2 students between 2016 – 2017 and 2018 – 2019 school years, we can examine the same cohort, one year later. In other words, we can compare the Grade 2 reading levels relative to where they were when they were in Grade 1 the previous year. Among Grade 2 students, even in Mopti, from the end of Grade 1 to the end of Grade 2, zero scores declined and the percentage in every other tranche up to 30-39 words increased. And these modest improvements occurred even though the Grade 1 scores for this cohort were already higher than those in 2015 – 2016. This evidence suggests that the second cohort of students continued to make progress – just not nearly as much progress as the first cohort that moved from Grade 1 in 2015 – 2016 to Grade 2 in 2016 – 2017. The difference between the progress of these cohorts would indicate some rather drastic difference in the implementation context between 2016 – 2017 and 2018 – 2019, which has been identified as mainly school closures resulting from teacher strikes (and insecurity in Mopti). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mopti Koulikoro 2015-2016 (Dec) 2015-2016 (June) 2017-2018 Grade 1 reading level IMPAQ International, LLC Page 67 McGovern-Dole III Endline Evaluation Report Exhibit 59. Number of Words Read from Passage by 2nd Graders Source: student assessment; authors’ calculations. N = 313 schools in Mopti and 248 in Koulikoro for 2017 – 2018; 311 in Mopti and 274 in Koulikoro for 2018 – 2019. Overall, Grade 1 reading proficiency increased by 2 percentage points by the end of the 2015 – 2016 school year and by 5 percentage points at the end of the 2017 – 2018 school year, compared to the beginning of the project. These increases are statistically significant at the 5 and 1 percent levels, respectively. The percentage of Grade 1 students who met the reading proficiency standard increased significantly (at the 1 percent level), from 2 percent in June 2016 to 5 percent in June 2018. Exhibit 60 shows the changes in reading proficiency at the national standard level over time by region. The percentage of Grade 1 students who met the reading proficiency standard increased significantly (at the 1 percent level), from 2 percent in June 2016 to 5 percent in June 2018. The percentage of Grade 2 students who met the standard decreased in both regions, but neither change was statistically significant. However, there were large regional disparities for Grade 1 students in 2017 – 2018 and Grade 2 students in both assessment years, with students in Koulikoro much more likely to meet the standard. This finding was supported by interviews in which education officials and teachers in Koulikoro described improvements in reading fluency. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 1-9 10-19 20-29 30-39 40-50 0 1-9 10-19 20-29 30-39 40-50 2016-2017 2018-2019 Mopti Koulikoro Grade 2 reading level IMPAQ International, LLC Page 68 McGovern-Dole III Endline Evaluation Report Exhibit 60. Students Meeting the Reading Proficiency Standard, by Region and Grade Source: student assessment; authors’ calculations. N = 254 schools in Mopti and 337 in Koulikoro in Dec. 2015; 250 in Mopti and 334 in Koulikoro in June 2016; 248 in Mopti and 313 in Koulikoro in 2016 – 2017; 228 in Mopti and 323 in Koulikoro in 2017 – 2018; and 259 in Mopti and 326 in Koulikoro in 2018 – 2019. We disaggregated the outcomes by curriculum. As shown in Exhibit 61, the percentage of Grade 1 students in classic schools who met the reading proficiency standard increased from 2 percent to 5 percent, a significant difference at the 1 percent level. However, the data did not show any improvements for Grade 1 students. At both assessments, five percent of Grade 1 students in bilingual schools met the national standard. Consistent with the decoding outcomes, the percentage of Grade 2 students in bilingual schools who met the reading proficiency standard fell dramatically, from 33 percent to 10 percent, a difference that is significant at the 1 percent level. For Grade 2 students in classic schools, there was no change between the first and second assessment. Although, these outcomes should be interpreted with caution given the small sample of bilingual schools, this area needs further research to understand the systematic differences between school curriculums for future programming. Exhibit 61. Reading Proficiency by Curriculum Curriculum and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent Number Percent Number (p-value) Bilingual Grade 1 5% 60 5% 75 0% (0.93) Grade 2 33% 60 10% 61 –24%*** (0.00) Classic Grade 1 2% 531 5% 476 3%*** (0.01) Grade 2 12% 501 12% 524 0% (0.94) 0% 4% 1% 8% 2% 0% 1% 8% 20% 20% 0% 5% 10% 15% 20% 25% 2015 - 2016 (Dec.) 2015-2016 (June) 2017 - 2018 2016 - 2017 2018 - 2019 Mopti Koulikoro Grade 1 Grade 2 IMPAQ International, LLC Page 69 McGovern-Dole III Endline Evaluation Report Source: EGRA Instrument; authors’ calculations. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. Students’ reading proficiency changes are roughly similar when the data are disaggregated by gender for both Grade 1 and Grade 2 students. Reading Proficiency with Comprehension The final key outcome, reading proficiency with comprehension, was measured using the reading proficiency score in addition to the number of comprehension questions answered correctly. Grade 1 students were considered to have passed if they could read at least 20 words and answer three comprehension questions. Grade 2 students who could read at least 31 words and answer four comprehension questions were considered to have passed. The results from the EGRA data show some improvementsfor Grade 1 students but no increases for Grade 2 students in reading comprehension. More specifically, the percentage of Grade 1 students who passed the reading proficiency with comprehension increased from one percent to two percent, a statistically significant increase at the 10 percent level. Grade 2 students saw a decrease from ten percent to nine percent; however, the change is not statistically significant. Observing no changes in Grade 2 students’ outcomes could be explained by lack of instructional time due to teachers’ strikes in 2018-2019. Reading proficiency with comprehension by region tracked very similarly to reading proficiency. As shown in Exhibit 62, the percentage of Grade 1 students who passed increased from 1 percent in June 2015 – 2016 to 2 percent in 2017 – 2018, a significant increase at the 10 percent level. The small changes in Grade 2 reading proficiency were not statistically significant in either region. While the changes by region were similar over time, reading comprehension for Grade 2 students was much higher in Koulikoro than in Mopti in both 2016 – 2017 and 2018 – 2019. Exhibit 62. Reading Proficiency with Comprehension by Region and Grade Region and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent (p-value) Number of Students Percent Number of Students Mopti Grade 1 2% 250 0% 228 –2% (0.21) Grade 2 4% 248 2% 259 –2% (0.16) Koulikoro Grade 1 0% 334 3% 323 3%*** (0.00) Grade 2 14% 313 14% 323 0% (0.96) Overall Grade 1 1% 584 2% 551 1%* (0.08) Grade 2 10% 561 9% 582 –1% (0.54) Source: EGRA Instrument; authors’ calculations.* p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. IMPAQ International, LLC Page 70 McGovern-Dole III Endline Evaluation Report Similar to the decoding and reading proficiency outcomes, Exhibit 63 shows a substantial decrease among Grade 2 students in bilingual schools (25 percentage points, a significant difference at the 1 percent level). The increase among Grade 2 students in classic schools is not statistically significant. Again, we are not able to make a strong conclusion based on curriculum disaggregation given their sample in EGRA data. Exhibit 63. Reading Proficiency with Comprehension by Curriculum Curriculum and Grade EGRA Administration 1 EGRA Administration 2 Difference in Means Percent (p-value) Number of Students Percent Number of Students Bilingual Grade 1 0% 60 3% 75 3% (0.21) Grade 2 30% 60 5% 58 –25%*** (0.00) Classic Grade 1 1% 531 2% 476 1% (0.18) Grade 2 7% 501 9% 524 2% (0.94) Source: EGRA Instrument; authors’ calculations.* p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. For both Grade 1 and Grade 2 students, there were no significant differences by gender in reading proficiency with comprehension. It seems that some of these low performance levels were identified by the project, because the project added a tutoring activity to the project later on to address this concern. After noting that some students in first-cycle project schools had low academic performance levels, SMC members suggested that volunteer teachers support these students, but the SMCs did not have enough resources to provide incentives. CRS and EDC coordinated to implement volunteer tutoring. In 27 secondary schools in Mopti, the project implemented an activity to select the best students to tutor younger students in first-cycle schools. In return, the 268 volunteer older students received THRs after completing at least 20 tutoring sessions. These rations were available because large quantities of commodities were in storage due to teacher strikes. Overall, 1,521 students in Grades 1 to 3 participated in the tutoring activity. Partners noted that this activity worked well. According to the project’s evaluation tests, 87 percent of students in the reached the average reading score for their grade at the end of the activity. Furthermore, these results seem to be in contradiction with our midline report outcomes, which showed significant large improvements in the percentage of students who were able to read at their grade level from 2016 to 2018. There are two important notes to consider with that regard. First, at midline, we compared Grade 2 students in the 2015 – 2016 and 2017 – 2018 school years. Now we are comparing the 2016 – 2017 and 2018 – 2019 school years. This could suggest that there may have been large improvements after the first year and a plateau thereafter probably due to receiving less instructional time. However, more importantly, we used ASER during the baseline and midline evaluations, which measured whether students were meeting the literacy standards of their grade levels (i.e., ability to read simple sounds for Grade 1 and decode simple words for Grade 2). This might suggest that young children, IMPAQ International, LLC Page 71 McGovern-Dole III Endline Evaluation Report who might not be able to read a passage with comprehension, could still do better on the foundational reading skills. Thus, we examined other sub-tests of EGRA that focused on foundational reading skills to provide more background information. Those seven sub-tests include: 1. Phonetic awareness – The number of initial sounds a student could identify orally out of 10. For example, an enumerator might say the word “soup” and the correct response would be the sound “sssss” 2. Alphabet knowledge – The number of graphemes identified in one minute, out of 10021 3. Word recognition – The number of familiar, disconnected words read aloud in one minute, out of 10 4. Decoding ability – The number of invented words read aloud in one minute, out of 50 5. Oral reading fluency – The number of words read correctly from a passage in one minute, out of 50 6. Reading comprehension – The number of reading comprehension questions based on the reading fluency passage answered correctly, out of 6 7. Listening comprehension – The number of listening comprehension questions answered correctly based on a passage read aloud by an enumerator, out of 6 We compared these sub-tests, from the first to the second administration, for Grade 1 and Grade 2 students. To allow for easy comparisons across sub-tests, we measured the percentage of correctly 1) identified sounds (phonetic awareness); 2) recognized letters (alphabet knowledge); 3) read common used (word recognition); 4) pronounced invented words (decoding ability); 5) read words in a passage (oral reading fluency); and 6) answered comprehension questions after reading (reading comprehension) a passage and 7) answered comprehension questions after listening to a passage (listening comprehension). Consistent with the three key outcome indicators described above, as Exhibit 64 shows, Grade 1 students showed significant improvements in all seven foundational reading skills. The largest improvements came in students’ alphabet knowledge (8 percentage points) and listening comprehension (6 percentage points). Students had the smallest gains in reading comprehension, at one percentage point, but this difference is still significant at the 1 percent level. 21 A grapheme is the smallest unit of writing. It can be a single letter or a short combination of letters that make a sound such as “ph”. IMPAQ International, LLC Page 72 McGovern-Dole III Endline Evaluation Report Exhibit 64. Grade 1 Performance on Foundational Reading Skills Reading Skills EGRA Administration 1 EGRA Administration 2 Difference in Means (p-value) Percent Number of Students Percent Number of Students Phonological awareness 55% 584 60% 551 5%* (0.06) Alphabet knowledge 19% 584 27% 551 8%*** (0.00) Word recognition 5% 584 8% 551 3%*** (0.00) Decoding ability 3% 584 5% 551 2%*** (0.00) Oral Reading fluency 3% 584 6% 551 3%*** (0.00) Reading comprehension 2% 584 3% 551 1%*** (0.00) Listening comprehension 21% 571 27% 551 6%*** (0.00) Source: EGRA Instrument; authors’ calculations.* p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. Exhibit 65 shows the differences in the performance of Grade 2 students on the seven early literacy skills from EGRA between the 2016 – 2017 and 2018 – 2019 school years. Three of the sub-tests decreased by significant margins: phonological awareness, oral reading fluency, and listening comprehension. The only positive significant increase was the percentage of students who answered reading comprehension questions correctly, which increased from 19 percent to 32 percent. These changes explain the lack of improvements in reading proficiency with comprehension for Grade 2 students. In addition to decreases in their reading proficiency, Grade 2 students were still not able to answer 67 percent of reading comprehension questions correctly (four questions out of six) to “pass.” Exhibit 65. Grade 2 Performance on Foundational Reading Skills Reading Skills EGRA Administration 1 EGRA Administration 2 Difference in Means (p-value) Percent Number of Students Percent Number of Students Phonological awareness 80% 561 77% 584 –3%* (0.09) Alphabet knowledge 38% 561 39% 585 1% (0.55) Word recognition 19% 561 17% 585 –2% (0.15) Decoding ability 15% 561 13% 585 –2% (0.22) Oral Reading fluency 27% 561 19% 585 –8%*** (0.00) Reading comprehension 19% 561 32% 582 13%*** (0.00) Listening comprehension 39% 561 12% 556 –27%*** IMPAQ International, LLC Page 73 McGovern-Dole III Endline Evaluation Report Reading Skills EGRA Administration 1 EGRA Administration 2 Difference in Means (p-value) Percent Number of Students Percent Number of Students (0.00) Source: EGRA Instrument; authors’ calculations. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors are clustered at the school level. However, observing lack of improvements for Grade 2 students should be interpreted with caution as mentioned earlier, and might not be attributed to inefficiency of BLA. Recall that at midline, we found that exposure to BLA-trained teachers for two and three years led to improvements in children’s literacy, as measured by their performance on reading scores. The endline evaluation analysisfor Grade 1 students further bolstered these findings about the efficacy of the training and contributed to scaling-up of BLA training further. These findings are still aligned with qualitative interviews. According to education officials, principals, and teachers shared that children learned to read more quickly with BLA and had improved decoding capacity. Teachers and education officials in Koulikoro noted improved fluency for students taught with BLA techniques. Teachers in Mopti and Koulikoro shared examples of increased reading levels, for example, that some children could read better than children three or four grades above them. Principals perceived improved performance levels related to BLA. Findings for Grade 2 students show that the external effects might have outweighed all the effort that project has done to maintain the effect of BLA, especially in 2018-2019. However, still we should interpret the comparison results between students’ outcomes in a normal year with no challenges and a year with approximately 64 days of instructional loss for students with caution. MGD 2.1 Improved Knowledge of Safe Food Preparation and Storage Practices The McGovern-Dole III posits that if students, parents, and teachers receive training on nutrition, health, and hygiene practices (Khatoon et al., 2017); if knowledge of food preparers at the school and community levels is improved in the areas of safe food preparation and storage practices, then health and hygiene practices and behaviors will be enhanced to provide students with safer meals. Unlike baseline and midline evaluations, in the absence of primary quantitative data and school observations, our analysis in this section was limited to qualitative interviews. SMC members described being able to manage the canteen better and respect hygiene rules after receiving training from the project on safe food preparation and storage. Project and partner staff added that SMCs also received training on warehouse maintenance for safe food storage, as well as hygiene and recipe preparation. Local education officials stated that the canteens were preparing good-quality food and that food preparers were respecting hygiene rules after receiving training. They confirmed these observations by monitoring the canteens, reviewing storage practices in the warehouse, and overseeing the quantity of food used for meals each day. They also noted that the training on hygiene and food “BLA increased students’ performance in reading. A student from Grade 1, through BLA, can read on the same way as a student from Grade 6 who was not exposed to the BLA.” –Teacher, Mopti IMPAQ International, LLC Page 74 McGovern-Dole III Endline Evaluation Report preparation had a positive impact on student health. National government stakeholders also said that good food storage practices were being followed. Although there was limited discussion of handwashing during interviews, teachers, principals, and project and partner staff noted that the project improved hygiene and handwashing practices, especially when handwashing stations were provided in schools. One teacher shared that students now washed their hands before eating and got sick less often due to better handwashing habits. Another teacher stated that students were already washing their hands on some occasions before COVID-19. MGD 2.2 Increased Access to Preventive Health Interventions McGovern-Dole III also hypothesized that if access to micronutrients and deworming medications is increased to take preventive action against the spread of infections associated with malnutrition and limited healthcare access (Keating et al., 2019; Ulukanligil, 2006), then communities will be better positioned to prevent disease (Keating et al., 2019; Ulukanligil, 2006). We looked at this path in the results framework mostly with qualitative findings with support from CRS monitoring data. Planned project activities included the distribution of medical supplies, specifically vitamin A and deworming pills, to schoolchildren with the hope that enabling access to these preventative health interventions would improve health outcomes. According to CRS monitoring data on deworming pills distributed to schools, which we examined for a single quarter of each school year, the average number of deworming pills was mostly consistent year over year but declined slightly in the 2019 – 2020 school year, as shown in Exhibit 66. This decline is in line with the McGovern-Dole III sustainability plan, in which communities are intended to take ownership of health interventions such as vitamin A and deworming pills. This means that while the project distributed vitamin A and deworming pills initially, partners provided awareness and training for SMCs to mobilize financial resources and community support to purchase and distribute these items. Regional differences in deworming pill distribution were minimal during the data collection period. In stakeholder interviews, one education official noted that children are getting sick less often since the project was implemented. Project and partner staff had varied experiences with sustainability. One respondent gave the example of a community that purchased vitamin A and deworming medication for all schools with support from health facilities. Another respondent said that other communities were expected to take ownership but were not able to purchase these items for their schools. IMPAQ International, LLC Page 75 McGovern-Dole III Endline Evaluation Report Exhibit 66. Average Number of Deworming Pills Distributed at Schools in October to December per School Source: CRS monitoring data. N = 228 schools in 2016 – 2017; 236 in 2017 – 2018; 81 in 2018 – 2019; 14 in 2019 – 2020. Note: Showing single-quarter output for October to December as representative of typical quarterly contribution amounts. Exhibit 67. Average Number of Deworming Pills Distributed at Schools in October to December per School, by Gender Source: CRS monitoring data. N = 228 schools in 2016 – 2017; 236 in 2017 – 2018; 81 in 2018 – 2019; 14 in 2019 – 2020. Note: Showing single-quarter output for October to December as representative of typical quarterly contribution amounts. CRS monitoring data on vitamin A pill distribution to schools is limited to data from the first two years of the intervention. As with deworming pills, distribution of vitamin A supplements serves as a preventative health benefit for children attending school. The recorded distribution of these pills over the two school years declined from an average of 307 pills distributed per school in one quarter of the 2016 – 2017 school year to 228 in 2017 – 2018, as shown in Exhibit 68. Gender and regional discrepancies in vitamin A supplement distribution, shown in Exhibits 69 and 70, were most pronounced in the 2016 – 2017 data, with female students receiving and Koulikoro schools distributing more pills than their counterparts. 256 231 245 218 190 200 210 220 230 240 250 260 2016-2017 2017-2018 2018-2019 2019-2020 Deworming Pills 122 114 127 114 129 115 116 102 0 20 40 60 80 100 120 140 2016-2017 2017-2018 2018-2019 2019-2020 Deworming Pills Boys Girls IMPAQ International, LLC Page 76 McGovern-Dole III Endline Evaluation Report Exhibit 68. Average Number of Vitamin A Pills Distributed at School in October to December per School Source: CRS monitoring data. N = 227 schools in 2016 – 2017; 236 in 2017 – 2018. Note: Showing single-quarter output for October to December as representative of typical quarterly contribution amounts. Exhibit 69. Average Number of Vitamin A Doses Distributed at School in October to December per School, by Gender Source: CRS monitoring data. N = 227 schools in 2016 – 2017; 236 in 2017 – 2018. Note: Showing single-quarter output for October to December as representative of typical quarterly contribution amounts. 307 228 0 50 100 150 200 250 300 350 2016-2017 2017-2018 Vitamin A Pills 120 113 133 115 100 105 110 115 120 125 130 135 2016-2017 2017-2018 Vitamin A Pills Boys Girls IMPAQ International, LLC Page 77 McGovern-Dole III Endline Evaluation Report Exhibit 70. Average Number of Vitamin A Doses Distributed at School in October to December per School, by Region Source: CRS monitoring data. N = 100 schools in Mopti and 127 in Koulikoro in 2016 – 2017; 108 in Mopti and 128 in Koulikoro in 2017 – 2018. Note: Showing single-quarter output for October to December as representative of typical quarterly contribution amounts. SO2 Increased Use of Health and Dietary Practices The second main SO of McGovern-Dole III posits that if health and hygiene knowledge and practices are improved at the school and community levels, specifically in safe food preparation and storage (Khatoon et al., 2017); if communities are better positioned to prevent disease (Keating et al., 2019; Ulukanligil, 2006), then, use of health and dietary practices will be enhanced. In addition, an increase in use of health and dietary practices is also expected combat health-related school absenteeism, contributing to SO1 by improving school attendance. According to qualitative interviews, the safe food preparation and storage trainings for SMC members seem to have a positive effect on school canteens, which are preparing good quality food, and respecting hygiene rules. Qualitative and quantitative data both suggest that preventative health provisions such as deworming, and vitamin A pills are positively associated with students’ health. 4.4.2 Exploratory Impact Evaluation This section presents the findings from the exploratory impact analyses for Grade 1. We present first the impact results on foundational reading skills that have been shown to be predictive of later reading achievement, such as phonemic awareness, alphabet knowledge, and decoding ability. We then present the impact results on the three McGovern-Dole reading proficiency indicators required by USDA to track program performance. Each table of results is structured as follows. In the first column of results, we report the total program effects for each outcome (the coefficient β from the impact equation in Section 3.3.3), that is, the change in students’ literacy outcomes due to one year of exposure to BLA-trained teachers, conditional on other characteristics controlled for in the regression and the associated standard errors. The next two sets of 226 223 370 233 0 50 100 150 200 250 300 350 400 2016-2017 2017-2018 Vitamin A Pills Mopti Koulikoro IMPAQ International, LLC Page 78 McGovern-Dole III Endline Evaluation Report columns present the program effects separately by region and by student gender.22 We present the regression results from the sample that includes all schools with valid EGRA data in June 2016 and June 2018. Regression results performed as a robustness check from a restricted sample including only the schools for which we have valid EGRA data in both years are presented in Appendix E. Although our main focus is to assess the program’s effect on reading proficiency, examining the impacts on the foundational reading skills provides useful information, especially when examining the effects on young children who cannot yet read a passage with comprehension. Therefore, we examined the BLA effect on student progress toward acquiring the foundational skills by estimating the program impact on performance on the seven EGRA sub-tests developed specifically to assess those foundational skills. As described before, the foundational reading skills are measured as continuous variables, such as the number of sounds correctly identified. The corresponding estimated effects represent the average number of additional correct on the sub-test associated with one year of exposure to BLA-trained teachers. The three reading proficiency outcome variables analyzed in this section, as defined in Section 4.4.1 on the performance evaluation, are measured as binary indicators: whether the child meets the grade-level standard or not. The estimated effects represent point differences in proficiency rate between students exposed and students not exposed to BLA-trained teachers. Program Effects on the Literacy of Grade 1 Students As shown in Exhibit 71, one year of exposure to BLA is associated with significant increase in alphabet knowledge, decoding ability, and reading comprehension for Grade 1 students. Specifically, Grade 1 students were able to read per minute, on average, 13.6 more letters or sounds, decode 1.8 more invented words, and answer correctly 0.2 additional reading comprehension questions. Positive findings in these areas were significant for both boys and girls, except that the change in girls’ decoding ability is not statistically significant. Interestingly, we find evidence that the program effect was much stronger in Koulikoro than in Mopti. In fact, Grade 1 students in Koulikoro showed significant improvement in six out of the seven foundational reading skills examined. Exhibit 71. Program Effects on Foundational Reading Skills for Grade 1 Reading Skills Total Mopti Koulikoro Girls Boys Phonological awareness 0.817 (0.946) –1.530 (1.774) 2.613*** (0.896) 0.712 (1.131) 0.722 (0.991) Alphabet knowledge 13.558*** (4.294) –4.722 (5.936) 27.485*** (4.825) 11.495** (5.092) 15.500*** (4.574) Word recognition 1.810 –1.295 4.176** 1.758 1.695 22 Analyses were also performed by school curriculum to explore whether there were differential patterns between classic and bilingual schools. However, due to the small sample size of schools with bilingual instruction, these results may be biased and are therefore not reported. IMPAQ International, LLC Page 79 McGovern-Dole III Endline Evaluation Report Reading Skills Total Mopti Koulikoro Girls Boys (1.265) (1.758) (1.670) (1.527) (1.367) Decoding ability 1.785* (0.918) –0.635 (1.260) 3.621*** (1.212) 1.800 (1.164) 1.756** (0.853) Oral reading fluency 1.991 (1.634) –1.924 (2.393) 4.979** (2.096) 2.214 (1.881) 1.750 (1.615) Reading comprehension 0.217** (0.106) 0.042 (0.122) 0.352** (0.157) 0.228* (0.128) 0.210* (0.126) Listening comprehension 0.212 (0.462) 0.307 (0.495) 0.145 (0.722) 0.183 (0.489) 0.287 (0.542) Total number of students 2,086 1,027 1,059 1,040 1,046 Total number of schools 92 49 43 92 92 Source: Student EGRA data. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors shown in parentheses are clustered at the school level. The significant improvements in foundational reading skills for Grade 1 students did not translate into large significant improvements in McGovern-Dole reading proficiency indicators. As shown in Exhibit 72, for the sample as a whole, we find a small positive significant effect in reading with comprehension and no significant changes in decoding proficiency or in reading proficiency at the national standard level for Grade 1 students. One year of exposure to BLA-trained teachers is associated with a 2.5 percentage point increase in Grade 1 reading with comprehension. The sub-group analysis by region showed that students in Koulikoro exhibited significant positive improvements in decoding proficiency, a 28.6 percentage point increase, while students in Mopti showed no improvement. In terms of gender, girls showed a 3.2 percentage point increase in reading proficiency with comprehension, while boys showed an increase of 13.0 percentage points in decoding proficiency. Exhibit 72. Program Effects on Reading Proficiency for Grade 1 McGovern-Dole Reading Proficiency Indicator Total Mopti Koulikoro Girls Boys Decoding proficiency 0.120 (0.077) –0.098 (0.131) 0.286*** (0.082) 0.110 (0.093) 0.130* (0.073) Reading proficiency at national standard level 0.050 (0.040) –0.021 (0.033) 0.105 (0.065) 0.055 (0.046) 0.050 (0.044) Reading proficiency with comprehension 0.025* (0.014) 0.020 (0.013) 0.028 (0.022) 0.032** (0.015) 0.020 (0.022) Total number of students 2,086 1,027 1,059 1,040 1,046 Total number of schools 92 49 43 92 92 Source: Student EGRA data. * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. Standard errors shown in parentheses are clustered at the school level. 4.4.3 Perceived Impact Interview topics focused on stakeholders’ perception of the effect of the project on children’s education and health outcomes and the activities with the greatest and least medium- and long-term effects. Stakeholders were also asked to share perceptions of barriers to education and how these barriers were addressed by the project. IMPAQ International, LLC Page 80 McGovern-Dole III Endline Evaluation Report Addressing Barriers to Education At baseline and midline, the main barriers to education reported by parents and SMC members were distance to school, cultural beliefs, and local security concerns. At endline, SMC members, mayors, teachers, principals, MONE officials, and project and partner staff primarily noted early marriage for girls, housework for boys and girls, urban migration, distance to school, and poverty. Respondents shared that sensitization helped address the first three barriers. Sensitization efforts included SMC members going door to door or investigating student absences, as well as local radio spots on early marriage. In a village in Mopti (Kendie), the SMC met with a religious leader and a community leader to persuade them to allow girls to continue with school after early marriage. To address distance to school and poverty, respondents said that hot meals and THRs motivated students to attend school and stay for the full day rather than going home for lunch and not returning. Teachers and principals mentioned lack of awareness about the importance of education, especially among parents with low literacy skills, as another barrier to education. The project addressed this need through sensitization efforts and the provision of school canteens. A principal, teacher, SMC member, and SILC member connected to the same school all mentioned the poor condition of a bridge between the city and the school; parents were afraid to let children cross the bridge. To address this issue, the SMC was working with the municipality to repair the bridge, and the SILC raised money to cover the renovation costs. A few project and partner staff members mentioned that insecure conditions led parents to enroll their children in madrasahs rather than government-sponsored schools due to safety concerns that government-sponsored schools might be attacked by religious extremists. Identifying Activities with the Greatest and Least Impact From the perspective of local community and government respondents, the activities with the greatest impact were the hot meals and THRs, school gardens, and BLA. Additional activities mentioned by local community and government stakeholders included collective fields to supply the canteens and hygiene, sanitation, and food preparation training for cooks. Education officials and SMC members highlighted parent sensitization on education and schools. Principals said that colored report cards were an effective tool for engaging parents, many of whom had low or no literacy skills, in monitoring their children’s performance. Stakeholders across all levels were reluctant to identify any activities as having the least impact and noted that activities were complementary. One mayor said that the SMC training was not particularly successful, but he said that SMC functions improved after SMCs started competing to receive grants. IMPAQ International, LLC Page 81 McGovern-Dole III Endline Evaluation Report Like local beneficiaries, national project and partner staff and MONE officials also found the hot meal and THR distribution, BLA, and school gardens to have had the greatest impact. Many acknowledged the role of the school canteen in bringing students to school; however, they stressed the importance of BLA for improving literacy and school performance. These respondents also said that SILC groups and community sensitization on the importance of education had substantial impact. Project staff saw the SILC groups as a way to promote savings, enable members to pool resources, and support school canteens. Although project and partner staff were reluctant to share any activities with limited impacts, they provided the example of efforts to improve community capacity to take over the canteen after the project ends. 4.5 SUSTAINABILITY This section describes the sustainability of core project components and then discusses the factors that stakeholders said have contributed to the successful continuation of activities when the project ends. Opinions on sustainability varied among respondents no matter the stakeholder type or location, reflecting nuanced and thoughtful perspectives on stakeholder ownership and assumption of responsibilities. 4.5.1 Sustainability of Core Project Components Balanced Literacy Approach Perspectives on the continuation of BLA varied among the types of stakeholder and their locations. Project stakeholders explained that a significant barrier to continuing BLA was the high cost of necessary education resources, such as materials and consistent monitoring under the CAP. Education officials confirmed that they could ensure that teachers receive continuous training and monitoring “if we are given means”; additionally, one education official in Koulikoro commented that, if the government “authorizes” the ongoing usage of BLA, the CAP will continue to train teachers to implement BLA in their classrooms. One project staff member remarked that collaboration with other partners in the education space will be critical to ensure that BLA reaches more schools. When hesitancy and uncertainty emerged among education officials regarding the sustainability of BLA, they pointed most often to the available resources and participation of teachers rather than their own capacity to continue training and monitoring teachers. One respondent noted that “teachers are not “The majority of students come from far away, they cannot stay at school all day without eating. Suddenly, the canteen component not only improves attendance, but also keeps students in school. As well, one could say that the level of learning for students has increased, again, all due to the canteen component.” –Education official, Mopti IMPAQ International, LLC Page 82 McGovern-Dole III Endline Evaluation Report trustworthy” and may abandon BLA for the syllabic approach.23 This education official added that teachers trained in BLA could relocate, so that their BLA training would no longer be available to the community in which they were trained. One principal in Koulikoro and one in Mopti believed that BLA could continue without project support, despite limited resources. The Mopti principal stated that teachers would retain BLA “because it is interesting,” and therefore “even without the presence of the project, we will continue to practice with the little means we have.” However, one teacher in Koulikoro, expressing a perspective similar to that of some project staff, explained that, although teachers would retain knowledge from BLA trainings they had already received, regular educational monitoring and follow-up would not occur “without the influence of the project.” One project staff member expressed confidence that school principals and teachers would take ownership of trainings intended to enhance their job responsibilities. As proof, this stakeholder pointed to demonstration videos of several teachers facilitating classes that have been used effectively to spread teacher training throughout the country, given continued support from MONE. Canteens Stakeholders held varying opinions on whether the canteens could operate without project support. A few individuals stated that the incorporation of school meals into government policy reaffirmed their belief that the canteens will be sustainable. One project staff member commented that CNCS had already achieved results in terms of capacity building through support developing the school feeding law and training modules on nutrition, hygiene, and food preparation which have been translated into five local languages. However, almost all respondents pointed to the ability of communities to mobilize resources as the determinant of whether canteens will ultimately function independent of project support. An education official in Koulikoro offered a promising example of communities mobilizing to supply the canteen with grains when a food shortage occurred because of a two-month delivery delay. Generally, respondents in Koulikoro seemed to hold a more positive outlook on sustaining school feeding compared 23 The syllabic approach references a literacy methodology promoted by MONE that has used widely in the country except for schools with BLA approach. The syllabic method consists of getting the learner to first identify letters, vowels and consonants, and combine them to make syllables. After the end of the project, the canteen can continue to operate thanks to the community contribution. –Mayor in Koulikoro Region IMPAQ International, LLC Page 83 McGovern-Dole III Endline Evaluation Report to Mopti respondents. No education officials in Koulikoro commented negatively on the continuation of the canteens. In Mopti, one education official said, “It’s certain that, after the project ends, [the communities] will fail with school feeding” because of their inability to mobilize contributions to enhance school meals for children. The national school feeding law designates communities, local authorities and the state to all provide financial support for school feeding.24 One teacher in Koulikoro explained that the sustainability of canteens might depend on the specific conditions in each locality. Communities that have successfully contributed grains and planted fields to support the canteen will likely be better equipped to take on school meals independently. Affirming this perspective, one SMC representative in Koulikoro explained that, though the quality of the food may not be as good, communities will continue to contribute because they have become accustomed to this practice during the project. Other SMC members in Koulikoro and Mopti said that items such as rice and oil might disappear from the rations. Thus, according to a teacher in Mopti, even if communities have been able to establish school fields and contribute to the canteens, “without external aid, they will not go far.” A few stakeholders pointed out that THRs present formidable costs, and their communities could not provide similar THRs after the project closes. In discussing the sustainability of canteen operations, one project staff member emphasized the importance of enhancing ongoing support to local municipalities to ensure they receive significant financial support and capacity building. Consequently, collectivities can better include canteen operations in their PDSEC plan. Chiming in on that point of community capacity, one national government stakeholder explained that the effectiveness of community action will depend on social cohesion and leadership, noting that “social mechanisms for managing communities are not the same everywhere.” Savings and Internal Lending Communities Despite hesitation about whether BLA and canteen operations will be sustained without project interventions, almost all project stakeholders praised the SILC groups and believed that they would remain functional after the project ends. One project staff member pointed to SILC groups created in the second phase of the project that were still operating even though the third phase was not being implemented in their communities. Interviews with SILC representatives confirmed that they would continue to participate in their SILCs after the project ends because their involvement has been beneficial. One education official in Mopti noted the important contribution of SILC groups to the performance of canteens. The ongoing operation of the SILC groups could bring sustainable benefits to school meals. 24 The School Feeding Law (Loi N˚2019-013) states that “targeting, construction, and equipping canteens are the responsibility of the National Center for School Canteens” (Article 28, Title III, Chapter I). However, “territorial collectivities and communities must contribute to the facilitation and sustainability of school canteens” (Article 25, Title II, Chapter V of the Law). Additionally, “the State, territorial collectivities, and communities support subsidies intended for school feeding” (Article 26, Title III, Chapter I). IMPAQ International, LLC Page 84 McGovern-Dole III Endline Evaluation Report School Management Committees Although SMCs will continue to exist all schools after project end because of government mandates on its operation, project staff commented on the possible lasting improvements upon project end. One respondent indicated that capacity building of SMC members will end. Another project staff member remarked that SMCs will still engage in door-to-door sensitization to encourage parents to enroll their children at school, but whether other activities will continue remains uncertain. A few project stakeholders felt confident that SMCs had retained knowledge about how to develop school project and action plans and would continue to contribute to these plans in the future. Although only project staff remarked on specific aspects of SMCs that may be sustainable, a few other stakeholders provided some general comments. For example, a mayor in Mopti explained that capacity building of the SMCs improved SMC operation; meanwhile, the participation of the municipality in those same trainings served to coordinate knowledge and competencies. Principals in Koulikoro also noted the value of the SMCs in keeping activities, such as the school canteens and the BLA approach, sustainable beyond the end of the project. Other Some other activities that a few stakeholders considered sustainable were collective fields and gardens, which “empower the canteens,” according to a mayor in Mopti; distribution of vitamins; and promotion of good hygiene practices. 4.5.2 Critical Factors for Sustainability Community Mobilization A strong concern that emerged among all stakeholders is the ability of communities to assume ownership of project activities given the resources they have available. Project staff noted on several occasions that communities cannot replenish the materials supplied to schools for BLA once the project ends because doing so requires significant financial investment. Especially in regard to canteen maintenance and supplying schools with sufficient food for meals, stakeholders explained that communities vary greatly in their ability to utilize collective fields, school gardens, and pooled resources to offer a consistent and adequate stock of quality food items to feed students. Although most stakeholders noted that communities would be willing to assist the schools if they simply had sufficient resources, one education official in Mopti raised the point that, regardless of financial capacity, some communities may feel less inclined to engage with schools. In the cases of low levels of interaction between communities and schools, the “possibility of sustainability will be reduced.” One project staff membersuggested that some communities might pretend they do not have resources so that they do not need to contribute as much. We need to make sure communities acquire knowledge, skills, and some form of organization that will allow them to continue without any external aid. –National government stakeholder IMPAQ International, LLC Page 85 McGovern-Dole III Endline Evaluation Report Despite concerns about whether communities have the incentive and capacity to engage with schools and continue project activities, several project staff members described promising examples. For example, one project staff described having visited two schools that had adopted a collaborative approach with the village chief to determine the in-kind and financial contributions of individual households to cover one month of food for the canteens. At these two schools, the project staff member explained, “social mechanisms” were in place to help the most vulnerable households, which might not be able to contribute. The respondent did not explain what these mechanisms entailed. No matter the motivation behind community engagement or disengagement, almost all stakeholders agreed that the ability to continue project activities rests upon how well communities can mobilize resources. Government Buy-in As part of maintaining high levels of community engagement, some project staff members suggested that government ought to be better equipped to “take ownership in the provision of food in the canteen.” Generally, these project stakeholders felt confident that the government has sufficient technical capacity to lead project activities and applauded CRS for support provided to MONE and CNCS in joint missions and stakeholder workshops. One project stakeholder pitched that CRS should develop a monitoring and evaluation (M&E) system for CNCS to take ownership of school feeding and adjust activities in relation to revealed needs for all canteens. Adding to the call for support of CNCS, one national government stakeholder stated that CNCS can take on school feeding only if it receives appropriate training and support from the project. Now, CNCS plans to raise US$1 million so that it can work with cooperatives and production groups to supply canteens. Below the national government level, a few project staff indicated that local municipalities should have greater involvement not only in managing and mobilizing resources for canteens but also, broadly, in monitoring school activities. Challenging the potential for government support, some project stakeholders noted that political change and high turnover of municipal and national staff mean that established relationships may be disrupted. Such disruption effectively “turns back the clock” on the effect of capacity-building trainings. The sustainability depends on the degree of engagement of communities expressed during the lifetime of the project. In case this engagement decreases, the possibility of sustainability will be reduced. –Education official in Mopti IMPAQ International, LLC Page 86 McGovern-Dole III Endline Evaluation Report Preparedness and Planning Almost all respondents commented on the preparedness of CRS and its partners for passing along ownership of project activities. According to one project staff member, this preparedness reflects the integration of a sustainability component from the very beginning of project design. Thus, trainings, workshops, and collaboration with CNCS were ongoing and integral aspects of the project, which resulted in, for example, a five-year plan for development and leadership of endogenous (community) canteens. Additionally, several project stakeholders remarked on CRS’s close relationship with the national government, a relationship that facilitated adoption of a law on school feeding. Of note, one project staff member pointed out that the government had already taken steps to increase by about 10 percent the number of canteens in the country. One national government stakeholder noted that mayors and local municipal officials ought to take the lead and coordinate with MONE if they require support to sustain activities. At the community and school level, preparations for project withdrawal were underway at the time of data collection. Project staff members noted that they had organized trainings and established teacher learning communities to keep the knowledge dispensed to communities and schools active even after the project concludes. Sensitization and information campaigns were undertaken a year in advance of project termination so that communities could establish systems and procedures toward uninterrupted continuation of project activities. According to one project staff respondent, all SMC groups and local authorities had been informed of the withdrawal plan, which contains at least 55 activities with specific days, months, and quarters related to the exit strategy. Subsequently, communities were “ready to say goodbye to the project,” as one project staff member phrased it, and had started planning to support the schools independently. For example, several stakeholders mentioned that collective fields to benefit schools had been established; additionally, one project staff stakeholder commented that SILCs had agreed to make regular monetary contributions. A principal in Koulikoro and a teacher in the same area explained that monthly meetings had been held with the SMC to plan for the end of the project. However, another principal in this same region explained that no preparations had yet taken place. Active and Trained School Management Committees As crucial as government involvement is to enhance community support, stakeholders also raised the value of well-organized and trained SMCs for the sustainability of project activities. SMCs with a solid understanding of their roles and responsibilities can lead communities to develop robust plans to support schools and monitor performance. One project staff member noted that the structure given to SMC operations had “decreased diversion and misuse of funds” at the local government level so that SMCs can receive their designated portion of the funds allocated to schools. However, a potential threat to the functionality of the SMCs may be the election of new individuals every three years. This turnover may diminish the effectiveness of capacity building already completed with current SMC members. In our context of insecurity and instability the best practice to keep in mind is the good communication and collaboration between stakeholders. –Project staff IMPAQ International, LLC Page 87 McGovern-Dole III Endline Evaluation Report Generally, SMC respondents seemed to understand that they need to manage the quantity of contributions to the canteen and mobilize communities so that households are inclined to provide in-kind or financial support. An SMC member in Mopti commented broadly that SMCs should work with communities to ensure that canteens continue to operate “at 100 percent.” Social Cohesion The term social cohesion emerged several times among different stakeholders in their assessment of whether communities could sustain project activities. One project staff member noted that certain external factors (see below) could threaten social cohesion and so endanger project sustainability. Meanwhile, a national government stakeholder discussed social cohesion in relation to a lack of leadership to better organize community priorities. Additionally, a mayor in Mopti noted that the success of activities rests upon the level of social cohesion in the community. As one education official in Mopti reflected, the “consciousness of the community” in feeling a sense of responsibility to the school could have a significant impact on stakeholder ownership. External Factors Stakeholders discussed several external challenges to the success and subsequent sustainability of project activities. For example, strikes, insecurity, climate change, and the lack of water at schools all complicate lasting effectiveness and sustained operations. As one education official in Mopti noted, “It is impossible for the project to step in before an armed man to make him accept children’s school enrollment.” Another education official in Mopti remarked that suspected jihadists had threatened some SILC group members. A mayor in this same region commented that school canteens could suffer because of the lack of a safe water source, which will “work against sustainability of the school garden.” Please refer to Section 5 for COVID-19 implications for sustainability. CRS shared adaptations to external factors including providing support through local partners and community structures such as SMCs, clear communication during food deliveries to avoid hijacking, organizing meetings with communities to identify strategies given school closures, capacity building for mayors and SMCs to manage canteens in a context of insecurity. Other An education official in Koulikoro commented that the sustainability of BLA might be threatened by the transfer of teachers. To address this challenge, this respondent said, the project should expand its training to include a larger number of teachers. A different education official in the same region commented on the poor state of school infrastructure, which could keep parents from sending children to school. IMPAQ International, LLC Page 88 McGovern-Dole III Endline Evaluation Report Section 5. COVID Limitations on Implementation After the confirmation of the first COVID-19 case in Mali, the Government of Mali implemented measures to limit the spread of the virus. Effective March 20, the Government prohibited flights originating from countries with confirmed cases of COVID-19, including the United States. Additionally, effective March 26, schools were closed until at least May 9, 2020; workshops, meetings, and gatherings larger than 50 people were banned; and the government imposed a curfew from 9:00 pm until 5:00 am daily (COVID-19 Information | U.S. Embassy in Mali, n.d.). Later, the school closure was extended to the end of school year. We asked interviewees to describe their perceptions of how the pandemic affected project implementation, outcomes, and sustainability in the last six months of the project. The main effect of COVID on project activities, according to respondents, was that CRS converted hot meals to THRs due to the school closure. Additionally, CRS suspended or canceled workshops, trainings, and meetings to avoid large gatherings. However, some SILC members reported that they continued to meet in smaller groups, following health protocols such as handwashing at the entrance to the meeting. One project stakeholder pointed out that, for meetings that have moved online, poor connectivity and network quality have limited the engagement of all stakeholders. The postponement and cancellation of workshops and meetings intended to ease the handover of project activities to relevant community and government stakeholders may affect sustainability. For example, one project staff member commented that validation sessions with CNCS to document and organize the management of canteens had been delayed several times because of COVID restrictions on gathering. Other relevant trainings, such as working with mayors to improve their abilities to implement assessments of municipalities, also were put on hold. Interestingly, slightly more stakeholders at the community level felt that COVID was having little effect on sustainability. This reflection did not seem to vary based on geography, as community stakeholders across regions expressed skepticism or uncertainty that COVID had changed planned sustainability activities. One teacher in Mopti explained that schools had faced closures before because of teacher strikes. This sentiment that COVID had little effect on sustainability was especially prevalent among SILC and SMC members. However, the one SMC member who believed that COVID was affecting sustainability stated that SMC sensitization of communities had been limited because of COVID restrictions on gatherings. Generally, respondents did not think COVID had constrained project effectiveness; as one USDA respondent noted, COVID occurred “in last twilight hours of implementation” and therefore would likely have minimal impact. A mayor in Koulikoro reiterated this belief that COVID had little effect on project results because of its timing close to the end of the project. Further, COVID may not have influenced project effectiveness because of government measures to “limit the damage within community and schools” for children’s education, though the national government stakeholder who offered this opinion did not elaborate on the specific actions taken. IMPAQ International, LLC Page 89 McGovern-Dole III Endline Evaluation Report Despite the generally positive assessment of continued project effectiveness against the backdrop of COVID, some individuals conceded that the school closures and subsequent disruption in coursework for students could have implications for student learning. The suspension of teaching activities and classroom supervision and monitoring were seen as especially likely to have impacts. Additionally, pedagogical activities encountered some obstacles to continuation and sustainability because activities to strengthen the capacity of education advisors and the development of a teacher learning community were paused. One education official in Mopti explained that, without regular support for teachers, student performance would suffer. Two principals in this region said that literacy would decrease. This sentiment extended to Koulikoro, as a principal commented that children forget quickly after six months without classes. However, an education official in Koulikoro explained that mitigation measures had been put in place, such as using national radio and television to deliver remote classes and providing children with USB drives and educational radio spots so that they can learn independently. Even as schools began to lift restrictions, a few stakeholders stated that COVID would affect student enrollment because of social distancing restrictions and an insufficient number of classrooms to keep students sufficiently distanced; in addition, parents might be afraid to send children to school. Of note, one change to project implementation in response to COVID that received stakeholder praise was the provision of handwashing kits and sanitation supplies such as masks to help communities stay healthy. However, an education official in Koulikoro commented that the personal protective equipment delivered to communities by the government and the project did not sufficiently cover all households. Separate from project activities, some commentary emerged among stakeholders regarding data collection. Most project staff felt that COVID had not hindered efficient data collection for project monitoring purposes during the last six months of project. However, one respondent noted some delays to the schedule and the timing of data collection because of COVID restrictions. IMPAQ International, LLC Page 90 McGovern-Dole III Endline Evaluation Report Section 6. Conclusion This endline evaluation report assesses the relevance, efficiency, effectiveness, and impact of the project over the past five years in achieving its intended results. This report also provides recommendations on sustainable exit strategies and outlines lessons learned for future implementation of any new McGovern￾Dole phases. Reviewing all available data, reports, and other relevant documents shared by CRS, IMPAQ conducted a short evaluability assessment to design the endline evaluation. Our alternate design covered four approaches: (1) an in-depth document review, which would not have been necessary with primary data collection; (2) remote qualitative data collection, which was necessary because of travel restrictions; (3) analysis of secondary quantitative data in the absence of primary data; and (4) rigorous triangulation of data to address the limitation of the other three approaches. 6.1 LIMITATIONS OF THE STUDY This study faced limitations in the evaluation design, as well as quantitative and qualitative analysis. 6.1.1 Qualitative Study Limitations The most significant limitation of the qualitative study is the remote approach necessitated by the COVID￾19 pandemic. Conducting interviews remotely can limit evaluators’ ability to assess the context or observe a respondent’s facial expression, unspoken cues, and gestures as they do during fieldwork and face-to￾face interviews. Another limitation is the sampling methodology. Although we aimed to achieve equal gender distribution among interview participants, the remote approach and availability of stakeholders limited the ability to achieve representation. While there was equal gender distribution for SMC and SILC stakeholders, nearly all other stakeholders were male due to challenges in including female education officials, principals, and teachers with the remote approach. While some SMC and SILC members interviewed were parents, we were also not able to conduct focus groups with parents. A final limitation of the qualitative design is selection bias. Participation was limited to stakeholders who were able to communicate by phone or on online platforms. 6.1.2 Quantitative Study Limitations Because our quantitative evaluation design rests on secondary data, it has several limitations. First, the scope of our analysis is restricted to the available secondary data. We were not able to construct the indicators in the same way as at baseline and midline or able to evaluate the performance of the project across all indicators. Moreover, the analyses presented at baseline and midline are not directly comparable to the analysis presented at endline because we could not construct some indicators in exactly the same way. Instead, the endline evaluation analysis complements the findings from baseline and midline evaluations. Appendix D shows the measures and outcomes on which we were unable to assess performance. IMPAQ International, LLC Page 91 McGovern-Dole III Endline Evaluation Report Second, because we are combining several sources of data, it is difficult to identify whether data are missing at random or because of challenges during data collection. Moreover, because we are using data across several years, listing the challenges is beyond the scope of the evaluation. Third, because the endline evaluation used different samples and methodologies from the baseline and midline evaluation, our findings serve only to complement the impacts reported at midline. Finally, as third-party evaluator, we are expected to collect independent and unbiased data to substantiate our findings about program activities implementation and achievement of stated goals. Due to the COVID-19 pandemic, in-person field data collection was not possible. Therefore, all quantitative analysis presented here is based on implementer-provided data. 6.2 KEY OUTCOMES AND IMPLICATIONS FOR MCGOVERN-DOLE III RESULTS FRAMWORK Triangulation of the quantitative and qualitative findings in Section 4 on the relevance, efficiency, effectiveness, impact, and sustainability of McGovern-Dole III reveals several important outcomes at this stage of implementation. These outcomes have valuable implications for the results framework. Overall, the performance evaluation data suggest that the project generally was able to achieve the intended objectives, though achievements in some areas were limited. In addition, external factors such as frequent teachers’ strike and political instability caused some school disruptions that ultimately affected some of the outcomes related to SO1. We outline below the implications of the endline evaluation outcomes for the McGovern-Dole III results framework.  Quality of literacy instruction. Classroom observation data showed that teachers’ perceptions of how students learn and how they should teach had changed dramatically since the beginning of the project. Fewer teachers believed that students must memorize a text to understand it, that they must correct all student errors, or that writing is about good handwriting. More teachers agreed that all students who attend school are capable of learning to read. The data also show that teachers were using a wider variety of resources in the classroom including alphabet posters and cards, textbooks, flash cards, and story books, compared to the beginning of the project. Qualitative interviews showed that teacher attendance increased during the project, except during school closures due to teacher strikes and COVID-19. Quantitative data from SMCs showed that SMC members’ perception that teachers regularly attended was mostly steady in Koulikoro but decreased dramatically in Mopti. But there were significant drops in their attendance, which could be explained by school disruptions. In addition, in the qualitative interviews, partners and other stakeholders reported that teachers received training on BLA and that teachers had mastered the BLA techniques, as determined by regular monitoring, assessments, and feedback sessions.  Student enrollment and attendance. Our analysis found strong correlations between the number of students enrolled in a school and the amount of distributions from the project including hot meals, THRs, deworming pills, and vitamin A pills. The attendance data show a steady attendance IMPAQ International, LLC Page 92 McGovern-Dole III Endline Evaluation Report rate over the years save for a few large dips that are likely explained by external factors such as inclement weather and teacher strikes. The findings suggest that students responded to incentives to enroll in and attend school.  Literacy. Overall, we found gains among Grade 1 students. The exploratory impact evaluation found strong impacts from BLA on foundational reading skills such as alphabet knowledge, decoding ability, and reading comprehension; however, these gains were not large enough to raise students’ literacy skills to the national standards. Students in Koulikoro drove most of the gains we found. The performance analysis shows consistent results suggesting improvements for Grade 1 students, though there were with regional disparities. On the other hand, consistent with EDC findings in 2019, our results did not show any improvements in Grade 2 students’ achievement of national reading proficiency standards. The string of teacher strikes in 2018 – 2019 school year led to approximately 64 school days loss of instructional time, which could explain the limited improvements, and some declines, in literacy outcomes. In addition, there were differences between the regions: Grade 2 students in Mopti showed lower levels of decoding proficiency and reading proficiency than those in Koulikoro. Political instability in Mopti that caused school closures in 2017-2018, and 2018-2019 years could also explain this regional differences. That said, observing lack of improvements for Grade 2 students should be interpreted with caution and should not be attributed to inefficiency of BLA. Recall that at midline, we found that exposure to BLA-trained teachers for two and three years led to improvements in children’s literacy, as measured by their performance on reading scores. The endline evaluation analysis for Grade 1 students further bolstered these findings about the efficacy of the training and contributed to scaling-up of BLA training further.  Health and dietary practices. According to interview respondents, the safe food preparation and storage practices trainings for SMC members seemed to have had a positive effect on school canteens, which were serving good-quality food while respecting hygiene rules. Qualitative and quantitative data both suggest that the preventative health provisions — that is, deworming and vitamin A pills — were positively associated with student health. 6.3 RECOMMENDATIONS We present the following recommendations to CRS based on lessons learned from qualitative data obtained through remote fieldwork, document review, and quantitative data obtained from secondary sources. We do not intend the recommendations to address all challenges identified in the study. Rather, they focus on the main drivers of project success, as well as any required changes for future projects. The recommendations are grouped into two categories: best practices and sustainability. IMPAQ International, LLC Page 93 McGovern-Dole III Endline Evaluation Report 6.3.1 Best Practices with Respect to Current Activities Continue collaboration and engagement with communities and local governments. Across different types of stakeholders, respondents praised the level of engagement and collaboration between CRS and the entities involved in coordinating and executing project activities. CRS worked with individuals at the local governance level to train them on monitoring school performance and taking greater initiative and direction in supporting children’s education. SMC members shared that they collaborated with local governments to promote project activities. Stakeholders recommended encouraging communities to include continuing support for project activities in their planning, such as in their PDSEC. Increase sensitization on SILC support for schools. Stakeholders provided positive feedback on the establishment and operation of SILCs. They saw the SILC groups as an example of a good practice to continue moving forward. SILC groups empowered women to contribute to children’s needs and carry out projects for the community. Most SILC groups provided funds for members’ income-generating activities. However, there is room for more sensitization to motivate SILC groups to increase contributions to schools. Although SILC group members said that they provided funds or in-kind donations to canteens, other stakeholders said that only about half of SILC groups contributed to local school canteens. 6.3.1 Sustainability and Future Programming Support the government to expand ownership of canteens and apply the school feeding law. Stakeholders identified the school feeding component as an impactful intervention that was critical in encouraging school attendance. Stakeholders believed that this successful strategy, which supports both food security and educational goals,should continue. CRS has successfully built buy-in and capacity among key actors for operation and sustainability including the state, local authorities, and school management committees. Given the strategy’s importance, future program design should consider continuing what CRS started to promote its sustainability after this project ends through tracking execution of the school feeding law. 25 Future program implementers should focus on scaling up and assessing and strengthening capacity for the state, local authorities and communities to open and operate school canteens in new regions across Mali. This could also include exchanges visits between project areas successfully operating canteens to share lessons learned and best practices with new areas in other regions planning to open canteens. Engage with communities to mobilize resources for school canteens. Respondents noted that community support to provide funds and food for the canteen was critical for sustainability. Although 25 The School Feeding Law (Loi N˚2019-013) states that “the State, territorial collectivities, and communities support subsidies intended for school feeding” (Article 26, Title III, Chapter I). Additionally, “targeting, construction, and equipping canteens are the responsibility of the National Center for School Canteens” (Article 28, Title III, Chapter I). However, “territorial collectivities and communities must contribute to the facilitation and sustainability of school canteens” (Article 25, Title II, Chapter V of the Law). IMPAQ International, LLC Page 94 McGovern-Dole III Endline Evaluation Report communities were able to mobilize for short periods in the face of shortages or commodity delivery delays, there were challenges with communities providing adequate quantities of food to canteens. Stakeholders reported success with school gardens to improve the quality of hot meals, but they also noted difficulty with cultivating collective fields to provide food for canteens. We suggest strengthening agricultural capacities in the communities so that households can produce and contribute more. This support could include assistance with water points and irrigation to promote successful school gardens to enhance school meals. Similar to the second phase of the project, this could include assistance with construction or maintenance of water points and irrigation for schools. Communities could also continue selling crops harvested during the summer to generate supplemental income for schools. In addition to promoting food production and capacity to contribute to canteens, we suggest sensitization to raise awareness about the benefits of the school canteen, such as how it provided students with nutritional meals at school and increased student attendance, to motivate community members to provide support. Responses from national government stakeholders emphasized the need for increased community capacity to provide food and resources for the canteen. Flexible program design and tailored activities to meet regional differences. Consistent with the findings from the midline evaluation, we found persistent regional differences in most of the outcomes during endline. We observed that changes in Mopti were limited compared to the Koulikoro region. For example, students in Koulikoro reported much higher levels in reading proficiency over time. This finding was triangulated and confirmed by interviews with local stakeholders. Political instability in Mopti caused multiple school closures over years during the project which affected the project’s outcomes. However, McGovern-Dole III has made multiple efforts to mitigate these challenges which should be either scaled up and/or continued in future program design and implementation. For example, the project added a tutoring program in 27 secondary schools in Mopti in December 2018. The tutoring program selected the best students to tutor younger students in first-cycle schools, and provided them with THRs in return. The project also started training principals and community-paid teachers to address teacher turnover in public schools. In addition, future programs could consider setting separate targets tailored to each region to deliver activities adapted to the regional context or consider a reallocation of resources to ensure both regions improve (e.g., more training in Mopti compared to Koulikoro). Although certain external factors that affect outcomes are more prevalent in Mopti, such as political instability and terrorist activities, future program design and implementation should be more flexible and tailored to deliver activities adapted to the regional context to ensure both regions improve. Promote BLA at the national government level and collaborate with other partners. Education officials reported having the capacity to provide training on BLA, and principals and teachers indicated willingness to continue applying BLA techniques. However, stakeholders emphasized that the government must support BLA by incorporating it in a dedicated section of its training module for teachers. This could include working with IFM and IPEG to include BLA in teacher training curriculums. This would allow for teachers to build skills and knowledge to use best practices from BLA techniques. Additionally, while BLA is not nationally mandated, along with this project, other schools across the country are using BLA as part of the Selective Integrated Reading Activity (SIRA) project funded by the U.S. Agency for International IMPAQ International, LLC Page 95 McGovern-Dole III Endline Evaluation Report Development. Implementing partners for future programs should consider coordinating and collaborating with other US-funded education projects to exchange BLA lessons learned and enhance its reach. Promote regular teacher attendance in school. Preventing prolonged teacher strikes was beyond the scope of the current project. However, the endline results suggest that school closures over long periods of time lead to substantial reduction in instructional time (about 64 days during the 2018 – 2019 school year) which adversely affects student learning, even in the presence of promising programs such as BLA. The midline evaluation showed that students who were exposed for two or three years to teachers trained in BLA had significantly higher reading proficiency than students taught by teachers not trained in BLA. The endline evaluation, however, showed that Grade 2 students during 2018 – 2019 showed significantly lower reading skills than Grade 2 students two years earlier. Further interventions and incentives aimed at maintaining regular instructional time by encouraging high teacher attendance or promoting alternative instruction (e.g. via radio) will be beneficial for sustainability of the training efforts and for lasting effects on student learning. Promote sustainability of activities within COVID restrictions. CRS faced school closures due to COVID, teacher strikes, or security concerns and adapted programming effectively. For school feeding, adaptation included continuing to distribute commodities as THRs rather than as hot meals. Regarding student literacy, implementing partners can continue to explore options for remote lessons, such as mobile libraries and providing funding for printed learning materials and USB drives (in case of having access to technology) for students and teachers to use at home. Future implementers could continue working with MONE to ensure students have access to radios even in remote areas and providing radio-based lessons to mitigate the education loss caused by COVID. Moreover, if collecting data in person or remotely becomes feasible, future programs could conduct an assessment to understand how the applied mitigations strategies by the project affected various outcomes and use the findings to refine future design, given that no data have been collected after COVID. IMPAQ International, LLC Page 96 McGovern-Dole III Endline Evaluation Report References Ahmed, A. & del Ninno, C. (2002). The Food for Education Program In Bangladesh: An Evaluation Of Its Impact On Educational Attainment And Food Security. FCND Discussion Paper 138. https://ageconsearch.umn.edu/record/16395/ Bradley, B. & Greene, A. (2013). Do Health and Education Agencies in the United States Share Responsibility for Academic Achievement and Health? A Review of 25 Years of Evidence About the Relationship of Adolescents' Academic Achievement and Health Behaviors. Journal of Adolescent Health. 52:5, 523-532. https://www.sciencedirect.com/science/article/pii/S1054139X13000505 Center for International Education (CIE). (2016). Mali: Teacher Preparation and Continuing Professional Development in Africa (TPA). Brighton, England: University of Sussex. http://www.sussex.ac.uk/cie/projects/completed/tpa/mali. Cornille, T., Mullis, R., Mullis, A., Ritchson, A., & Sullender, N. (2004). Early Literacy Outcomes and Parent Involvement. Central Panhandle Regional Readiness HUBS. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.662.3746&rep=rep1&type=pdf COVID-19 Information | U.S. Embassy in Mali. (n.d.). Retrieved September 2, 2020, from https://ml.usembassy.gov/covid-19-information/ Dani, J., Burrill, C., & Demmig-Adams, B. (2005). The remarkable role of nutrition in learning and behaviour. Nutrition & Food Science. https://www.emerald.com/insight/content/doi/10.1108/00346650510605658/full/html Education Development Center. (2019). Rapport D'Evaluation Des Competences Des Eleves et Les Pratiques Pedagogiques. Fan, X., & Chen, M. (2001). Parental Involvement and Students' Academic Achievement: A Meta-Analysis. Educational Psychology Review. 13, 1-22. https://link.springer.com/article/10.1023/A:1009048817385 Fisher, K. (2001). Building Better Outcomes: The Impact of School Infrastructure on Student Outcomes and Behaviour. Schooling Issues Digest. https://eric.ed.gov/?id=ED455672 Fullan, M., & Langworthy, M. (2014). A Rich Seam: How New Pedagogies Find Deep Learning. London: Pearson. https://michaelfullan.ca/wp-content/uploads/2014/01/3897.Rich_Seam_web.pdf GardaWorld. (2019, March 12). Retrieved from Garda: https://www.garda.com/crisis24/news￾alerts/211806/mali-teachers-plan-further-strikes-march-11-april-5-update-2 Glewwe, P., Jacoby, H., & King, E. (2001). Early childhood nutrition and academic achievement: a longitudinal analysis. Journal of Public Economics. 81:3, 345-368. https://www.sciencedirect.com/science/article/abs/pii/S0047272700001183 Gorard, S., Huat See, B., Davies, P. (2012). The impact of attitudes and aspirations on educational attainment and participation. Joseph Rowntree Foundation. https://www.researchgate.net/profile/Stephen_Gorard/publication/265288552_The_Impact_of_Attitud es_and_Aspirations_on_Educational_Attainment_and_Participation/links/54ae6cd70cf24aca1c6fcf42.p df IMPAQ International, LLC Page 97 McGovern-Dole III Endline Evaluation Report Keating, E., Tadesse, G., Dejene, N. G., Yard, E., Appleby, L. J., & Cardwell, J. M. (2019). Local Perceptions of an Integrated School Health and Nutrition Programme involving WASH, school feeding and deworming in Southwest Ethiopia. https://osf.io/preprints/socarxiv/s9ac7/ Khatoon, R., Sachan, B., Khan, M., & Srivastava, J.P. (2017). Impact of school health education program on personal hygiene among school children of Lucknow district. Journal of Family Medicine and Primary Care. 6:1, 97-100. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5629909/ Leithwood, K., Jantzi, D., Earl, L., Watson, N. Levin, B., & Fullan, M. (2004) Strategic leadership for large‐scale reform: the case of England's national literacy and numeracy strategy. School Leadership & Management. 24:1, 57-79. Levinger, B. (2005). School Feeding, School Reform, and Food Security: Connecting the Dots. SAGE Journals. Food and Nutrition Bulletin. https://journals.sagepub.com/doi/abs/10.1177/15648265050262S207 Mattioli, F. (2016). Reforms to Increase Teacher Effectiveness in Developing Countries. 135. Rasberry, C., Lee, M., Robin, L., Laris, B.A., Russell, L., Coyle, & K., Nihiser, A. (2011). The association between school-based physical activity, including physical education, and academic performance: A systematic review of the literature. Preventive Medicine. 52:S, S10-S20. https://www.sciencedirect.com/science/article/abs/pii/S0091743511000557 Ross, J., Bruce, C., Esmonde, I., Dookie, L., & Beatty, R. (2010). The effects of sustained classroom-embedded teacher professional learning on teacher efficacy and related student achievement. Teaching and Teacher Education. 26:8, 1598-1608. https://www.sciencedirect.com/science/article/abs/pii/S0742051X10000922#! Tracking Conflict Worldwide | International Crisis Group. Retrieved July 2020 from: https://www.crisisgroup.org/crisiswatch/print?page=1&location%5B0%5D=26&date_range=cust&t=Cris isWatch+Database+Filter Ulukanligil, M. (2006). Community perception of school-based deworming program in Sanliurfa, Turkey. The American journal of tropical medicine and hygiene, 75(6), 1063-1068. https://www.ajtmh.org/content/journals/10.4269/ajtmh.2006.75.1063 IMPAQ International, LLC Page 98 McGovern-Dole III Endline Evaluation Report Appendices A. McGovern-Dole III Results Framework B. McGovern-Dole III Activity Status C. Key Evaluation Questions D. Unreportable Indicators E. Additional Tables and Complementary Outcomes F. McGovern-Dole III Indicators G. Qualitative Protocols IMPAQ International, LLC Page A1 McGovern-Dole III Endline Evaluation Report Appendix A: McGovern-Dole III Mali Results Framework MGD 1.2: Improved Attentiveness MGD 2.1: Improved Knowledge of Safe Food Preparation and Storage Practices MGD 2.5: Increased Access to Preventative Health Interventions MGD 1.1.1: More Consistent Teacher Attendance MGD 1.1.3: Improved Literacy Instructional Materials MGD 1.1.4: Increased Skills & Knowledge of Teachers MGD 1.1.5: Increased Skills & Knowledge of School Administrators MGD 1.2.1: Reduced Short￾Term Hunger MGD 1.3.1: Increased Economic & Cultural Incentives (Or Decreased Disincentives) MGD 1.3.2: Reduced Health￾Related Absences MGD 1.3.3: Improved School Infrastructure MGD 1.3.4: Increased Student Enrollment MGD 1.3.5: Increased Community Understanding of benefits of education Output: Teacher recognition Output: Distribution: Improved Literacy Materials Output: Training: Teachers Output: Training: School Administrators MGD S02: Increased Use of Health & Dietary Practices CRS assist SMC to access funds through mayors’ office to meet school needs Output: Enrollment Campaigns Output: Training: School Management Committees Training: Commodity management, Food Preparation and Storage Practices Distribution: De￾worming medication, vitamins & minerals Output: Teacher recognition Output: Student Recognition Foundational Results MGD SO2: Increased Use of Health and Dietary Practices Output: Capacity Building: Local, regional, national level MGD 1.4.1: Increased Capacity of Government Institutions MGD 1.4.3: Increased Government Support Output: Capacity Building: Local, regional, national level Output: Take Home Rations MGD 1.4.4: Increased Engagement of Local Organizations and Community Groups Output: Form Savings and Lending Groups Output: Training: Government Officials Output: Training: School Management Committees MGD 1.2.1.1, 1.3.1.1: Increased Access to Food (School Feeding) Output: Provide School Meals MGD 1.1: Improved Quality of Literacy Instruction MGD 1.3: Improved Student Attendance MGD SO1: Improved Literacy of School-Age Children IMPAQ International, LLC Page B1 McGovern-Dole III Endline Evaluation Report Appendix B: McGovern-Dole III Activities Exhibit 73. McGovern-Dole Activities Activities in common between MGD I through III Activities Only Specific to MGD I and II Activities Only Specific to MGD III De-worming medication, vitamins & minerals distribution Nutrition education activities through the positive deviance approach Capacity Building: Local, regional, national level (Sustainability) Enrollment campaigns Establishment of water points and school gardens Training: Government Officials (quality of Education) Formation of Savings and Lending Groups Hygiene and sanitation activities: provision of handwashing kits, construction of latrines and sensitization of communities on good hygiene and sanitation practices Distribution: Improved Literacy Materials (quality of Education) Provision of School Meals Teacher recognition (quality of Education) Student recognition Training: School Administrators (quality of Education) Take home rations Training: Teachers (quality of Education) Training: Commodity management, Food Preparation and Storage Practices Training: School Management Committees Exhibit 74. McGovern-Dole III Activity’s Status McGovern-Dole III activities Modification/Status due to COVID Distribution: De-worming medication, vitamins & minerals The second distribution should have been done in May but with COVID 19 this administration could not be carried out Enrollment campaigns (Spot radio, Door to door awareness) This activity was completed before the pandemic (it is conducted every year in September-October before the opening of classes). Form Savings and Lending Groups Continued with providing procedures on how to adapt to COVID￾19 in group meetings Provide School Meals With the closure of schools due to the COVID 19, the canteen was no longer functioning. Thus, meals began to be provided only to secondary students in grade 9 who resumed classes. Student recognition No modification Take home rations (THRs) With the COVID 19 pandemic, the special THRs for two months were distributed to all students in the project. Another special distribution of the remaining THR in the schools has been planned for all students at the end of June. Training: Commodity management, Food Preparation and Storage Practices With the cessation of the canteen’s operation, this activity had also ended. Training: School Management Committees Monitoring of commodities at school was suspended due to COVID. However, the commodity controllers have planned to run IMPAQ International, LLC Page B2 McGovern-Dole III Endline Evaluation Report McGovern-Dole III activities Modification/Status due to COVID the final monitoring visit to check the difference because of special distributions: Special THR distribution Covid-19 from April 23 to June 15, 2020 and the Special THR distribution for all students at the end of June 2020. Capacity Building: Local, regional, national level (Sustainability) The commitment to find opportunities for the intersectoral U.S. government was evident at the start of the education closure in Mali when the combined efforts of EDC projects in Mali were mobilized for the department so that interactive audio instruction (IAI) programs were tailored to domestic use because children stayed at home during the pandemic. EDC, through its IAI Adaptation Toolkit, has created programs to assist parents as home instructors. Training: Government Officials (quality of Education) No modification Distribution: Improved Literacy Materials (quality of Education) During COVID, principals were invited to make the reading manuals available to teachers. Teacher recognition (quality of Education) No modification Training: School Administrators (quality of Education) No modification Training: Teachers (quality of Education) No modification Teacher support monitoring EDC continued collecting teacher attendance data over the phone from school principals to calculate the average number of teaching days per teacher during COVID. Establishing a tutoring program to receive support from the 27 secondary schools to support primary students in Grade 1,2, and 3 in Mopti The project completed three out of four cohorts planned. However, after the school closures due to COVID, this activity could not be carried out any longer. Distribution -Illustrated card No modification Establishment of school garden fields No modification Establishment of Giant Scoreboards No modification SMCs Subvention activity No modification IMPAQ International, LLC Page C1 McGovern-Dole III Endline Evaluation Report Appendix C: Key Evaluation Questions Relevance Data Sources Ability to Address Research Question 1. Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? Project staff/partners National government Mayors Education officials USDA Fully 2. Are the activities and outputs of the program consistent with the intended impacts and effects? Project staff/partners USDA Fully 3. Does the program meet communities and government priorities? Project staff/partners National government Mayors Principals SMC SILC USDA Partially 4. Are stakeholders (management committee, parents, teachers, local authorities) satisfied with their participation in the program? Why or why not? Project staff/partners Education officials Principals SMC SILC Partially 5. Does the project align with government policies and programs (local, national)? Does the project align, and compliment other donor, other non-governmental organizations (NGO)s and/or local organizations managed programs? Project staff/partners National government Mayors Principals USDA Fully 6. To what extent are the objectives of the McGovern-Dole III intervention consistent with beneficiaries’ expectations, the country's needs, global priorities, political partners and USDA? Project staff/partners National government Mayors Principals Partially 7. To what extent were the objectives of the program valid? Project staff/partners National government Fully Effectiveness Data Sources Ability to Address Research Question 8. To what extent were the objectives of McGovern-Dole achieved / are likely to be achieved? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully IMPAQ International, LLC Page C2 McGovern-Dole III Endline Evaluation Report 9. Which activities have been affected the most by COVID-19? Project staff/partners Mayors Education officials Principals Teachers SILC Fully 10. What were the major factors influencing the achievement or non- achievement of the objectives? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully 11. Were the implementation strategies relevant and effective enough to improve: 1) enrollment and attendance among pupils particularly girls? 2) Community participation and engagement? 3) A better learning environment? Are there more effective strategies that would have agreater impact? Project staff/partners Mayors Principals Partially 12. What are the project’s major limitations? Project staff/partners Fully 13. Is the staffing structure and capacity sufficient and appropriate? Is the coordination mechanism effective? What if anything should be changed? Project staff/partners USDA Fully 14. Has program implementation been effectively monitored? How well did the monitoring and evaluation mechanism in place help the implementation of the project? Project staff/partners National government Principals USDA Fully 15. How did COVID-19 change efficiency of collecting regular and reliable data for monitoring and evaluation? Project staff/partners USDA Fully Performance and Impacts Data Sources Ability to Address Research Question 16. What is the overall project outcome to date? To what extent have project objectives and the yearly benchmark indicators have been achieved? What is facilitating or not the achievement of results and objectives in a timely manner? Project staff/partners National government Mayors USDA EDC Student survey Fully 17. Has COVID-19 affected any of the project outcomes? Project staff/partners National government Mayors USDA Fully 18. What evidence suggests that the BLA has contributed to improved literacy? Project staff/partners Principals EDC Student survey EDC Teacher survey Partially 19. Have there been changes in students’ attendance, particularly among girls? Project staff/partners National government Mayors Education officials CRS Attendance registry Fully IMPAQ International, LLC Page C3 McGovern-Dole III Endline Evaluation Report 20. Is the incentive strategy effectively promoting student attendance? Are strategy modifications needed to improve attendance? Please explain. Project staff/partners Principals CRS Attendance registry Partially 21. How has the project affected girls and boys? Is there an observable difference? What? Project staff/partners USDA EDC Student survey Partially 22. Have community barriers to education been identified? If so, how are they being addressed? How could the project better support behavior and social change? Project staff/partners National government Mayors Principals Partially 23. How are parents encouraged to be involved in their children’s education? How might they be encouraged to be more involved (including illiterate parents)? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Partially 24. How has teacher attendance and motivation changed? What more could be done? Project staff/partners Education officials Principals Teachers USDA SMC survey Partially 25. How do teachers find instructional materials? How are they using them? What could be done to promote greater/more effective use? Project staff/partners Principals Teachers SMC EDC Teacher survey EDC Principal survey Fully 26. How are community-based structures (e.g. schools, SMC, SILC) supporting project implementation? Are they on track to assume ownership of key activities beyond the life of the project? Are they satisfied with their participation? How might they be encouraged and/or supported to participate more? Fully 27. How have capacity building activities for SMC improved their capacities? What obstacles persist? What more should be done to ensure they will have the capacity to manage the school canteens beyond the life of the project? Project staff/partners Principals Teachers SMC Fully 28. What innovations, lessons learned, and good practices can be documented so far? Project staff/partners National government Education officials Teachers Fully IMPAQ International, LLC Page C4 McGovern-Dole III Endline Evaluation Report 29. What has happened as a result of the McGovern-Dole program and why? What real difference has the activity made to the beneficiaries? Project staff/partners National government Mayors Principals Teachers SMC EDC Student survey EDC Teacher survey EDC Principal survey SMC survey Fully 30. Did the theory of change to improve literacy through complementary support to student attendance, literacy instruction and student attentiveness hold? Why or why not? Mayors Principals Teachers SMC Partially Sustainability Data Sources Ability to Address Research Question 31. What activities and/or outcomes (both expected and unexpected) of the program are likely to be sustained? What evidence is there to suggest this? Education officials Principals Teachers SMC Partially 32. Has COVID-19 or its restrictions changed the activities planned for sustainability? How? Which activities were most affected by COVID? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully 33. What is the level of ownership acquired by the stakeholders? And how do they use? How can they evolve and / or continue the benefits resulting from the action after the end of the intervention? Principals Teachers Fully 34. What are the major factors which can influence the achievement or non- achievement of the sustainability of the project? Project staff/partners Principals SMC SILC Fully 35. How do the government’s capacities, policies, procedures, and priorities contribute to sustainability? Project staff/partners Principals SMC Fully 36. What strategies should be used to obtain long lasting support from communities and local/central administration that goes beyond the time of the project? Project staff/partners National government Mayors Education officials Principals Teachers SILC USDA Fully 37. How did capacity building enable community-based structures (e.g. schools, SMC, SILC) to support program implementation? To what degree of participation? Project staff/partners National government Principals Teachers Fully IMPAQ International, LLC Page C5 McGovern-Dole III Endline Evaluation Report 38. How has local, regional and national capacity changed regarding literacy instruction in treatment schools? School feeding programs? Student enrollment and attendance monitoring? Is there evidence that their capacity and ability to provide quality programming has improved? Project staff/partners Education officials Principals Teachers Fully 39. How have the national capacities, policies, procedures and priorities changed? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully 40. What innovations, lessons learned, and good practices may be taken away from the project? Project staff/partners National government Mayors Education officials Principals Teachers SMC USDA Fully 41. How could outcome replication or scaling up be supported by future interventions? Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully Efficiency Data Sources Ability to Address Research Question 42. Were objectives achieved on time? Project staff/partners SMC USDA Fully 43. Was the project implemented in the most efficient way compared to alternatives? Project staff/partners SMC SILC Fully 44. Does the food supply chain (including transport and storage) minimize loss and damages? Project staff/partners National government Education officials SMC Fully 45. Were resources managed in compliance to United States Government (USG) and USDA policies? National government Fully IMPAQ International, LLC Page C6 McGovern-Dole III Endline Evaluation Report 46. Are activities cost-efficient? Are objectives achieving on time? Is McGovern-Dole III implementing in the most efficient way compared to alternatives? (Efficiency and Value for Money) Project staff/partners National government Mayors Education officials Principals Teachers SMC SILC USDA Fully IMPAQ International, LLC Page D1 McGovern-Dole III Endline Evaluation Report Appendix D: Unreportable Outcomes Exhibit 75 lists outcomes reported by IMPAQ at midline including keys McGovern-Dole indicators that we could not report this time Exhibit 75. Unreportable Outcomes Indicator Student hunger, minimum acceptable diet, and dietary diversity Household food insecurity School infrastructure including food storage, access to water, latrines, etc. Student reasons for missing school Knowledge and self-reported handwashing practices at critical moments (students, principals, and parents) Knowledge on intestinal worms Caregivers’ involvement in prevention activities Caregivers’ aspirations for their children, school decision making, and reasons for supporting girls’ education Changes in households’ livelihood due to participation in SILC Challenges faced by school principals Number of general assemblies organized Best practices for food storage reported by SMC members SMC members passing best practices of safe food storage and preparation Proportion of teachers trained in BLA Percent of female students reporting they feel encouraged to participate in class by their teachers IMPAQ International, LLC Page E1 McGovern-Dole III Endline Evaluation Report Appendix E: Additional Tables Exhibit 76. Grade 1 Foundational Reading Skills Effects (𝜷), Same Schools (Overall, by Region, and by Gender) MGD Reading Proficiency Indicators Overall Koulikoro Mopti Girls Boys Phonological Awareness 0.852 (0.942) 2.603*** (0.896) -1.402 (1.773) 0.786 (1.116) 0.698 (0.991) Alphabet Knowledge 13.738*** (4.304) 27.468*** (4.853) -4.305 (5.955) 11.802** (5.043) 15.457*** (4.586) Word Recognition 1.850 (1.267) 4.161** (1.674) -1.163 (1.772) 1.808 (1.516) 1.703 (1.373) Decoding Ability 1.798* (0.924) 3.596*** (1.219) -0.574 (1.273) 1.784 (1.170) 1.764** (0.852) Oral Reading Fluency 1.934 (1.649) 4.843** (2.110) -1.862 (2.431) 2.130 (1.887) 1.705 (1.618) Reading Comprehension 0.206* (0.106) 0.334** (0.155) 0.045 (0.124) 0.215* (0.125) 0.201 (0.125) Listening Comprehension 0.190 (0.465) 0.122 (0.724) 0.305 (0.510) 0.155 (0.493) 0.264 (0.540) Total Number of Students 1,674 953 721 816 858 Total Number of Schools 61 35 26 61 61 Source: Student EGRA Data; * p-value < 0.1 ** p-value < 0.05 *** p-value < 0.01; Standard errors shown in parentheses are clustered at the school level. Note: This sample includes only schools with data in both 2015-16 and 2017-18. Exhibit 77. Grade 1 Reading Proficiency Effects (𝜷), Same Schools (Overall, by Region, and by Gender) MGD Reading Proficiency Indicators Overall Koulikoro Mopti Girls Boys Decoding Proficiency 0.124 (0.078) 0.287*** (0.082) -0.091 (0.132) 0.114 (0.093) 0.130* (0.073) Reading Proficiency at National Standard Level 0.045 (0.041) 0.098 (0.065) -0.023 (0.033) 0.048 (0.046) 0.048 (0.044) Reading with Comprehension Proficiency 0.021 (0.014) 0.023 (0.021) 0.020 (0.014) 0.025* (0.015) 0.019 (0.022) Total Number of Observations 1,674 953 736 816 858 Total Number of Schools 61 35 26 61 61 Source: Student EGRA Data; * p-value < 0.1 ** p-value < 0.05 *** p-value < 0.01; Standard errors shown in parentheses are clustered at the school level. Note: This sample includes only schools with data in both 2015-16 and 2017-18. IMPAQ International, LLC Page F1 McGovern-Dole III Endline Evaluation Report Appendix F: PMP McGovern-Dole III Indicators Due to COVID-19, data collection for some of the indicators was no longer feasible. IMPAQ and CRS used various datasets in different times to measure some of the indicators to report in Exhibit 76 of the endline evaluation report. Thus, “Final Percentage (Number)” column refers to the measured outcome from the latest available dataset, which could be different from one indicator to another. The timing issue in addition to the reduction in class time due to teacher strikes and COVID-19 may explain why some of the McGovern-Dole III targets were not met. Exhibit 78. McGovern-Dole III Indicators McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text IMPAQ/ Evaluation Boys: 2% Boys: 16% Boys: 8% 20% No Girls: 2% Girls: 13% Girls: 9% 10% No Overall: 2% Overall: 14.5% Overall: 9% 20% No Number of individuals benefiting directly from USDA-funded interventions CRS/ Monitoring Male: 0 32,618 36,920 37,935 No Female: 0 33,298 38,810 39,169 No Overall: 0 65,916 75,653 77,104 No Number of individuals benefiting indirectly from USDA-funded interventions CRS/ Monitoring 0 252,988 252,988 231,312 Yes Number of individuals benefiting directly from USDA-funded interventions (new) CRS/ Monitoring 0 10,796 66,755 2,699 Yes Number of individuals benefiting directly from CRS/ Monitoring 0 55,834 60,268 74,405 No IMPAQ International, LLC Page F2 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) USDA-funded interventions (continuing) Value of public and private sector investments leveraged as a result of USDA assistance (Host Government) CRS/ Monitoring 0 165,848 230,098 1,804,234 No Value of public and private sector investments leveraged as a result of USDA assistance CRS/ Monitoring 0 820,304 1,425,537 1,936,234 No Number of Parent-Teacher Associations (PTAs) or similar "school" governance structures supported as a result of USDA assistance CRS/ Monitoring 0 251 271 264 Yes Value of public and private sector investments leveraged as a result of USDA assistance (Other Public) CRS/ Monitoring 0 654,456 1,195,439 132,000 Yes Number of Savings and Internal Lending Community (SILC) groups supported as a result of USDA assistance CRS/ Monitoring 242 487 595 427 Yes Average amount of contribution per Savings and Internal Lending Community (SILC) group to CRS/ Monitoring 5 11.39 11 15 No IMPAQ International, LLC Page F3 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) school canteens (per year, in US dollar)1 Number of Savings and Internal Lending Community (SILC) groups contributing to their school canteen CRS/ Monitoring 171 279 291 300 No Number of individuals actively participating in Savings and Internal Lending Community (SILC) groups as a result of USDA assistance CRS/ Monitoring 3,993 10,993 13,619 7,500 Yes Number of household members benefitting from the creation of Savings and Internal Lending Community (SILC) groups formed as a result of USDA assistance CRS/ Monitoring 31,944 65,958 81,714 45,000 Yes Number of School Management Committee members trained on MONE modules CRS/ Monitoring 0 1,266 2,790 1,324 Yes Number of Action Plans created by School Management Committees as a result of USDA assistance CRS/ Monitoring 0 28 265 264 Yes Number of Community Giant Scoreboards created as a result of USDA assistance CRS/ Monitoring 0 254 254 264 No IMPAQ International, LLC Page F4 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Number of matching grants awarded to eligible School Management Committees CRS/ Monitoring 0 0 127 198 No Number of national-level organizational weaknesses in school canteen management addressed as a result of USDA assistance. CRS/ Monitoring 0 2 2 5 No Number of local, regional or national education officials participating in sustainability events CRS/ Monitoring 0 57 134 35 Yes Number of government officials certified as Teacher Trainers CRS/ Monitoring 0 36 52 36 Yes Number of trained government officials participating in the Early Grade Reading Assessment (EGRA) CRS/ Monitoring N/A 48 44 26 Yes Percent of students who demonstrate decoding abilities IMPAQ/ Evaluation Girls: 7% Girls: 17% Girls: 17% 21% No Boys: 9% Boys: 20% Boys: 22% 21% Yes Percent of students who reach the national reading standards by the end of the school year. EDC/ Evaluation 2% 13% 12% 12% Yes Average number of days present to teach per teacher CRS/ Monitoring 0 117 137 155 No IMPAQ International, LLC Page F5 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Percent of teachers who have received feedback from school structures CRS/ Monitoring 60% 65% 2% 80% No Number of teachers who have received feedback from school structures CRS/ Monitoring 0 319 2,029 144 Yes Number of teachers that have literacy instructional materials as a result of USDA assistance CRS/ Monitoring 0 488 583 703 No Number of textbooks and other teaching and learning materials provided as a result of USDA assistance CRS/ Monitoring 0 18,884 38,300 1,494 Yes Number of balanced literacy kits distributed to schools (French) CRS/ Monitoring 0 736 1,032 1,494 No Number of balanced literacy kits distributed to schools (Bamanankan) CRS/ Monitoring 0 39 64 180 No Number of balanced literacy kits distributed to schools (Soninke) CRS/ Monitoring 0 0 0 108 No Number of balanced literacy kits distributed to schools (Dogo-so) CRS/ Monitoring 0 3 14 78 No Number of students benefiting from the distribution of school supplies and materials CRS/ Monitoring 0 33,480 33,480 77,104 No Number of schools receiving school supplies CRS/ Monitoring 0 252 1,527 264 Yes IMPAQ International, LLC Page F6 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) and materials as a result of USDA assistance Number of teachers/educators/teachin g assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance CRS/ Monitoring 0 319 356 633 No Percent of girl students reporting they feel encouraged to participate in class by their teachers IMPAQ/ Evaluation 62% 65% 66% 10% Yes Number of teachers/educators/teachin g assistants trained or certified as a result of USDA assistance CRS/ Monitoring 0 488 488 703 No Number of school administrators and officials in target schools who demonstrate use of new techniques or tools as a result of USDA assistance CRS/ Monitoring 0 324 324 257 Yes Number of school administrators and officials trained or certified as a result of USDA assistance CRS/ Monitoring 0 357 357 293 Yes Percent of students in target schools identified by their teachers as attentive during class/instruction EDC/ Evaluation 50% 60% 58% 80% No IMPAQ International, LLC Page F7 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Percent of students in target schools who indicate that they are "not hungry" during the school day IMPAQ/ Evaluation 91% 91.5% Unreporteda 20% n/a Percent of school-age children receiving a minimum acceptable diet IMPAQ/ Evaluation Boys: 28% Boys: 34.2% Unreporteda 10% Girls: 29% Girls: 33.9% Unreporteda n/a Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance CRS/ Monitoring 0 Boys: 31,838 36,920 37,935 No Girls: 33,859 37,086 39,169 No Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (new) CRS/ Monitoring 0 10,796 65,697 2,699 Yes Number of school-aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (continuing) CRS/ Monitoring 0 52,451 60,268 74,405 No Number of daily school meals (breakfast, snack, lunch) provided to school￾age children as a result of USDA assistance CRS/ Monitoring 0 12,078,582 27,748,613 42,721,386 No IMPAQ International, LLC Page F8 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Number of take-home rations provided as a result of USDA assistance CRS/ Monitoring 0 354,886 1,023,771 19,499 Yes Number of individuals receiving take-home rations as a result of USDA assistance (new) CRS/ Monitoring 0 5,198 34,705 975 Yes Number of individuals receiving take-home rations as a result of USDA assistance (continuing) CRS/ Monitoring 0 9,202 10,442 18,524 No Number of individuals receiving take-home rations as a result of USDA assistance CRS/ Monitoring Boys: 0 Boys: 7,014 22,267 9,453 Yes Female: 0 Girls: 7,696 22,880 10,046 Yes Number of individuals receiving take-home rations as a result of USDA assistance (Others) CRS/ Monitoring 0 926 1,286 1,101 Yes Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance CRS/ Monitoring 0 Boys: 31,838 36,920 37,935 No 0 Girls: 33,859 37,086 40,270 No Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (new) CRS/ Monitoring 0 10,796 65,697 2,737 Yes IMPAQ International, LLC Page F9 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (continuing) CRS/ Monitoring 0 53,323 60,268 75,468 No Total quantity of commodities (MT) distributed as family rations to cooks as a result of USDA assistance CRS/ Monitoring 0 134.96 251 70 Yes Number of individuals trained in commodity management, food preparation and storage practices at the community￾level CRS/ Monitoring 0 1,380 2,112 1,324 Yes Number of school canteen cooks trained in safe food preparation and storage CRS/ Monitoring 0 375 374 1,101 No Number of government staff in relevant ministries/offices trained in commodity management, food preparation and storage practices CRS/ Monitoring 0 15 48 14 Yes Number of school-aged children receiving school meals (breakfast, snack, lunch) as a result of USDA assistance CRS/ Monitoring 0 65,697 74,006 77,104 No Number of individuals receiving take-home rations CRS/ Monitoring 0 15,478 46,045 20,600 Yes IMPAQ International, LLC Page F10 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) as a result of USDA assistance Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance CRS/ Monitoring 0 66,623 75,292 78,205 No Number of students regularly (80%) attending USDA supported classrooms/schools CRS/ Monitoring Boys: 0 Boys: 29,570 34,943 34,142 Yes Girls: 0 Girls: 30,294 34,938 35,252 No Average number of days per student of school attended CRS/ Monitoring 0 96.53 144 143 Yes Percent of 5th and 6th grade students having at least 90% school attendance CRS/ Monitoring 0 85% 96% 90% Yes Average number of days missed per student per school year due to student health issues CRS/ Evaluation 38 10 2 23 No Number of students receiving Vitamin A tablets CRS/ Monitoring 0 230,911 63,693 71,839 No Number of de-worming treatments provided CRS/ Monitoring 0 230,911 357,730 516,245 No Number of Vitamin A supplements provided CRS/ Monitoring 0 230,911 364,690 516,245 No Percent of households reporting school aged children NOT in school IMPAQ/ Evaluation 34.1% (1,963) 30.5% (2,077) 30.5% 23% Yes IMPAQ International, LLC Page F11 McGovern-Dole III Endline Evaluation Report McGovern-Dole Indicators Data Collection methods Baseline Percentage (Number) Midline Percentage (Number) Final Percentage (Number) Life of Project Target Life of Project Target Met? (Yes/No) Number of students enrolled in schools receiving USDA assistance CRS/ Monitoring 0 Boys: 31,838 36,920 39,169 No 0 Girls: 33,859 37,086 39,169 No Number of target communities benefitting from enrollment campaigns CRS/ Monitoring 0 252 252 264 No Number of target communities benefitting from community-level barrier analyses CRS/ Monitoring 0 252 252 80 Yes Percent of community members demonstrating knowledge of educational benefits CRS/ Evaluation 88.5% (2,338) 86.0% (2,431) 86% 92% No Number of students whose parents received illustrated report cards distributed to literate and illiterate parents CRS/ Monitoring 66,933 65,610 53,345 77,104 No Number of students who receive certificates that recognize academic achievement CRS/ Monitoring 0 2,288 4,134 5,280 No a COVID-19 prevented endline data collection and this was not included in the secondary data used for endline. Note: IMPAQ was not able to collect data for the 2019-2020 school year due to COVID-19. Therefore, indicator definitions and data sources differ at endline compared to baseline and midline and may not be comparable over time. Additionally, due to COVID-19 all “Final” indicators were measured using the latest available data which in some cases is the 2018-2019 school year. IMPAQ International, LLC Page G1 McGovern-Dole III Endline Evaluation Report Appendix G: Qualitative Protocols USDA Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 project in Mali. With your permission, I will audio record the discussion to assist with notetaking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? The interview will last about 60 minutes and will work best if you do most of the talking. Feel free to speak openly and candidly about your experiences and perspectives regarding this project. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. The data gathered through these interviews will be reported in an aggregate manner, highlighting informational points from specific sites and not from particular individuals. You will not be identified by name. Do you have any questions for me before we begin? Okay, let’s get started. Organization: Title: Gender: IMPAQ International, LLC Page G2 McGovern-Dole III Endline Evaluation Report Background 1. To begin, could you tell me your title and a bit about your role for the McGovern-Dole Food for Education (FFE 3) project in Mali? How long have you been involved with this project? Relevance 2. What do you think was the main goal of the FFE 3 project in Mali? What, specifically, was it trying to achieve? Do you think these were reasonable goals? Why or why not? 3. Do you think the planning and organization of the FFE 3 project were well-planned and realistic in terms of its objectives, desired outcomes, and targets? Why or why not? From your perspective, what were the strengths and weaknesses of the project’s design? From your understanding, to what extent did the project effectively consider economic, cultural, and political contexts? 4. How well and in what ways did the project align with USDA’s priorities and trends? Effectiveness 5. What outcomes did the project achieve? What factors influenced whether the project met its goals? (Probe on: student enrollment, attendance, and drop-outs especially for girls, community participation and engagement; improved learning environment) 6. Was implementation of some activities more successful than others? If so, which ones? Why? (Probe on: school meals, take home rations, BLA training, picture report cards) 7. How much did the Monitoring, Evaluation, Accountability and Learning (MEAL) mechanism help with implementation of the project? What improvements could have been made? 8. Overall, what were the successes and challenges experienced in implementing the project? How could they be addressed for better achievements in the future? Efficiency 9. Were project resources managed in compliance with U.S. Department of Agriculture (USDA) and U.S. Government policies? How? What were any challenges? IMPAQ International, LLC Page G3 McGovern-Dole III Endline Evaluation Report Impact 10. What were the impacts of the activities on communities where the FFE project was being implemented? Which activities do you think had the greatest impact? Which activities had the least impact? Why? How successful was BLA? What were incentives for teachers to use BLA? 11. Were there any external factors that prevented the project from achieving its goals? How did the project address those external factors? Sustainability 12. From your perspective, which activities and processes will be sustainable beyond project funding? Which will not be sustainable? What factors influence project sustainability? (Probe on school canteens, BLA literacy teaching approach, SILCs, SMCs.) 13. How do the government’s capacities, policies, procedures, and priorities contribute to sustainability? 14. From your perspective, what strategies should be used to obtain long-lasting support from communities and local and national levels of government that extends beyond the life of the project? 15. What specific efforts have been undertaken to prepare for the phase out of the project’s funding? How could the project be replicated or scaled up in the future? Overall 16. What were some innovations and best practices? Please share any lessons learned for future project phases. 17. How do you plan to use the findings from the evaluation? What key questions do you hope the evaluation will be able to inform? COVID-19 Now we would like to think about some of the questions above related to COVID-19 restrictions. 18. Did COVID-19 affect any of the project outcomes? If yes, how and to what extent? In your opinion, the impact of which activity was been affected the most by COVID? Why? 19. How did COVID-19 change efficiency of monitoring? Were any resources reallocated to address COVID-19 in the last months of the project? IMPAQ International, LLC Page G4 McGovern-Dole III Endline Evaluation Report 20. Has COVID-19 or its restrictions changed sustainability of activities? How? Sustainability of which activity has been affected the most by COVID? Conclusion 21. Is there anything that I did not ask about that you would like to share with me? or Do you have any additional thoughts about what we have discussed today? Thank you for your time and comments. IMPAQ International, LLC Page G5 McGovern-Dole III Endline Evaluation Report Staff projet/Partnaires Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 program in Mali. Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. With your permission, I will audio record the discussion to assist with note-taking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? [After starting the recording, say that consent was provided.] Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. Organisation: Titre/responsabilité: Sexe: IMPAQ International, LLC Page G6 McGovern-Dole III Endline Evaluation Report The interview will last about 90 minutes . Feel free to speak openly and candidly about your experiences and perspectives regarding this project. L’interview prendra environ 90 minutes . Sentez-vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. The data gathered through these interviews will be reported in an aggregate manner, highlighting informational points from specific sites and not from particular individuals. You will not be identified by name. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Do you have any questions for me before we begin? Okay, let’s get started. Auriez-vous des questions avant qu’on ne commence ? Background 22. What is your title? How long have you been with [CRS, EDC, CARITAS Bamako, Guamina, AMPRODE, CARITAS Mopti]? What are your main responsibilities in your position, particularly related to the FFE 3 project? How long have you been involved with this project? Quelle est votre fonction ? Depuis combien de temps travaillez-vous pour cette institution ? Quelles sont vos principales responsabilités dans cette fonction et en particulier en lien avec le projet Vivres pour l’éducation ? Depuis combien de temps êtes-vous impliqué dans ce projet ? Relevance 23. What do you think is the main goal of the FFE 3 project? What, specifically, is it trying to achieve? Do you think these are reasonable goals? Why or why not? IMPAQ International, LLC Page G7 McGovern-Dole III Endline Evaluation Report Selon vous quel est l’objectif principal visé par le programme Vivres pour l’Education 3 ? Quels sont les objectifs spécifiques liés à ce projet ? Pensez-vous que ces objectifs soient raisonnables ? Si oui/non pourquoi ? 24. Do you think the planning and organization of the FFE 3 project were well-planned and realistic in terms of its objectives, desired outcomes, and targets? Why or why not? Pensez-vous que la planification et l'organisation du projet FFE 3 étaient bien bonnes et réalistes en termes d'objectifs, de résultats escomptés et de cibles? Pourquoi ou pourquoi pas? 25. How do you think the FFE 3 project’s goals fit with the government’s priorities and goals? (Probe priorities at the national level, regional level, and local level, especially around scaling up the balanced literacy approach at the national level.) Dans quelle mesure pensez-vous que les objectifs du projet entrent en droite ligne avec les priorités du gouvernement? (Pensez aux priorités au niveau national, régional et local surtout celles concernant la mise à l’échelle au niveau national de l’approche équilibrée de lecture) 26. Since the project began, have beneficiaries’ needs changed over time in a way that has affected the project? If so, how has the project responded to changing needs? In your opinion, how satisfied are stakeholders with their participation in the FFE 3 project? Depuis le début du projet, les besoins des bénéficiaires ont-ils évolués au cours du temps de manière à affecter le projet ? Si oui, comment le projet a-t-il répondu à ces changements de besoins ? Effectiveness 27. What outcomes did the project achieve? To what extent were the project objectives achieved? What factors influenced whether the project met its goals? (Probe if not included by respondent: student enrollment, attendance, and drop-outs especially for girls, community participation and engagement; improved learning environment) Quels résultats ont pu être atteints dans le cadre du projet ? Dans quelle mesure les objectifs du projet ont-ils été atteints ? Quels facteurs ont influencé l’atteinte des objectifs du projet ? (Pensez au taux de scolarisation, de présence et à l’abandon surtout pour les filles, la participation et l’engagement communautaire, l’amélioration de l’environnement d’apprentissage) 28. Was implementation of some activities more successful than others? If so, which ones? Why? (Probe if not included by respondent: school meals, take home rations, BLA training, picture report cards) Y’a t-il des activités dont la mise en œuvre a réussi plus que d’autres ? Si oui, lesquelles et pourquoi ? (Pensez aux repas scolaires, rations à emporter, formation sur l’approche équilibrée, rapport imagés) IMPAQ International, LLC Page G8 McGovern-Dole III Endline Evaluation Report 29. Was the project’s staffing structure and capacity sufficient and appropriate? How well did CRS manage project partners? What could have been improved for coordination between CRS and project partners? Please provide examples. La structure du personnel du projet et leurs capacités ont-elles été appropriées ? Dans quelle mesure CRS a-t-il bien géré les partenaires de projet ? Qu’est ce qui aurait pu être amélioré dans la coordination entre CRS et les partenaires de projet ? Veuillez donner des exemples. 30. How effective was the monitoring strategy for collecting regular and reliable data on the project work? What were the strengths, challenges, and gaps in the monitoring system? Dans quelle mesure la stratégie de suivi-évaluation a-t-elle été efficace pour une collecte de données régulière et fiable ? Quelles ont été les forces, faiblesses et lacunes du système de suivi-évaluation ? 31. Overall, what were the successes and challenges experienced in implementing the project? How could they be addressed for better achievements in the future? Dans l’ensemble, quels ont été les enjeux/échecs et succès rencontrés au cours de la mise en œuvre du projet ? Comment le projet pourrait-il y remédier en vue de meilleurs résultats dans le futur ? Efficiency 32. To your knowledge, were project activities cost-efficient? Were the project objectives achieved on time? How could the FFE 3 project have been implemented more efficiently? A votre connaissance, les activités du projet ont-ils été cout-efficient ? Les objectifs du projet ont-ils pu être atteints à temps ? Comment le projet aurait-il pu être mis en œuvre de manière plus efficiente ? 33. How does the food supply chain for getting commodities from the United States to school canteens (including transport and storage) minimize loss and damages? Dans quelle mesure la chaine d’approvisionnement des vivres à partir des Etats Unis vers les cantines scolaires minimise-t-elle les pertes et dommages ? 34. Were project resources managed in compliance with U.S. Department of Agriculture (USDA) and U.S. Government policies? How? What were any challenges? Les ressources du projet ont-elles été géré en respectant les directives du département américain de l’agriculture ? Comment ? Quels ont été les enjeux ? Impact 35. What were the impacts of the activities on communities where the FFE project was being implemented? Which activities do you think had the greatest impact? Which activities had the least impact? Why? How successful was BLA? What were incentives for teachers to use BLA? What was the reason for starting tutoring? IMPAQ International, LLC Page G9 McGovern-Dole III Endline Evaluation Report Quels ont été les impacts des activités du projet sur les communautés bénéficiaires ? Selon vous quelles activités ont eu les impacts les plus importants et lesquelles ont eu les plus faibles ? Dans quelle mesure l’approche équilibrée a -t-elle été un succès ? Quels facteurs incitatifs ont motivé les enseignants à utiliser l’approche équilibrée ? Quelles ont été les motivations pour l’initiation du tutorat ? 36. Was there a difference in how project activities affected boys and girls? Please explain. Y’a t-il eu une différence dans la manière dont le projet a affecté les garçons et les filles ? Si oui expliquez ? 37. What barriers to children’s education were identified at the community level? How did the project address these barriers? How did project activities raise awareness among the community about the importance of education and promoting student attendance at school? Quelles barrières à l’éducation des enfants ont été identifiés au niveau communautaire ? Comment le projet s’est-il fait face à ces barrières ? Comment les activités du projet ont-elles suscité la prise de conscience communautaire sur l’importance de l’éducation et de promouvoir l’assiduité des élèves aux cours ? 38. Were there any external factors that prevented the project from achieving its goals? How did the project address those external factors? Y’a-t-il eu des facteurs externes qui ont mitigé l’atteinte des résultats du projet ? Comment le projet s’est￾il fait face à ces facteurs externes ? 39. How did community-based structures such as schools, SMCs, and SILCs support project implementation? What did the project do to build capacity for these groups? Are these groups on track to assume ownership of key activities (e.g. managing school canteens) beyond the life of the project? Please explain. Comment les structures Communautaires telles que les écoles, les CGS et les SILC ont-elles facilité la mise en œuvre du projet ? Quelles ont été les activités entreprises par le projet pour renforcer les capacités de ces structures ? Ces structures pourront-elles s’approprier les activités mises en œuvre après que le projet ait pris fin ? 40. How did the project affect student attendance? How did incentives such as providing school meals, take home rations, SMC grants, and teacher and student recognition, affect student attendance? Comment le projet a-t-il affecté le niveau de fréquentation/présence scolaire ? Dans quelle mesure les incitations telles que la fourniture de repas scolaires, rations à emporter, soutien des COGES, reconnaissance des enseignants et des élèves ont-elles affecté le niveau de présence des élèves aux cours ? IMPAQ International, LLC Page G10 McGovern-Dole III Endline Evaluation Report Sustainability 41. From your perspective, which activities and processes will be sustainable beyond project funding? Which will not be sustainable? What factors influence project sustainability? (If not mentioned in response, probe on sustainability of school canteens, BLA literacy teaching approach, SILCs, SMCs.) Selon vous, quelles activités et procédures pourront se pérenniser même après la fin des fonds projet ? Quelles activités ne seraient pas durables /pérennes ? Quels sont les facteurs ayant une influence sur la durabilité/pérennité du projet ? (Si non mentionné dans les réponses, évoquer la durabilité des cantines scolaires, de l’approche équilibrée d’enseignement, les SILC et CGS) 42. What is the level of ownership of project activities and outcomes by stakeholders at local and national levels? What will their roles be to sustain the project activities and outcomes after the project funding ends? What policies are in place at the national level to promote sustainability? Quel est le niveau d’appropriation des activités et résultats du projet par les partenaires au niveau local et national ? Quels seront leurs rôles dans la continuation des activités du projet après la fin des fonds du projet ? Quelles sont les politiques en place au niveau national pour promouvoir la durabilité ? 43. What specific efforts have been undertaken to prepare for the phase out of the project’s funding? How could the project be replicated or scaled up in the future? Quels efforts spécifiques ont-ils été entrepris en vue de se préparer à la fin du projet ? Comment ce projet pourrait-il être reproduit ou mis à l’échelle dans le futur ? 44. Do any socio-cultural or political aspects endanger the sustainability of the project and what actions are being taken to sensitize local institutions and target groups to these issues? Y’a t-il des facteurs socio-culturels et politiques qui peuvent menacer la pérennité du projet ? Si oui, quelles sont les actions entreprises pour sensibiliser les institutions et groupes cibles concernés sur ces enjeux ? Overall 45. What were some innovations and best practices? Please share any lessons learned for future project phases. Quelles ont été les innovations et les meilleures pratiques ? Veuillez partager toute leçon apprise pour le futur ? COVID-19 Now we would like to think about some of the questions above related to COVID-19 restrictions. A présent, nous souhaiterions aborder des questions spécifiques aux restrictions liées au COVID 19 IMPAQ International, LLC Page G11 McGovern-Dole III Endline Evaluation Report 46. Which activities have been affected the most by COVID? Has COVID-19 affected any of the project outcomes? Selon vous quelle est l’activité qui a été la plus affectées par le COVID ? Le COVID 19 a-t-il affecté d’un résultat quelconque du projet ? 47. How did COVID-19 change efficiency of collecting regular and reliable data for monitoring and evaluation? Comment le COVID a-t-il changé l’efficience dans la collecte régulière et fiable de données pour le suivi et l’évaluation ? 48. Has COVID-19 or its restrictions changed the activities planned for sustainability? How? Which activities were most affected by COVID? Le COVID ou ses restrictions ont-ils affecté les activitiés planifiées dans le cadre de la mise en œuvre de la stratégie de pérennisation/durabilité ? Dans quelle mesure ? Quelles activités ont le plus été affectées ? Conclusion 49. Is there anything that I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Y’a t-il quelque chose d’autre dont nous n’avons pas parlé et que vous souhaiteriez abordé avec moi ? Auriez vous des compléments d’informations à donner par rapport à tout ce qui a été abordé au cours de l’entretien d’aujourd’hui ? Thank you for your time and comments. Merci pour votre temps et vos éléments de réponse IMPAQ International, LLC Page G12 McGovern-Dole III Endline Evaluation Report National Government Stakeholders Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 project in Mali. Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. With your permission, I will audio record the discussion to assist with notetaking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière Organisation: Titre/responsabilité: Sexe: IMPAQ International, LLC Page G13 McGovern-Dole III Endline Evaluation Report à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. The interview will last about 60 minutes. Feel free to speak openly and candidly about your experiences and perspectives regarding this project. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. L’interview prendra environ 60 minutes. Sentez-vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuer et ceci sans aucun préjudice. The data gathered through these interviews will be reported in an aggregate manner, and not by particular individuals. You will not be identified by name. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Do you have any questions for me before we begin? Okay, let’s get started. Auriez-vous des questions avant qu’on ne commence ? Background 1. To begin, what is your title? Please tell me about your role at the [Ministry of National Education (MONE)/National Center for School Canteens (CNCS)] and with the FFE 3 project. Pour commencer, quelle est votre responsabilité ? Pouvez vous s’il vous plait m’en dire plus sur votre rôle au niveau du ministère de l’éducation nationale/Centre national des cantines scolaires et votre rôle dans le cadre du projet FFE 3 ? Relevance 2. What do you think were the main goals of the FFE 3 project? Do you think these were reasonable goals? Why or why not? IMPAQ International, LLC Page G14 McGovern-Dole III Endline Evaluation Report Selon vous quels étaient les objectifs clé du projet FFE 3 ? Pensez vous que ces objectifs aient été raisonnables ? Pourquoi ou pourquoi pas ? 3. How well did the FFE 3 project respond to the needs and priorities of [MONE/CNCS]? Please explain. 4. Dans quelle mesure le projet a-t-il bien répondu aux besoins et priorités du ministère de l’éducation national/Centre des Cantines Scolaires? Efficiency 5. To your knowledge, how well did the project collaborate with external stakeholders, such as government, including MONE/CNCS, and other NGOs? D’après vous; dans quelle mesure le projet a-t-il bien collaboré avec les partenaires externes tels que le gouvernement notamment le ministère de l’éducation nationale et le centre national de cantines scolaires et les autres ONG ? 6. How could collaboration between the project and government have been improved? La collaboration auraient-elle pu être meilleures ? Comment ? Effectiveness 7. To what extent do you think the FFE 3 project met its goals? How could it have been improved? Selon vous, dans quelle mesure le projet FFE 3 a-t-il atteint ses objectifs ? Comment le projet aurait-t-il pu être amélioré ? Impact 8. What were the impacts of the activities on communities where the FFE 3 project was implemented? Which activities do you think had the greatest impacts? Quels ont été les impacts des activités du projet dans les communautés bénéficiaires ? Quelles activités ont eu les impacts les plus importants ? 9. In your opinion, were there activities or results for the project that could have been strengthened or done differently? If yes, which activities and how could they have been strengthened? Selon vous, y’aurait-t-il des activités ou résultats du projet qui auraient pu être différemment ou renforcés ? Si oui, lesquels et comment corriger/renforcer ? 10. For MONE only: From your perspective, what are the barriers to children’s education? What have been the contribution of the project in reducing these barriers?? How did project activities raise awareness among the community about the importance of education and promoting student attendance at school? IMPAQ International, LLC Page G15 McGovern-Dole III Endline Evaluation Report Pour le ministère de l’éducation uniquement : Selon vous quelles sont les barrières à l’éducation des enfants ? Quelles ont été la contribution du projet dans la réduction de ces barrières ? Comment le projet a-t-il sensibilisé la communauté sur l’importance de l’éducation et la promotion de la présence des élèves à l’école ? Sustainability 11. From your perspective, which activities and processes will be sustainable beyond FFE 3 project funding? Which will not be sustainable? What factors influence project sustainability? Selon vous, quelles activités et procédures pourront se pérenniser même après la fin des fonds projet ? Quelles activités ne seraient pas durables /pérennes ? Quels sont les facteurs ayant une influence sur la durabilité/pérennité du projet ? 12. What role, if any, does [MONE/CNCS] have in ensuring sustainability of the FFE 3 project outcomes? How is your organization planning to take ownership of some program activities to ensure sustainability of project outcomes? For CNCS: Probe on school meals Le MEN/CNCS aurait-t-il un rôle à jouer en vue de pérenniser les acquis du projet FFE 3 ? Comment votre institution entend entreprendre l’appropriation des activités du projet dans le but de garantir la durabilité/pérennité des acquis ? Pour le CNCS : évoquez la question des repas scolaires. 13. How could the project be replicated or scaled up in the future? Comment le projet pourrait-t-il être répliqué ou mis à l’échelle ? 14. Has the project contributed to changes in national policies and priorities? Please share any examples. Le projet a-t-il contribué à des changements dans les politiques et priorités nationales ? S’il vous plait partagez des exemples ? COVID-19 Now we would like to think about some of the questions above related to COVID-19 restrictions. A présent, nous souhaiterions aborder des questions spécifiques aux restrictions liées au COVID 19 15. Did the project have contingency plans during COVID-19 pandemic? Did [MONE/CNCS] provide feedback for designing the contingency plan? Le projet a-t-il mis en place un plan de réponse d’urgence dans le contexte de la pandémie du Covid ? Est-ce que le ministère/CNCS a contribué à la formulation de ce plan d’urgence ? IMPAQ International, LLC Page G16 McGovern-Dole III Endline Evaluation Report 16. Do you think that in the future, COVID-19 restrictions will affect barriers to education ? Pensez vous que dans l’avenir les restrictions liées au Covid fortifieront les barrières à l’éducation ? 17. How has COVID-19 affected any of the project outcomes? De quelle manière le covid aurait-t-il affecté quelconque résultats escomptés du projet ? 18. How has COVID-19 changed the activities planned for sustainability? Comment le Covid a-t-il affecté les activités planifiés pour la durabilité ? Conclusion 19. Overall, what were some innovations, good practices, or lessons learned from the project? Dans l’ensemble, quelles ont été les innovations ; bonnes pratiques et leçons apprises dans le cadre du projet ? 20. Is there anything else that you would like to share with me today? Y’a t-il quelque chose dont nous n’avons pas parlé et que vous souhaiteriez abordé avec moi ? Thank you for your time and comments. IMPAQ International, LLC Page G17 McGovern-Dole III Endline Evaluation Report Maires Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 project in Mali. Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. With your permission, I will audio record the discussion to assist with notetaking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière Organisation: Titre/responsabilité: Sexe: IMPAQ International, LLC Page G18 McGovern-Dole III Endline Evaluation Report à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. The interview will last about 60 minutes. Feel free to speak openly and candidly about your experiences and perspectives regarding this project. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. L’interview prendra environ 60 minutes. Sentez-vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. The data gathered through these interviews will be reported in an aggregate manner, and not by particular individuals. You will not be identified by name. Do you have any questions for me before we begin? Okay, let’s get started. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Auriez-vous des questions avant qu’on ne commence ? Background 1. To begin, what is your title? Please tell me about your role with the FFE 3 project? How long have you been involved with this project? Pour commencer, quelle est votre responsabilité au sein de la Mairie? Pourriez vous me parler du rôle que vous jouez dans le cadre du projet FFE 3 ? Depuis combien de temps êtes-vous impliqué dans ce projet ? Relevance 2. What do you think were the main goals of the FFE 3 project? Do you think these were reasonable goals? Why or why not? Selon vous quels étaient les objectifs clé du programme FFE3 ? Pensez-vous que ces objectifs aient été raisonnables ? Si oui/non pourquoi ? IMPAQ International, LLC Page G19 McGovern-Dole III Endline Evaluation Report 3. From your understanding, to what extent did the FFE 3 project consider economic, cultural, social, and political contexts? Selon votre compréhension, est ce que le projet a tenu compte du contexte économique, culturel, social, et politique ? 4. How well did the FFE 3 project respond to the needs and priorities of your commune? Please explain. Dans quelle mesure le projet a-t-il bien répondu aux besoins et priorités de votre commune ? Project Implementation (Effectiveness and Efficiency) Mise en œuvre du projet (efficacité et efficience) 5. To what extent did the FFE 3 project collaborate with you as mayor? How could collaboration have been improved? Dans quelle mesure le projet FFE 3 a-t-il collaboré avec vous ? Cette collaboration aurait-elle pu être meilleure ? 6. From your knowledge, what did the project achieve? Selon vous, qu’est ce que le projet a réalisé comme acquis ? 7. What internal or external factors impacted the project’s success? From your knowledge, how did project implementers respond to such factors? Quels facteurs internes ou externes ont eu un impact sur la réussite du projet ? Selon vous , comment les responsables du projet ont-il tenu compte de ces facteurs ? Impact 8. Which components of the project brought the most positive change to the community and learning environment? Why? Selon vous quelle composante à apporter plus de changement positif dans la communauté et dans l’environnement d’apprentissage ? 9. What are the barriers to children’s education in your commune? Were there different barriers for boys and girls? How did the FFE 3 project contribute towards reducing these barriers in your commune? Quelles sont les barrières à l’éducation qui sont spécifiques à votre commune ? Ces barrières sont-elles différentes selon le sexe ? Quelle contribution du projet dans la réduction des barrières à l’éducation des enfants dans votre commune ? IMPAQ International, LLC Page G20 McGovern-Dole III Endline Evaluation Report 10. Did project activities raise awareness among the community about the importance of education and promoting student attendance at school? How? Comment le projet a-t-il sensibilisé la communauté sur l’importance de l’éducation et a promu la présence/l’assiduité des élèves à l’école ? Sustainability 11. From your perspective, which activities and processes will be sustainable beyond FFE 3 project funding? Which will not be sustainable? What factors influence project sustainability? Selon vous, quelles activités et procédures pourront se pérenniser même après la fin des fonds projet ? Quelles activités ne seraient pas durables /pérennes ? Quels sont les facteurs ayant une influence sur la durabilité/pérennité du projet ? 12. With respect to sustainability, how did the commune prepare for the absence of support from USDA and CRS? Par rapport à la pérennisation des activités et acquis du projet ; comment votre commune se prépare t￾elle à prendre le relais une fois que les bailleurs notamment USDA et CRS se seront retirés ? COVID-19 Now we would like to think about some of the questions above related to COVID-19 restrictions. A présent, nous souhaiterions aborder des questions spécifiques aux restrictions liées au COVID 19 13. How has COVID-19 affected any of the project activities and outcomes? Le Covid a-t-il affecté une quelconque des activités ou résultats du projet ? Si oui, comment ? 14. Has COVID-19 changed any of the activities planned for sustainability? Please explain how. Comment le Covid a-t-il affecté les activités planifiés pour la durabilité ? Conclusion 15. Overall, reflecting on implementation and outcomes of the program so far, what lessons have you learned? What have you seen as good practices? What were limitations with the program? D’une manière Générale, tenant compte de la mise en œuvre et des résultats atteints jusque là dans le cadre du programme ; quelles leçons avez-vous apprises ? Quelles sont les bonnes pratiques que vous avez pu constater ? Quelles ont été les limites du projet ? 16. Is there anything that I did not ask about that you would like to share with me? IMPAQ International, LLC Page G21 McGovern-Dole III Endline Evaluation Report Y’a t-il quelque chose dont nous n’avons pas parlé et que vous souhaiteriez abordé avec moi ? Thank you for your time and comments. Merci pour votre temps et vos commentaires IMPAQ International, LLC Page G22 McGovern-Dole III Endline Evaluation Report Officiels de l’administration scolaire – Conseillers en éducation, CAP, AE Directeurs Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. L’interview prendra environ 60 minutes et se déroulera très bien si vous êtes disposé à parler. Sentez￾vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Auriez-vous des questions avant qu’on ne commence ? Organisation: Titre/Responsabilité: Sexe: IMPAQ International, LLC Page G23 McGovern-Dole III Endline Evaluation Report Background 17. Let’s start by talking about your background. How long have you been an education official? What is your title? How long have you been in this role? Commençons par parler de vous : votre niveau d’ancienneté en tant que cadre de l’éducation ? fonction occupée ? Votre rôle et vos responsabilités ? 18. Can you describe how you and your department are involved with this project? What is your role and responsibilities related to the FFE 3 project? S’il vous plait, pouvez vous nous décrire le rôle ou les responsabilités qui incombent à votre service dans le cadre de mise en œuvre du projet FFE3 ? Vos responsabilités spécifiques dans ce cadre de ce projet ? 19. How was the collaboration with the project staff (CRS and partners)? Is there anything that could have been done better in terms of collaboration? Comment s’est passé la collaboration entre vous et CRS et ses partenaires au cours de la mise en œuvre du projet ? Pensez vous que cette collaboration aurait pu être meilleure ? Si oui, en quoi celle-ci aurait pu être améliorée ? Training I would like to learn about the training you have received through the FFE 3 project to support teachers’ implementation of the balanced literacy approach (BLA) in the classroom. Je souhaiterais à présent que nous parlions de la formation que vous avez reçu sur l’approche équilibrée dans le cadre du projet FFE3 en vue d’accompagner les enseignants dans la mise en œuvre de cette approche dans les écoles. 20. Could you describe the training you received? (Probe for BLA training )Who provided it, and when/how often? What did you like about these trainings? S’il vous plait pourriez vous nous décrire la formation que vous avez reçu (évoquez la formation sur l’approche équilibrée ). Qui vous a dispensé cette formation ? Ou l’avez-vous reçu et combien de fois avez￾vous reçu cette formation ? Quel aspect de cette formation avez-vous particulièrement aimé ? 21. Was the training you received sufficient for you to train and support principals and teachers in your area? Pensez-vous que les formations reçues (leur nombre et leur qualité) vous ont permis d’être suffisamment outillés pour appuyer et accompagner les directeurs d’écoles et enseignants de votre zone ? 22. How do you think teachers were using the balanced literacy approach techniques? Do you think they will continue to use the BLA techniques in the future? IMPAQ International, LLC Page G24 McGovern-Dole III Endline Evaluation Report Comment les enseignants utilisent-t-ils l’approche BLA ? Pensez-vous qu’ils continueront à l’utiliser ? 23. How often did you observe teachers in the classroom? How receptive were teachers to getting feedback? How could the observation and feedback process have been improved? A quelle fréquence faites-vous des visites d’observation de classes ? Les enseignants sont-t-ils réceptifs à accueillir les retours et suggestions que vous leur faites après les visites d’observations ? Pensez vous qu’il y ait quelque chose à améliorer en matière de visites d’observations et des retours faits aux enseignants ? Student and Teacher Outcomes Elèves et enseignants Now I would like to learn about your perspective on changes in schools in your area over the last year. A présent, je souhaiterais que l’on parle des éventuels changements que vous auriez observé au niveau scolaire dans votre zone ? 24. In what ways has the quality of teaching changed since teachers received literacy instruction trainings and materials? Which tools or techniques do you think have been the most impactful on the quality of teaching? Pensez-vous que la qualité de l’enseignement a changé à la suite des formations et matériels didactiques reçus ?Quelle(s) technique(s) particulière(s) ou matériel(s) ont eu le plus d’impact sur la qualité de l’éducation ? 25. Since receiving training, what changes have you observed in teachers? (Probe on teacher attendance and motivation) Depuis que le projet a initié les formations, avez-vous constaté des changements quelconques dans le comportement des enseignants (assiduité aux cours, niveau de motivation) ? 26. How has FFE 3 had an influence on students in your school? What are the key achievements, if any, resulting from the program? Are there any differences in the way it is affecting boys and girls? Probe for:  Increased student enrollment  Increased attendance  Decreased drop-outs (especially girls)  Improvements in student literacy  Increased attentiveness Comment le programme FFE 3 a-t-il eu une influence sur les élèves de votre école ? Quels ont été les principaux acquis ou résultats atteints dans le cadre de ce programme ? Le programme affecterait-t-il différemment les filles et les garçons ? Evoquez les aspects suivants:  Augmentation des taux d’inscription scolaires  Niveau de fréquentation scolaire ou de présence à l’école IMPAQ International, LLC Page G25 McGovern-Dole III Endline Evaluation Report  Diminution des abandons scolaires surtout pour les filles  Amélioration du niveau de lecture des élèves  Augmentation du niveau de concentration/ d’attention des élèves 27. (If not answered above) What aspects or activities of the project had the most impact on the results you just mentioned? What aspects had the least impact? (Si non mentionné en haut), quels aspects ou activités du projet ont eu le plus d’impact sur les résultats que vous avez mentionné précédemment ? Quels aspects ont eu le moins d’impact ? Sustainability Durabilité 28. From your perspective, which activities and processes will be sustainable beyond project funding and which will not be sustainable? Please explain. Selon vous, quelles activités et procédures seront à même d’être poursuivis au-delà de la fin du projet dans une optique de pérennisation? Quelles sont celles qui ne seront pas viables au-delà de la fin du projet ? Pourquoi ou Pourquoi pas? 29. What role, if any, does your department have in continuing FFE 3 project activities and sustaining outcomes achieved? Votre service aurait-t-il un rôle spécifique à jouer dans l’appropriation et la poursuite des activités du projet en vue d’en garantir la pérennité ? COVID-19 Now we would like to think about some questions related to COVID-19 restrictions. A présent parlons des questions spécifiques liées au Covid 19 et à ses restrictions 30. Which project activities were most affected by COVID-19? What measures, if any, were adopted to mitigate that? 31. Quelles sont les activités qui ont été les plus impacté par le COVID 19 ? Quelles sont les éventuelles mesures d’atténuation prises pour y faire face ? 32. Has COVID-19 or its restrictions changed the activities and efforts planned for sustainability? How? Le Covid a-t-il affecté les activités et les efforts planifiés en vue d’en assurer la durabilité/pérennité? Relevance, Good Practices, and Lessons Learned IMPAQ International, LLC Page G26 McGovern-Dole III Endline Evaluation Report Pertinence, Bonnes pratiques et leçons apprises 33. Overall, how well do you think that the FFE 3 project responded to the needs and priorities of the community? Dans l’ensemble, pensez-vous que le projet FFE3 a-t-il bien répondu aux besoins et priorités de la communauté ? 34. Overall, how satisfied are you with participating in the FFE 3 project? Dans l’ensemble quelle est votre niveau de satisfaction en tant que participant aux activités du projet FFE 3 ? 35. Reflecting on the project, what lessons have you learned? What have you seen as good practices? What are limitations with the program? Rétrospectivement, quelles leçons avez-vous apprises ? Avez-vous vu de bonnes pratiques ? Si oui donnez-nous quelques exemples ; quelles ont été les limites ou insuffisances du projet ? 36. Is there anything that I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Auriez-vous autre chose à partager avec moi avant la fin de cet entretien ? Thank you for your time and comments. IMPAQ International, LLC Page G27 McGovern-Dole III Endline Evaluation Report Directeurs d’école Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 project in Mali. Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. With your permission, I will audio record the discussion to assist with notetaking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. Organisation: Titre/Responsabilité: Sexe: IMPAQ International, LLC Page G28 McGovern-Dole III Endline Evaluation Report The interview will last about 60 minutes. Feel free to speak openly and candidly about your experiences and perspectives regarding this project. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. L’interview prendra environ 60 minutes et se déroulera très bien si vous êtes disposé à parler. Sentez￾vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. The data gathered through these interviews will be reported in an aggregate manner, and not from particular individuals. You will not be identified by name. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Do you have any questions for me before we begin? Okay, let’s get started. Auriez-vous des questions avant qu’on ne commence ? Background 1. What is your title? How long have you been at this school? Quel est votre titre ou responsabilité ? Depuis combien de temps êtes-vous dans cette école ? 2. How long has your school been involved with this project? What are your main responsibilities in your position related to the FFE 3 project? Depuis combien de temps votre école est-elle impliquée dans ce projet ? Quelles sont vos principales responsabilités en lien avec le projet FFE 3 ? Training and Teacher Oversight Formation et supervision des enseignants IMPAQ International, LLC Page G29 McGovern-Dole III Endline Evaluation Report 3. I want to start by talking about the balanced literacy approach. From your perspective, were the training and instructional materials supplied by the project sufficient? Je souhaiterais aborder la question de l’approche équilibrée d’apprentissage de la lecture. Selon vous, la formation et le matériel didactique acquis dans le cadre de ce projet ont-ils été suffisants ? 4. How do you think teachers were using the balanced literacy approach techniques? Do you think they will continue to use the BLA techniques in the future? En quoi vous pensez que les enseignants appliquaient l’approche équilibrée ? Pensez-vous qu’ils continueront à utiliser cette approche dans un futur proche et lointain ? 5. How often did you observe teachers in the classroom? How receptive were teachers to getting feedback? How could the observation and feedback process have been improved? Do you think you will continue with observation and feedback in the future? A quelle fréquence avez-vous tenu une session d’observation des cours de vos enseignants ? Dans quelle mesure les enseignants étaient-t-ils disposés à tenir compte de vos suggestions, remarques et recommandations à l’issue des observations ? Comment l’approche et le processus d’observation des cours incluant les suggestions et remarques faites aux enseignants auraient-t-ils pu être améliorés ? 6. Since receiving training, what changes have you observed in teachers? (Probe on teacher attendance and motivation) Depuis la mise en œuvre du projet, avez-vous constaté des changement de comportement chez vos enseignants ? (Creuser la question pour apprécier l’évolution du niveau de présence et de motivation des enseignants) Student Outcomes Résultats en lien avec les élèves ou apprenants Now I would like to learn about your perspective on changes at your school since the FFE program began. A présent ; je souhaiterais en savoir davantage sur votre opinion à propos des changements observés dans votre école depuis le début du projet ? 7. How has FFE 3 had an influence on students in your school? What are the key achievements, if any, resulting from the program? Are there any differences in the way it is affecting boys and girls? Probe for:  Comment le programme FFE 3 a-t-il eu une influence sur les élèves de votre école ? Quels ont été les principaux acquis ou résultats atteints dans le cadre de ce programme ? Le programme affecterait-t-il différemment les filles et les garçons ? Evoquez les aspects suivants: IMPAQ International, LLC Page G30 McGovern-Dole III Endline Evaluation Report  Increased student enrollment  Augmentation des taux d’inscription scolaires  Increased attendance  Niveau de fréquentation scolaire ou de présence à l’école  Decreased drop-outs (especially girls)  Diminution des abandons scolaires surtout pour les filles  Improvements in student literacy  Amélioration du niveau de lecture des élèves  Increased attentiveness  Augmentation du niveau de concentration/ d’attention des élèves 8. (If not answered above) What aspects or activities of the project had the most impact on the results you just mentioned? What aspects had the least impact?  (Si non mentionné en haut), quels aspects ou activités du projet ont eu le plus d’impact sur les résultats que vous avez mentionné précédemment ? Quels aspects ont eu le moins d’impact ? Community Attitudes Attitudes de la communauté 9. What barriers to education are there in your community? How did the project address these barriers? How did project activities raise awareness among the community about the importance of education and promoting student attendance at school?  Quelles sont les barrières à l’éducation qui existent au sein de votre communauté ? Comment le projet s’est-t-il adressé à ces barrières ? Comment le projet a-t-il sensibilisé la communauté sur l’importance de l’éducation et de la fréquentation scolaire ? 10. How concerned are parents and caregivers about their children’s performance at school? Did the project change their attitudes? If yes, in what ways? What factors could encourage parents to be more involved with their children’s education, particularly illiterate parents? 11. Les parents d’élèves et les tuteurs se préoccupent-ils du rendement/de la performance de leurs enfants à l’école ? Le projet a-t-il changé leurs attitudes ? Si oui de quelle manière ? Quels facteurs pourraient encourager les parents (surtout les parents analphabètes) à s’impliquer davantage dans l’éducation de leurs enfants ? IMPAQ International, LLC Page G31 McGovern-Dole III Endline Evaluation Report Project Implementation (Effectiveness and Efficiency) Mise en œuvre du projet (efficacité et efficience) 12. Was the implementation of some activities more successful than others? If so, which ones? Why? Probe if not included by respondent: La mise en œuvre de certaines activités a-t-elle réussit mieux que d’autres ? Si oui, lesquelles ? Pourquoi ? Evoquez ces aspects si non mentionné par le répondant  School meals  Repas scolaires  Take home rations  Rations à emporter  Balanced literacy approach (BLA) training Approche équilibrée pour la lecture  Picture report cards Carnets illustratifs de performance scolaire 13. (If not answered earlier) How have SMCs and SILCs supported FFE 3 project activities at your school? Si non mentionné ; comment les COGES et SILC ont-t-ils accompagné les activités du projet FFE 3 dans votre école ? 14. Overall, what were the challenges experienced in implementing project activities? How could they be addressed for better achievements in the future? Dans l’ensemble, quelles ont été les défis rencontrés dans la mise en œuvre des activités du projet ? Comment ces défis pourraient-ils être relevés à l’avenir pour de meilleurs résultats ? 15. In your experience, did the project activities proceed on schedule? What helped to stay on schedule and what made it difficult? (Probe: receiving food commodities, literacy materials) D’après votre expérience avec le projet, les activités du projet ont-elles été mises en œuvre dans les délais impartis ? Qu’est ce qui a permis de rester dans ces délais et qu’est ce qui l’en a empêché ? Sustainability Durabilité 16. From your perspective, which activities and processes in your school will be sustainable beyond project funding and which will not be sustainable? Please explain. IMPAQ International, LLC Page G32 McGovern-Dole III Endline Evaluation Report Selon vous , quelles activités et procédures seront à même d’être poursuivis au-delà de la fin du projet dans une optique de pérennnisation? Quelles sont celles qui ne seront pas viables au-delà de la fin du projet ? Pourquoi ou Pourquoi pas? 17. With respect to sustainability, how has your school prepared for the absence of support from USDA and CRS? What is your role? Comment votre école s’est-elle préparée à prendre le relais des activités et à capitaliser les acquis du projet une fois à terme (retrait de USDA et CRS) ? Quel est votre role en ce sens? COVID-19 If not answered earlier: Si non mentionné précédemment 18. Which project activities were most impacted by COVID-19 restrictions? What measures, if any, were adopted to mitigate that? Quelles sont les activités qui ont été les plus impacté par les restrictions du COVID 19 ? Quelles sont les éventuelles mesures d’atténuation prises pour y faire face ? 19. In your opinion, how will COVID-19 restrictions affect future enrollment and students’ overall literacy? Selon vous, comment les restrictions liées au COVID affecteront la scolarisation et le niveau de lecture des élèves ? 20. Has COVID-19 changed the activities and efforts planned for sustainability? Le Covid a-t-il changé les activités et les efforts planifiés en vue d’assurer la durabilité/pérennité des activités du projet? Relevance, Good Practices, and Lessons Learned Pertinence, Bonnes pratiques et leçons apprises 21. Overall, how well do you think that the FFE 3 project responded to the needs and priorities of your community? Dans l’ensemble, pensez-vous que le projet FFE3 a-t-il bien répondu aux besoins et priorités de votre communauté ? 22. What were some innovations and good practices? Please share any lessons learned for potential future project phases. IMPAQ International, LLC Page G33 McGovern-Dole III Endline Evaluation Report Quelles ont été les innovations et les bonnes pratiques ? Pourriez-vous partager quelques leçons apprises à capitaliser pour le futur ? 23. Overall, how satisfied are you with participating in the FFE 3 project? Dans l’ensemble quelle est votre niveau de satisfaction en tant que participant aux activités du projet FFE 3 ? Conclusion 24. Is there anything that I did not ask about that you would like to share with me? Auriez-vous autre chose à partager avec moi avant la fin de cet entretien ? Thank you for your time and comments. Merci pour votre disponibilité et votre contribution IMPAQ International, LLC Page G34 McGovern-Dole III Endline Evaluation Report Enseignants Note to interviewer: We want to know primarily about responses to questions thinking broadly about the time period before COVID-19. We will ask more questions about COVID-19 specifically at the end of the interview. Throughout the interview, please ask the respondent to provide specific examples (especially after very short responses (e.g. yes, no, good, very well). Note pour l’interviewer : En général noussouhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Good morning/afternoon. My name is [insert name]. I am working with IMPAQ International, a US-based research company. The purpose of this interview is to hear about your experience related to the Food for Education (FFE) 3 project in Mali. Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. With your permission, I will audio record the discussion to assist with notetaking. No one outside the evaluation team will have access to this recording. Do I have your permission to record the conversation? Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière Organisation: Titre/Responsabilité: Sexe: IMPAQ International, LLC Page G35 McGovern-Dole III Endline Evaluation Report à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. The interview will last about 60 minutes. Feel free to speak openly and candidly about your experiences and perspectives regarding this project. Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty. L’interview prendra environ 60 minutes et se déroulera très bien si vous êtes disposé à parler. Sentez￾vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. The data gathered through these interviews will be reported in an aggregate manner, and not from particular individuals. You will not be identified by name. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Do you have any questions for me before we begin? Okay, let’s get started. Auriez-vous des questions avant qu’on ne commence ? Background 50. Let’s start by talking a little bit about your background. How long have you been teaching? How long have you been teaching at this school? What grade do you usually teach? Si vous le permettez nous allons commencer par parler de vous c’est-à dire votre parcours. Depuis combien de temps êtes-vous dans l’enseignement et depuis combien de temps enseignez-vous dans cette école ? Quelle(s) classe(s) enseignez-vous typiquement ? Balanced Literacy Approach (BLA) Approche équilibrée pour l’apprentissage de la lecture IMPAQ International, LLC Page G36 McGovern-Dole III Endline Evaluation Report 51. I would like to get a better understanding of the literacy instruction training and materials you have received through the project. Were you satisfied with the quantity and quality of materials and supplies provided by the project? Why or why not? What other materials and supplies would be helpful? Je voudrais que l’on parle des formations et du matériel que vous auriez reçu dans le cadre de l’approche d’apprentissage de la lecture initiée par ce projet ? Etiez-vous satisfaits de la quantité et de la qualité du matériel didactique reçu de la part du projet ? Pourquoi ou pourquoi pas ? Quels autres matériels et équipements auraient été utiles ? 52. How useful was the literacy training for you? Did you face any challenges in using the literacy teaching techniques? Please explain. La formation sur l’approche d’apprentissage de la lecture a-t-elle été utile ? Quels sont les défis auxquels vous faites face dans l’application des techniques apprises ? 53. Do you think the application of these techniques and materials had an influence on your students’ literacy skills? If yes, in which sense and to what extent? Pensez-vous que l’application de l’approche équilibrée dans votre classe a eu une influence sur les capacités de lecture et d’écriture de vos élèves ? Si oui, dans quel sens et dans quelle mesure ? 54. Now I would like to learn about classroom observations and support from school administrators. How often did school administrators observe your classroom during the school year? How helpful was their feedback? In what ways have you incorporated observation feedback into your teaching techniques, if any? How could the observation and feedback process be improved? A présent, je souhaiterais que l’on parle des séances d’observations de classes conduites par les conseillers pédagogiques. A quelle fréquence avez-vous reçu de telles visites pendant l’année scolaire ? Dans quelle mesure leurs rapports d’observations et/ou remarques vous ont-t-ils été utiles ? Comment avez-vous pris en compte ces observations dans vos cours ? Le processus d’observation et de rapportage peut-t-il être amélioré ? Si oui comment ? Teacher Attendance and Motivation Assiduité des enseignants aux cours et motivations 55. Is it sometimes the case that some teachers in your school are not able to come to school for reasons other than health or family concerns? Why? Arrive-t-il souvent que des enseignants n’arrivent pas à être présents à l’école pour la tenue des cours pour des raisons autres que la santé ? Pourquoi ? IMPAQ International, LLC Page G37 McGovern-Dole III Endline Evaluation Report 56. Do you think that some teachers are sometimes not motivated to come to school? Why? If so, what could be done to motivate them? Pensez-vous qu’il y’a des enseignants qui ne sont pas souvent motivés pour aller à l’école ? Pourquoi ? Si oui, qu’est-ce qui pourrait être fait pour les motiver ? Student Outcomes Résultats liés aux élèves I would like to learn about your perspective on changes at your school since the FFE program began. Je souhaiterais en savoir davantage sur les changements constatés au niveau de votre école depuis la mise en œuvre du projet FFE. 57. How has FFE 3 had an influence on students in your school? What are the key achievements, if any, resulting from the program? Are there any differences in the way it is affecting boys and girls? Probe for: Comment le programme FFE 3 a-t-il eu une influence sur les élèves de votre école ? Quels ont acquis ou résultats clé atteints dans le cadre de ce projet ? Le projet affecterait-t-il différemment les filles et les garçons ? :  Increased student enrollment  Augmentation du taux d’inscription scolaire  Increased attendance  Niveau de fréquentation scolaire ou de présence à l’école  Decreased drop-outs (especially girls)  Diminution des abandons scolaires surtout pour les filles  Improvements in student literacy  Amélioration du niveau de lecture des élèves  Increased attentiveness  Augmentation du niveau de concentration/ d’attention des élèves  Increased handwashing  Augmentation du lavage des mains  Decreased student hunger  Diminution de faim des élèves 58. (If not answered above) What aspects or activities of the project had the most impact on the results you just mentioned? What aspects had the least impact? Si non mentionné en haut, quels aspects ou activités du projet ont eu le plus d’impact sur les résultats que vous avez mentionné précédemment ? Quels aspects ou activités ont eu le moins d’impact ? IMPAQ International, LLC Page G38 McGovern-Dole III Endline Evaluation Report Community Attitudes Attitudes de la communauté 59. What barriers to education are there in your community? How did the project address these barriers? How did project activities raise awareness among the community about the importance of education and promoting student attendance at school? Quelles sont les barrières à l’éducation qui existent au sein de votre communauté ? Comment le projet s’est-t-il adressé à ces barrières ? Comment le projet a-t-il sensibilisé la communauté sur l’importance de l’éducation et de la fréquentation scolaire ? 60. How concerned are parents and caregivers about their children’s performance at school? Did the project change their attitudes? If yes, in what ways? What factors could encourage parents to be more involved with their children’s education, particularly illiterate parents? Les parents d’élèves et les tuteurs se préoccupent-ils du rendement/de la performance scolaire de leurs enfants ? Le projet a-t-il changé leurs attitudes ? Si oui de quelle manière ? Quels facteurs pourraient encourager les parents (surtout les parents analphabètes) à s’impliquer davantage dans l’éducation de leurs enfants ? Sustainability Durabilité 61. From your perspective, which activities and processes in your school will be sustainable beyond project funding and which will not be sustainable? Please explain. Selon vous quelles activités et procédures seront à même d’être poursuivis au-delà de la fin du projet dans une optique de pérennisation/durabilité des activités et acquis du projet ? Quelles sont celles qui ne seront pas viables/durables au-delà de la fin du projet ? Pourquoi ou Pourquoi pas 62. With respect to sustainability, how has your school prepared for the absence of support from USDA and CRS? Who has been leading these efforts? Comment votre école s’est-elle préparée à prendre le relais des activités et à capitaliser les acquis du projet une fois à terme (retrait de USDA et CRS) de manière à assurer la durabilité ? Qui initie ou pilote de tels efforts au niveau de votre école ? COVID-19 If not answered earlier: Si non mentionné précédemment IMPAQ International, LLC Page G39 McGovern-Dole III Endline Evaluation Report 63. Which project activities were most impacted by COVID-19 restrictions? What measures, if any, were adopted to mitigate that? Quelles sont les activités qui ont été les plus impacté par les restrictions liées au COVID 19 ? Quelles sont les éventuelles mesures d’atténuation prises pour y faire face ? 64. Do you think that in the future, COVID-19 restrictions will affect barriers to education ? Pensez-vous que dans l’avenir les restrictions liées au Covid fortifieront les barrières à l’éducation ? 65. Has COVID-19 changed the activities planned for sustainability? Le Covid a-t-il changé ou influencé les activités prévues pour assurer la durabilité du projet ? Relevance, Good Practices, and Lessons Learned Pertinence, Bonnes pratiques et leçons apprises 66. Overall, how well do you think that the FFE 3 project responded to the needs and priorities of this community? Dans l’ensemble, pensez-vous que le projet FFE3 a-t-il bien répondu aux besoins et priorités de cette communauté ? 67. Overall, how satisfied are you with participating in the FFE 3 project? Dans l’ensemble quelle est votre niveau de satisfaction en tant que participant aux activités du projet FFE 3 ? 68. What were some innovations and good practices? Please share any lessons learned for potential future project phases. Quelles ont été les innovations et les bonnes pratiques ? Pourriez-vous partager quelques leçons apprises à capitaliser pour le futur ? Conclusion 69. Is there anything that I did not ask about that you would like to share with me? Auriez-vous autre chose à partager avec moi avant la fin de cet entretien ? Thank you for your time and comments. Merci pour votre disponibilité et votre contribution IMPAQ International, LLC Page G40 McGovern-Dole III Endline Evaluation Report CGS - Comités de Gestion Scolaire Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. L’interview prendra environ 60 minutes et se déroulera très bien si vous êtes disposé à parler. Sentez￾vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Auriez-vous des questions avant qu’on ne commence ? Background 70. Can you describe the main responsibilities of the School Management Committee (SMC) particularly related to the FFE 3 project? What is your role in the SMC? Organisation: Titre/responsabilité: Sexe: IMPAQ International, LLC Page G41 McGovern-Dole III Endline Evaluation Report Pourriez-vous décrire les principales responsabilités du CGS (Comités de Gestion Scolaire) en particulier celles en lien avec le projet FFE3 ? Quel est votre responsabilité au sein du CGS ? Training and SMC activités Formation et activités CGS Now I would like to learn more about the training you received through the FFE 3 project. A présent, je souhaite en savoir davantage à propos de la formation que vous avez-reçu dans le cadre du projet FFE3 71. What types of training did you receive? Which training sessions were most useful? Which were least useful? Please share examples. (Probe for training on developing school action plans, training on food storage and preparation, etc.) Quel(s) type(s) de formation avez-vous reçu ? Quelles sessions ou modules vous ont particulièrement apparu plus utiles que les autres ? Quelles sessions ou modules ont été moins utiles ? S’il vous plait donnez quelques exemples. (Tenter d’explorer notamment sur la formation sur l’élaboration de plans d’actions, formation sur le stockage des vivres et la préparation des repas) 72. After receiving training, what were you able to do with the SMC? Après avoir reçu les formations mentionnées précédemment, qu’avez-vous été capable de faire avec le CGS 73. What successful activities took place with the SMC over the past several years? What made them successful? Were there any successful outcomes that were surprising to you? Quelles sont les activités du CGS qui ont remporté le plus de succès au cours de ces dernières années ? Quels sont les facteurs qui ont contribué à ce succès ? Ces activités ont-elles engendré des résultats positifs inattendus ? 74. Were there any activities with the SMC in the last several years which were not successful? What made them less successful? Were there any negative outcomes that were surprising to you? Y’a-t-il eu des activités du CGS moins réussies par rapport à celles évoquées précédemment au cours des dernières années ? Quels sont les facteurs qui ont joué contre ce succès ? Ces activités ont-elles eu des résultats négatifs surprenants ? 75. Does your SMC collaborate with the Mayor and with any SILCs within your community to manage school related activities for FFE3? Can you describe your different roles? How do you work together to achieve the objectives related to school management? IMPAQ International, LLC Page G42 McGovern-Dole III Endline Evaluation Report Votre SMC collabore-t-il avec le maire et avec les SILC de votre communauté pour gérer les activités liées à l'école pour FFE3? Pouvez-vous décrire vos différents rôles? Comment travaillez-vous ensemble pour atteindre les objectifs liés à la gestion scolaire? 76. What were the objectives of this engagement? Were they achieved? Why or why not? (probe on lessons learned, success stories, and challenges) Quels étaient les objectifs de cet engagement? Ont-ils été atteints? Pourquoi ou pourquoi pas? (enquête sur les leçons apprises, les réussites et les défis) Student and Teacher Outcomes Elèves et Enseignants Now I would like to ask about impacts of the FFE 3 project on students and teachers. A présent, je souhaiterais que l’on parle des impacts du projet FFE3 sur les élèves et les enseignants 77. How has FFE 3 had an influence on students in your school? What are the key achievements, if any, resulting from the program? Are there any differences in the way it is affecting boys and girls? Probe for: Comment le projet FFE 3 a-t-il eu une influence sur les élèves de votre école ? Le programme affecterait￾t-il différemment les filles et les garçons ? Quels sont les principaux acquis/résultats engrangés dans le cadre du projet par rapport à :  Taux de scolarisation  Niveau de fréquentation scolaire  Taux d’abandon des élèves (particulièrement des filles)  Niveau de lecture des élèves  Niveau de concentration/d’attention  Diminution de faim des élèves  Increased student enrollment  Increased attendance  Decreased drop-outs (especially girls)  Improvements in student literacy  Increased attentiveness  Decreased student hunger 78. (If not answered above) What aspects or activities of the project had the most impact on the results you just mentioned? What aspects had the least impact? IMPAQ International, LLC Page G43 McGovern-Dole III Endline Evaluation Report (Si non mentionné dans les réponses précédentes) ; quels aspects des activités du projet ont eu le plus d’impact sur les résultats précédemment mentionnés ? Quels aspects ont eu le moins d’impact ? 79. Since receiving training, what changes have you observed in teachers? (Probe on teacher attendance and motivation) Depuis la mise en œuvre du projet, avez-vous constaté des changement de comportement chez vos enseignants ? (Creuser la question pour apprécier l’évolution du niveau de présence et de motivation des enseignants) Community Attitudes Attitudes des communautés Now I would like to ask about attitudes towards education and awareness in your community. 80. In your opinion, what are the barriers to children’s education in your community? How did the project address these barriers? How did project activities raise awareness among the community about the importance of education and promoting student attendance at school? Quelles sont les barrières à l’éducation qui existent au sein de votre communauté ? Comment le projet s’est-t-il adressé à ces barrières ? Comment le projet a-t-il sensibilisé la communauté sur l’importance de l’éducation et de la fréquentation scolaire ? 81. How concerned are parents and caregivers about their children’s performance at school? In what ways, if any, has the project changed parent and caregiver attitudes? How could parents and caregivers be encouraged to be more involved, including illiterate parents? Les parents d’élèves et les tuteurs se préoccupent-ils du rendement/de la performance de leurs enfants à l’école ? Le projet a-t-il changé leurs attitudes ? Si oui de quelle manière ? Quels facteurs pourraient encourager les parents (surtout les parents analphabètes) à s’impliquer davantage dans l’éducation de leurs enfants ? Sustainability Durabilité 82. From your perspective, which activities and processes will be sustainable beyond project funding? Which will not be sustainable? IMPAQ International, LLC Page G44 McGovern-Dole III Endline Evaluation Report Selon vous , quelles activités et procédures seront à même d’être poursuivis au-delà de la fin du projet dans une optique de pérennnisation? Quelles sont celles qui ne seront pas viables au-delà de la fin du projet ? Pourquoi ou Pourquoi pas? 83. What role, if any, does your SMC have in continuing FFE 3 project activities and sustaining outcomes achieved? Votre CGS aurait-t-il un rôle spécifique à jouer dans l’appropriation et la poursuite des activités du projet en vue d’en garantir la pérennité ? COVID-19 Now I would like you to think about to the impact of COVID-19 on SMC activities as part of the FFE 3 project. 84. Has COVID-19 changed the activities planned for sustainability? Le Covid a-t-il changé les activités et les efforts planifiés en vue d’assurer la durabilité/pérennité des activités du projet? Good Practices and Lessons Learned Bonnes pratiques et leçons apprises 85. Overall, have you had a positive or negative experience participating in your SMC through the FFE 3 project? Please explain. Dans l’ensemble, comment appréciez-vous votre expérience (positive/négative) avec votre CGS dans le cadre du projet FFE 3 ? Donnez en les raisons. 86. In your opinion, was there anything about your SMC or overall FFE 3 project that could have been strengthened or done differently, to be a better fit for the needs of your community? Etes vous satisfait de la manière dont votre CGS ou le projet FFE 3 dans l’ensemble ont opéré ? Qu’aurait-t-il fallu faire de différent (le cas échéant) en vue de renforcer l’impact de votre CGS ou un meilleur arrimage de votre CGS avec les besoins des communautés ? Conclusion 87. Is there anything that I did not ask about that you would like to share with me? IMPAQ International, LLC Page G45 McGovern-Dole III Endline Evaluation Report Auriez-vous autre chose à partager avec moi avant la fin de cet entretien ? Thank you for your time and comments. Merci pour votre disponibilité et votre contribution IMPAQ International, LLC Page G46 McGovern-Dole III Endline Evaluation Report Communauté interne d’épargne et de crédit (SILC) Note pour l’interviewer: En général nous souhaitons des réponses dans un contexte normal ne tenant pas compte de la situation sanitaire liée au COVID. Des questions spécifiques liées à l’impact du COVID sont posées en fait d’interview. Tout au long de l’interview, demandez au répondant de fournir des exemples illustratifs surtout lorsque les réponses sont courtes et fermées (Oui ; Non ; Bon ; Très bon) Introduction Bonjour/Bonsoir Mr/Mme. Mon nom est [insérer le nom]. Je travaille pour IMPAQ, une compagnie de recherche américaine. Le but de cet entretien est de recueillir votre expérience avec le programme Vivres pour l’Education 3 au Mali. Avec votre permission, je vais enregistrer notre conversation afin de compléter mes notes. Je voudrais vous rassurer de la confidentialité et de l’anonymat de cet entretien. Personne en dehors de l’équipe de recherche n’aura accès à ces informations. Les données collectées ne seront pas exploitées de manière à identifier individuellement les répondants mais seront exploitées de manière agrégée dans le seul but d’analyser et de comprendre les expériences vécues avec le programme. L’interview prendra environ 60 minutes et se déroulera très bien si vous êtes disposé à parler. Sentez￾vous à l’aise de parler largement et librement de vos expériences et opinions sur le projet. Votre participation est volontaire. Vous êtes libre d’arrêter votre participation à l’entretien à tout moment si vous ne souhaitez plus continuez et ceci sans aucun préjudice. Les informations collectées durant ces entretiens seront exploitées de manière agrégée en mettant en relief des points clé en fonction des zones d’étude. Elles ne seront pas exploitées individuellement. Vous ne serez pas identifié nommément dans l’analyse des données. Auriez-vous des questions avant qu’on ne commence ? SILC Participation Organisation: Titre/Responsabilité: Gender: IMPAQ International, LLC Page G47 McGovern-Dole III Endline Evaluation Report Participation au SILC 88. How long have you been part of the SILC as part of the FFE 3 project? Why did you decide to join this SILC? How did you find out about it? What were you told that this group would do for you, and for your community? Depuis combien de temps êtes-vous membres d’un groupe SILC ? Pourquoi avez-vous décidé de devenir membre un groupe SILC ? Comment avez-vous découvert ou entendu parler d’un groupe SILC pour la première fois ? Que vous a-t-on dit à propos de ce que les groupes SILC peuvent faire pour vous et votre communauté ? 89. Let’s talk about how your SILC operates.  What kind of support or training does your group receive? Who provided it, and when or how often? What did you like about these trainings? What have you learned? Are there any areas where you need more support?  Votre groupe reçoit-t-il des appuis ou formations quelconques ? Si oui de quelles natures sont-ils ? Quand/ Combien de fois est-ce que ces appuis et/ou formations ont eu lieu et de qui les avez-vous reçus ? Qu’avez-vous aimé et qu’avez-vous appris de ces formations? Y’a-t-il des aspects sur lesquels vous auriez besoin de plus de soutien/d’appui ? 90. Did your SILC provide activities to support schools in your community? If yes, please describe which ones. Y’a-t-il des activités entreprises par votre SILC en vue d’appuyer les écoles de votre communauté ? Si oui lesquelles ? 91. Was the implementation of some activities more successful than others? If so, which ones? Why? (Probe: were there any positive or negative outcomes that were surprising to you?) Parmi ces activités y’en a-t-il qui ont réussi mieux que d’autres ? Si oui, lesquelles ? Pourquoi ? (Ces activités ont-elles eu des résultats positifs ou négatifs surprenants ou inattendus? ) 92. Has your SILC collaborated with the School Management Committees (SMC) and Mayor within your community to manage school related activities for FFE3? Can you describe your different roles? How do you work together to achieve the objectives related to school management? Votre SILC a-t-il collaboré avec les comités de gestion scolaire (SMC) et le maire de votre communauté pour gérer les activités liées à l'école pour FFE3? Pouvez-vous décrire vos différents rôles? Comment travaillez-vous ensemble pour atteindre les objectifs liés à la gestion scolaire? 93. From your perspective, has participating in a SILC had an effect on members: IMPAQ International, LLC Page G48 McGovern-Dole III Endline Evaluation Report  D’après vous, est-ce que le fait d’être membre d’un groupes SILC a eu un effet sur : Les moyens de subsistance (création d’un commerce/une activité génératrice de revenus, amélioration de vos conditions de travail, épargne) des membres a. Livelihoods (for example, starting a business, improving existing work, saving money) b. Ability to send children to school?  Scolarisation des enfants des membres des groupes SILC c. Did SILCs have different effects on women and men as members ? (Probe on differences in spending, livelihoods, education, etc.) d. Pensez-vous que les groups SILC ont des effets différents selon que le membre est un homme ou une femme (chercher à savoir s’il y’a des différences par rapport aux postes de dépenses ménagers, les moyens de subsistance, l’éducation ect.)?  What about other families who are not SILC members?  Qu’en est-t-il des autres ménages dans le village qui ne sont pas membres de groupes SILC ? 94. Generally speaking, are parents in your community involved in their children’s education? What do you think causes some parents to be closely involved in their children’s education? Has the FFE 3 project encouraged parents to get more involved? What should be done to encourage to be more involved, including illiterate parents?  De manière Générale, les parents dans votre communauté sont-t-ils impliqués dans l’éducation de leurs enfants ? Selon vous qu’est-ce qui pousse certains parents à s’impliquer étroitement dans l’éducation de leurs enfants ? Est-ce que le projet FFE3 a entrepris des actions en ce sens (inciter les parents à s’impliquer davantage)? Qu’est-ce qui devrait être fait pour pousser les parents qui ne sont pas impliqués dans l’éducation de leurs enfants à l’être davantage, en particulier les parents analphabètes ? Sustainability 95. Will you or your household continue to participate in SILC group activities after the FFE 3 project ends? Please explain. Est-ce que vous ou votre ménage, allez continuer à participer aux activités du groupe SILC après la fin du projet FFE3 ? s’il vous plait, donnez-en les raisons. COVID 19 Now we would like for you to consider the impact of COVID-19. IMPAQ International, LLC Page G49 McGovern-Dole III Endline Evaluation Report A présent, nous souhaitons parler de l’impact du Covid 19 96. With COVID-19 restrictions, were there changes in SILC group activities? Le Covid 19 a-t-il affecté d’une quelconque manière certaines des activités des groupes SILC ? Si oui, lesquelles ? 97. Has COVID-19 or its restrictions changed whether your SILC group activities will continue after the FFE 3 project ends? Please explain. Pensez-vous que le covid19 y compris ses restrictions affecteront la viabilité ou pérennité des activités des groupes SILC une fois le projet FFE 3 à terme ? S’il vous plait expliquez Relevance and Lessons Learned Pertinence et leçons apprises 98. Overall, have you had a positive or negative experience participating in a SILC through the FFE 3 project? Please explain. Has the SILC met your expectations? Why or why not? Dans l’ensemble, comment appréciez-vous votre expérience (positive/négative) avec le groupe SILC dans le cadre du projet FFE 3 ? Donnez en les raisons. Le groupe SILC a-t-il répondu à vos attentes ? Pourquoi ? Pourquoi pas ? 99. In your opinion, was there anything about the SILC or overall FFE 3 project that could have been strengthened or done differently, to be a better fit for the needs of your community? Etes vous satisfait de la manière dont les groupe SILC ou le projet FFE 3 dans l’ensemble ont opéré ? Qu’aurait-t-il fallu faire de différent (le cas échéant) en vue de renforcer l’impact des SILC ou un meilleur arrimage des SILC avec les besoins des communautés ? Conclusion 100. Is there anything that I did not ask about that you would like to share with me? Y’a t-il quelque chose d’autre dont nous n’avons pas parlé et que vous souhaiteriez partager ? Thank you for your time and comments. Merci pour votre temps et vos différentes contributions IMPAQ International, LLC Page H1 McGovern-Dole III Endline Evaluation Report Appendix H: Terms of Reference TERMS OF REFERENCE Baseline, Midterm, and Final Evaluation for CRS Mali’s McGovern-Dole (MGD) International Food for Education (FFE) and Child Nutrition Program Purpose The purpose of this Terms of Reference is to describe the tasks and responsibilities of the external consultant to conduct the baseline, midterm and final evaluations of the McGovern-Dole (MGD) International Food for Education (FFE) and Child Nutrition Program, implemented by Catholic Relief Services (CRS) in Mali. Background Catholic Relief Services (CRS) Food-for-Education (FFE) Project seeks funding from the United States Department of Agriculture (USDA) through the McGovern-Dole (MGD) International FFE and Child Nutrition Program. This third phase will be a five-year intervention (FY2016 – FY2020) with a budget of $29,899,945 million, including 8,840 MT of commodities (fortified milled rice, green split peas, lentils and vegetable oil). This third phase aims to improve literacy of school-aged children for 77,104 children in 264 primary schools in Mopti and Koulikoro Regions in Mali. CRS will directly implement parts of the project’s activities, with sub-agreements with the following implementing partners: Amprode, Caritas Bamako, Caritas Mopti, Education Development Center, Inc. (EDC) and Guamina. CRS will work in collaboration with the Ministry of National Education (MONE), Regional Education Offices (Academie), School District Offices (CAP) and School Management Committees (SMC) to build local capacity and promote sustainability for school feeding and literacy activities. Local, regional and national education officials will directly support monitoring and evaluation activities. FFE III Project Description CRS has implemented FFE school feedings projects in Mali in collaboration with local partners and the MONE since 1999. The previous Mali MGD FFE project (2011-2015) has reached 76,411 primary and secondary students in 310 schools and has achieved positive results in enrollment and attendance rates. Project activities include school meals, take home rations (THR) and Vitamin A and deworming medications distribution, School Managements Committees (SMC) members capacity building, formation of Savings and Internal Lending Community (SILC) groups and introducing illustrated report cards. In the new phase, the project will build on these successes to improve literacy of school-aged children (SO1). CRS theorizes that by improving student attendance of high-quality literacy instruction in an environment in which students are enabled to participate actively in class, higher literacy rates of school aged children will be achieved. To improve attendance and attentiveness in 264 primary schools identified in the previous phases, this FFE project will support community specific enrollment campaigns based on community-led barrier analysis results and continue school feedings to reduce short-term hunger and IMPAQ International, LLC Page H2 McGovern-Dole III Endline Evaluation Report improve student attentiveness. Vitamin A and de-worming medications will be distributed to enrolled students in Y1, 2 and 3, with a transition to local responsibility in Y3. THR will be distributed to all 5th and 6th grade students in USDA-supported primary schools with a minimum 90% attendance to promote attendance and retention. CRS will also build sustained capacity in school feeding and early grade literacy instruction at the local, regional, and national levels. For school feeding, this includes tailored training to ‘graduate’ SMCs in school management and matching grants to fund school action plans, incentivize graduation, and engender ownership by the decentralized structures that must provide a match. CRS will also provide technical support to the National School Canteen Center (NSCC) in response to weaknesses identified in their biennial evaluation of school feeding capacity. For early grade literacy instruction, this includes creating a network of government officials, school administrators, and teachers trained in the Balanced Literacy Approach, providing classroom kits, and monitoring student progress with the Early Grade Reading Assessment (EGRA). FFE III Evaluation Plan In compliance with the terms of CRS’ agreement with USDA, CRS has developed an evaluation plan that details the purpose, scope, methods and approximate timeline for all of FFE III’s evaluation activities. The evaluation plan a) explains how the project will be evaluated including the gathering of baseline data and special studies that will be conducted; b) gives information on the project’s midterm and final evaluations; and c) describes how the evaluation activities will be managed. It also includes the project’s theory of change, evaluation methods and the timing of their activities, plans for the sharing/dissemination of evaluation findings, the project’s target beneficiaries, the project results indicators and their targets. The evaluation plan will guide all actions taken by CRS and the external consultant/firm in regard to the project’s baseline, midterm and final evaluations. This ToR is based on the FFE III project’s evaluation plan and is subject to changes requested by USDA. CRS and the external consultant/firm may also suggest changes to the evaluation plan for the approval of USDA, which would then allow for changes to be made to this ToR. FFE III Evaluation Design The research questions will assess the extent to which FFE III has achieved the program objectives. The questions will focus on literacy and use of Health and dietary practices and will target various beneficiaries, including students, teachers, school administrators and officials, parents, Savings and Internal Lending Committee (SILC) groups and School Management Committee (SMC) members, with an emphasis on gender. The evaluation of FFE III will include a performance evaluation and an impact evaluation. The performance evaluation will measure and compare the outcomes of the program. Student outcomes include literacy, attentiveness, attendance, enrollment, continuation and health and dietary practices. Teacher outcomes measure knowledge of teaching practices and attendance. Outcomes related to parents, SMC members and other community stakeholders will focus on school governance and canteen management. The performance evaluation will evaluate the FFE III program using to school data provided by CRS, along with survey and qualitative data collected by the external consultant/firm. IMPAQ International, LLC Page H3 McGovern-Dole III Endline Evaluation Report The impact evaluation will measure the effects of the balanced literacy approach on students learning. The outcomes of interest for the impact evaluation are reading and comprehension scores of students in schools. a. Impact evaluation: For the impact evaluation a sample of students will be tested with the ASER-Reading Assessment and the same sample will also receive a short questionnaire about other relevant outcomes. The evaluation will take place in a randomized sample of the 264 schools targeted to receive project package including the Balanced Literacy Approach (BLA). b. Performance evaluation: At baseline, the external consultant/firm will collect survey data to set initial values for project indicators. Through the midline and endline, the external consultant/firm will collect survey data pertaining to the effect of the program on stakeholders including, students, teachers, school administrators and officials, SILC group members and SMC members. c. School attendance data:  CRS will provide the external consultant/firm with data collected by the schools about students’ attendance. Ideally those data will be organized and formatted in a way that it can be easily used for analysis. If students can be tracked across years, these data will not only be very useful on their own but can also serve as census data for impact evaluation. The external consultant/firm will advise CRS on the collection of these data and provide analysis. These data will be collected up until the moment of the evaluation and its analysis will be incorporated in all three reports at baseline, midline and endline.  d. The external consultant/firm will collect qualitative data through in-depth interviews with stakeholders including, students, teachers, administrators, officials, SILC group members and SMC members to understand what components of the program are more and less effective and why. Potential questions may include: How have your study techniques changed? Have you noticed a change in your classroom performance? If yes, what are the main reasons behind this change? Have you noticed a difference in your parents support for your education following receipt of periodic report cards? Why do you think this has happened? FFE III Evaluation Methodology The external consultant/firm will produce testing, survey and qualitative evaluation instruments, which will be submitted to CRS along with a 2-3-page summary Evaluation Design memo, which will may further refine the sampling strategy for the impact evaluation and the performance evaluation. The sample size and methodology will follow the approved FFE III evaluation plan and can be enhanced to be more rigorous should the external consultant/firm propose such changes. Significant changes to the evaluation plan (for example, decreases in sample size and/or a less rigorous methodology) will require advance approval of CRS and USDA. IMPAQ International, LLC Page H4 McGovern-Dole III Endline Evaluation Report Purpose and Scope The purpose of the evaluation is to assess USDA’s SO1: Improved literacy of school-age children overall and SO2: Increased use of health and dietary practices in program schools. For SO1, the impact evaluation will measure differences between control and treatment schools (final methodology will be finalized with consultant/firm) disaggregated by sex for:  Reading ability of students  Level of student attendance  Level of student attentiveness  Level of student participation in classroom activities  Level of teacher motivation  Level of parent/community engagement For SO2, the impact evaluation will measure differences across time on:  Knowledge of safe food prep and storage practices  Reported access to preventative health interventions and impact on the number of school days missed by students due to health issues Key Evaluation Questions The final evaluation will follow the same scheme of questions as the midterm evaluation and focus on assessing changes from the baseline levels of the project indicators. However, based on the findings of the midterm evaluation, the external consultant/firm and CRS will validate or modify them to ensure the most relevant questions are evaluated. Questions that may be interrogated are: Relevance: Is defined by the extent to which the project activities are suited to the priorities of the target group recipient and donor. Key questions to address are: - To what extent has the FFE project aligned with local, regional, and national policies, interventions, and initiatives in education and health? - To what extent were the objectives of the program valid? - Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? - Are the activities and outputs of the program consistent with the intended impacts and effects? Effectiveness: Is a measure of the extent to which project activities attain its objectives. Key questions are: - To what extent were the objectives of FFE achieved / are likely to be achieved? - What were the major factors influencing the achievement or non-achievement of the objectives? IMPAQ International, LLC Page H5 McGovern-Dole III Endline Evaluation Report - Please assess the utility of the monitoring and evaluation system and processes? Are there changes to the M&E system and processes that need to be taken in order to improve the utility, credibility and reliability of the data and information collected? Efficiency: Measures both qualitative and quantitative outputs in relation to inputs. It assesses the extent to which project uses costly resources possible in order to achieve the desired results. Key questions to address are: - Were objectives achieved on time? - Where resources managed in compliance to USG and USDA policies? Impacts: are positive and negative changes produced by the intervention directly and indirectly intended or unintended. This involves changes impacts and effects on the local social economic, environmental and other development indicators. Key questions are: - Have children in the FFE intervention schools improved their literacy during the course of the program? - To what extent have teachers improved their skills and knowledge to instruct literacy? - To what extent has student attendance in the FFE intervention schools improved during the course of the program? - To what extent has there been an increase in the use of standard hygiene and health practices among students in the FFE intervention schools during the course of the program? - To what extent has there been an increase in dietary diversity among students in the FFE intervention schools during the course of the program? - To what extent has there been an increase in access to preventative health interventions for students in the FFE intervention schools during the course of the program? - To what extent has there been an increase in access to food preparation and storage tools and equipment in FFE intervention schools during the course of the program? - To what extent have government officials increased their skills and knowledge in FFE intervention departments during the course of the program? - To what extent has there been an increase in the involvement of parents in FFE intervention schools during the course of the program? - Have there been any unintended negative effects of the project? If so, why? Sustainability: Is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Key questions are:  - What steps has the project taken to address the sustainability of the project activities? What additional steps need to be taken in order to improve the chances for sustainability of the activities and benefits derived from the project activities? IMPAQ International, LLC Page H6 McGovern-Dole III Endline Evaluation Report - How has local, regional and national capacity changed regarding literacy instruction in treatment schools? School feeding programs? Student enrollment and attendance monitoring? Is there evidence that their capacity and ability to provide quality programming has improved? - How have the national capacities, policies, procedures and priorities changed? FFE III Baseline Evaluation Purpose and Scope: The baseline study objectives are to establish baseline values for outcome indicators (see Annex 1 and 2), refine targets for performance indicators, generate data to be used for comparative analysis and validate project strategies and assumptions. The baseline study is expected to take 45 days allowing for survey tool design, enumerator training, data collection and report elaboration. The primary question of study is the level of literacy in students when receiving full treatment. Results from ASER￾Reading conducted at the close of the previous FFE project (a.k.a FFE II) as well as general information about teacher and schools administrators capacities as well as student participation collected during Year 1 initial ASER assessment will also provide a better understanding of student performance, teacher and school administrators’ capacity, community interests in education, learning environments (including student perceptions) and organizational capacity to support initial activity implementation. In addition to establishing indicator base values, the baseline study will ask initial questions around household demographic and socio-economic conditions, including household size, food security and coping mechanisms, education levels and asset and income sources. This information will aim to validate, or correct initial critical assumptions as outlined in the program framework. Baseline Methodology Quantitative method CRS proposes a cohort comparison method to evaluate the effects of the BLA on student literacy growth. The cohort comparison method will measure change over time of beneficiaries at a later point in time relative to the initial state of non-beneficiaries before the program started. Thus, for baseline, a sample of grade 2 -4 students will undergo ASER testing as comparison groups. Cohorts of grade 1-3 students whose teachers will be trained on BLA the life of the project on the balanced literacy approach project will serve as treatment group. ASER testing of the treatment groups will help to determine the two-year and three-year effects of the BLA at midline and reevaluated at endline. Exhibit 1 provides a graphical representation of the cohort comparison method. For example, Grade 2 students from Treatment Cohort 2 at midline will have been exposed to two years of teachers with BLA training (Year 2 and 3 – 2016-18). Comparing these students with Grade 2 students from baseline (comparison 3 group) will help to estimate the two-year program effect of having exposure to a BLA trained teacher on literacy growth. Similarly, Grade 3 students from Treatment Cohort 1 at midline will have been exposed to three years of teachers with BLA training (2015-2018). By comparing these students with Grade 3 students from baseline (comparison 1 group), CRS can estimate the three-year program effect of having exposure to a BLA trained teacher on literacy growth. CRS will reevaluate these program effects at endline by comparing Grade 3 students from Treatment Cohort 3 at endline with Grade 3 students from baseline (Comparison 2 group) we can estimate the three-year program effect of having exposure to a BLA trained teacher on literacy growth. IMPAQ International, LLC Page H7 McGovern-Dole III Endline Evaluation Report Lastly, the cohort comparison design will allow CRS Mali to determine spillover effects of the BLA intervention on students within BLA schools. For example, Grade 4 students at midline (comparison 3 group) will have not been taught by a BLA-trained teacher but may have benefited from the BLA intervention through spillover effects. By comparing this group with the Grade 4 students in Comparison 1 from baseline, the three-year spill-over effect of being in a BLA school on literacy progress can be determined. Exhibit 79: Cohort Comparison Approach to Program Evaluation Year 1 Year 2 Year 3 Year 4 Year 5 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020 Comparison 1 4 th grade 5 th grade 6 th grade Comparison 2 3 rd grade 4 th grade 5 th grade 6 th grade Comparison 3 2 nd grade 3 rd grade 4 th grade 5 th grade 6 th grade Treatment Cohort 1 1 st grade 2 nd grade 3 rd grade 4 th grade 5 th grade Treatment Cohort 2 1 st grade 2 nd grade 3 rd grade 4 th grade Treatment Cohort 3 1st grade 2 nd grade 3 rd grade Treatment Cohort 4 1 st grade 2 nd grade 2 year Program Effect – Impact Evaluation Data Collection 3 year Program Effect – Impact Evaluation Data Collection 3 year Spill-Over Effect – Impact Evaluation Data Collection Data Collection for Performance Evaluation IMPAQ International, LLC Page H8 McGovern-Dole III Endline Evaluation Report Qualitative Method: In addition, the evaluation will integrate a complimentary qualitative component to address some of the limitations of the quantitative methods and provide contextual understanding and interpretation of the quantitative results. Specifically, qualitative data will focus on current program realities and perceptions in order to help CRS design appropriate strategies and activities and address implementation challenges. The qualitative component will consist of key informant interviews. CRS will draw a purposive sample in order to include program beneficiaries and stakeholders who represent a broad range of perspectives. Specifically, selected informants who are likely to give the most insightful information based on the informants’ level of engagement with the program (e.g., national, regional, and community) and on the informants’ type of engagement with the program (e.g. program partners, program stakeholders, program beneficiaries). CRS Mali will finalize the informant list and the interview schedule in consultation with the external consultant. Exhibit 2 outlines the target respondents and qualitative tools that will be used to collect the qualitative data. Exhibit 80: Illustrative Qualitative Method and Tools Qualitative Tool Application Illustrative Respondents Focus group discussion (FGD) Open discussions on education beliefs, school infrastructure/ learning environment, reading, gender issues and parent/community involvement. FGD will be facilitated by one moderator and one note taker and one translator (as needed). Teachers, Parents (beneficiary), Community members. **6-10 individuals: same age group and sex for each FGD Key informant interviews Key respondents will be interviewed using a semi-structured questionnaire to assess perceptions about education, community involvement, learning environment and barriers. School directors, and SMC president Baseline Sampling The optimal sample sizes for the evaluation are estimated using the following formula. Equation 1:𝑁𝑠 = 𝑁𝑝𝑝(1−𝑝) (𝑁𝑝−1)( 𝐵 𝐶 ) 2 +𝑝(1−𝑝) Where: 𝑁𝑠 is completed sample size needed 𝑁𝑝is size of the population 𝑝 is proportion expected to answer a certain way (most conservative is 0.5 or 50%) 𝐵 is acceptable level of sampling error (usual values are 0.05 = ±5% and 0.03 = ±3%) IMPAQ International, LLC Page H9 McGovern-Dole III Endline Evaluation Report 𝐶 is the Z-statistic associated with confidence level interval (1.960 = 95% confidence level). To compare changes in outcomes between baseline, midline, and endline, the baseline will randomly select 173 primary schools from a total of 264 BLA schools in Mali. Using Equation 1 on page 6, 173 primary schools will be calculated using the total population of 264 BLA treated schools, adding 10% extra schools in case of attrition in intervention. The schools will be selected using a proportional sampling approach to allow for a range of regional characteristics to be considered (such as proximity to major roads, etc.). We will sample by district and region. Thus, CRS proposes to sample only Grade 1, Grade 2, Grade 3 and Grade 4 students at baseline according to the comparison cohort method detailed in Exhibit 1. This will limit the sample to 1,684 student and household beneficiaries at baseline. The external consultant will be able to utilize this sample to determine the effects of the BLA. While we will not receive information about potential spillovers of the FFE and BLA intervention in other grades, the sample will allow for sufficient size to demonstrate student, school, and household characteristics at a lower data collection cost. Please note that at midline and endline, IMPAQ recommends to sample from Grade 2 to Grade 6 in order to conduct a performance evaluation of literacy growth, increasing the total required sample to 2,105 students. Exhibit 3 contains a comprehensive list of respondents, sampling strategy, and key information to be collected by the evaluator at each data collection stage. CRS used Equation 1 on page 6 to calculate the sufficient sample size, adding 10% extra respondents in case of attrition. In each school, the evaluator will randomly sample two or three students from each grade level to evaluate literacy growth using ASER, nutrition, hunger, and learning habits. CRS will also sample at least one girl and one boy from Grades 1 – 4 at baseline to conduct sub-sample analysis by gender. In total, there will be at least 564 girls and 564 boys in the sample of students at baseline. It will interview the same respondents at midline and endline. In order to gather information on relevant households, CRS will also sample the households of selected students to collect data on demographic characteristics, education perception, etc. It will collect data from 1,684 households in total at baseline. The same sample at baseline, midline, and endline will be interview. Since the BLA is being implemented in Grade 1 only at time of baseline (2015-2016), CRS will sample one teacher from Grade 1 from each school for a total of 173 teachers. CRS will collect data on the utilization of BLA teaching techniques. At midline (2018), CRS will collect additional teaching data from 173 Grade 2 and 173 Grade 3 teachers, as well as collect follow-up data on the same 173 Grade 1 teachers. At endline (2020), the same Grade 1, Grade 2, and Grade 3 teachers will be sample Other respondents include the school director. The interview will concern 173 school directors in total (one from each school). The school director’s survey will provide data on school characteristics such as student and teacher attendance and school infrastructure. CRS will sample the school directors at baseline, midline, and endline. Lastly, CRS will also sample 173 School Management Committees (SMC), one in each of the 173 schools. It will ensure that SMC board members are identified in the household’s sample. The data will consist of information on the roles and responsibilities, the SMC financial management, and the functioning of the school canteens. The same SMC board members will be interviewed at baseline, midline, and endline. Exhibit 3: Sampling Strategy IMPAQ International, LLC Page H10 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Household Demographic characteristics, education perception, school expenses, number of children attending school and hunger Baseline (2016) Midline (2018) Endline (2020) 1,263 households (421 students from Grades 1-3 only) School Directors School characteristics, including (but not limited to) student enrollment and attendance rates, teacher attendance, and school infrastructure Baseline (2016) Midline (2018) Endline (2020) 157 + 10% = 173 Teachers BLA teaching practices, including literacy approach, teacher capacity and techniques used, student feedback and encouragement Baseline (2016) 157 + 10% = 173 (Grade 1 teachers only) Midline (2018) Endline (2020) 157 + 10% = 173 for Grades 1, 2, and 3 each Students ASER or EGRA, student perception of learning environment, hunger, nutrition, learning habits Baseline (2016) 383 + 10% = 421 from each Grades 1-4 only Midline (2018) Endline (2020) 383 + 10% = 421 from each Grades 2-5 only School Management Committee – Board Members Roles and responsibilities, governance, financial SMC management, action planning, school and canteen management Baseline (2016) Midline (2018) Endline (2020) 157 + 10% = 173 (will be a sub-sample of mother’s) All samples are based on 95% confidence and 5% margin of error Preliminary results will be presented during a restitution to permit initial clarifications on findings and highlight areas of interest that will be valuable to understand better for project strategy and may provide additional context for the consultant. The collected data will be used to set and/or revise realistic targets for the indicators within the Indicator Performance Tracking Table (IPTT) and serve as a benchmark for comparison against midline and endline data to determine project impact. Baseline timeline The following activities of baseline IMPAQ International, LLC Page H11 McGovern-Dole III Endline Evaluation Report Activities Proposed Timeline Prepare the draft TOR November 2015 Share the draft TOR with USDA November 2015 Recruit/Confirm contracted evaluator December 2015–Mars 2016 Refine evaluation methodology and assign roles and responsibilities Late Mars 2016 Conduct data collection and analysis May –June 2016 Conduct stakeholder validation workshop June 2016 Submit final baseline evaluation report to USDA July2016 Audiences and Stakeholders The final report will be shared with all relevant stakeholders, including USDA, implementing partners, MONE representatives, local authorities and SMC, to ensure a common understanding of the expectations and how roles and responsibilities will contribute to project outcomes. USDA will receive within six months of agreement signing the final baseline report with the baseline data (including codebooks and data analysis plans) per USDA M&E Policy. FFE III Midterm Evaluation One of the primary objectives of the midterm evaluation is to provide recommendations for necessary mid-course correction or modifications. The external consultant/firm will conduct the mid-term evaluation, from March – June 2018 (with data collection in March-April 2018). This mid-term evaluation will assess the progress of the MGD program’s implementation of project activities and their results towards intended outcomes. The findings will enable the project team to assess progress in implementation, assess relevance of interventions, and adjust strategies, management structures and/or activities for improved project effectiveness, efficiency, impact and sustainability. The midterm evaluation will also document lessons learned and recommend changes to activities or implementation strategies as needed. CRS will ensure that all key project staff and key stakeholders participate in the review process in order to bring together a range of viewpoints to inform the process, the overall evaluation and recommendations. The midterm evaluation report will be shared widely to promote greater ownership of the project and sharing of lessons learned. Purpose and Scope: The midterm evaluation will assess service delivery as outlined in the detailed implementation plan and technical narrative and will measure progress against stated goals and objectives, including the effectiveness, efficiency and timeliness of the FFE III program interventions in achieving targets against baseline values. The midterm will review the results frameworks with all project documents and critical assumptions and consider the implementing environment, including enablers and constraints, with an aim of making necessary modifications or mid-course corrections to support the project to meet its stated goals and objectives and achieve sustainability. IMPAQ International, LLC Page H12 McGovern-Dole III Endline Evaluation Report Preliminary key evaluation questions: The evaluation will consider questions specifically related to intervention relevance, implementation effectiveness, project efficiency, project achievements and sustainability. The midterm evaluation will address (but is not limited to) the following questions: Relevance - Are the activities and outputs of the project consistent with the overall goals and the attainment of its objective? - Are the activities and outputs of the project consistent with the intended impacts and effects? - Does the program meet communities and government priorities? - Are stakeholders (management committee, parents, teachers, local authorities) satisfied with their participation in the program? Why or why not? - Does the project align with government policies and programs (local, national)? Does the project align, and compliment other donor, other NGO and/or local organizations managed programs? Effectiveness - To what extent were the objectives of the project and the yearly benchmark indicators achieved/ are likely to be achieved? - Were the implementation strategies relevant and effective enough to improve: 1) enrolment and attendance among pupils particularly girls? 2) Community participation and engagement? 3) A better learning environment? Are there more effective strategies that would have a greater impact? - What are the project’s major limitations? - Is the staffing structure and capacity sufficient and appropriate? Is the coordination mechanism effective? What if anything should be changed? - Has program implementation been effectively monitored? How well did the monitoring and evaluation mechanism in place help the implementation of the project? - What changes are required in the project to achieve project goals and objectives? Efficiency To what extent resources (funds, expertise, time, etc.) are converting to results economically and quality? - Were objectives achieved on time? - Was the project implemented in the most efficient way compared to alternatives? - Does the food supply chain (including transport and storage) minimize loss and damages? - Are activities cost-efficient? Are objectives achieving on time? Is the FFE implementing in the most efficient way compared to alternatives? (Efficiency and Value for Money) Impacts - What is the overall project outcome to date? To what extent project objectives and the yearly benchmark indicators have been achieved? What is facilitating or not the achievement of results and objectives in a timely manner? - What evidence suggests that the BLA has contributed to improved literacy? - Have there been changes in students’ attendance, particularly among girls? - Is student attentiveness improving? Why or why not? What more could be done? IMPAQ International, LLC Page H13 McGovern-Dole III Endline Evaluation Report - Is the incentive strategy effectively promoting student attendance? Are strategy modifications needed to improve attendance? Please explain. - How has the project affected girls and boys? Is there an observable difference? What? - Have community barriers to education been identified? If so, how are they being addressed? How could the project better support behavior and social change? - How are parents encouraged to be involved in their children’s education? How might they be encouraged to be more involved (including illiterate parents)? - How has teacher attendance and motivation changed? What more could be done? - How do teachers find instructional materials? How are they using them? What could be done to promote greater/more effective use? - How are community-based structures (e.g. schools, SMC, SILC) supporting project implementation? Are they on track to assume ownership of key activities beyond the life of the project? Are they satisfied with their participation? How might they be encouraged and/or supported to participate more? - How have capacity building activities for SMC improved their capacities? What obstacles persist? What more should be done to ensure they will have the capacity to manage the school canteens beyond the life of the project? - What innovations, lessons learned, and good practices can be documented so far? Sustainability - Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Key questions are: - What activities and/or outcomes (both expected and unexpected) of the program are likely to be sustained? What evidence is there to suggest this? - What is the level of ownership acquired by the stakeholders? And how do they use? How can they evolve and / or continue the benefits resulting from the action after the end of the intervention? - What are the major factors which can influence the achievement or non-achievement of the sustainability of the project? - How do the government’s capacities, policies, procedures, and priorities contribute to sustainability? - What strategies should be used to obtain long lasting support from communities and local/central administration that goes beyond the time of the project? Midterm Methodology: The methodology will focus on evaluating the progress to date and will replicate the baseline methodology (see above) as appropriate to establish midterm values for impact and outcome indicators that will enable the evaluator to assess actual against expected performance. All quantitative data will be collected using digital questionnaires, as in the baseline study. CRS will ensure midterm evaluation methodology is consistent with USDA M&E Policy. Quantitative Evaluation Design As designed and implemented in the baseline evaluation, IMPAQ is conducting a five-year, longitudinal quasi-experimental design evaluation using two types of outcome measurement: a pre–post comparison and a cohort comparison. Similar to the baseline, we will integrate a complementary qualitative analysis at midline and endline to help address some limitations of the quantitative analysis, as well as provide contextual understanding and interpretation of the quantitative results. In addition to the initial design, IMPAQ International, LLC Page H14 McGovern-Dole III Endline Evaluation Report we propose to incorporate performance analysis of both mothers’ and fathers’ perceptions on education, improvement in school attendance, and mechanics of SILC groups. This section describes the two quasi￾experimental methods and the performance analysis for the midline evaluation in greater detail. Pre–Post Comparison Method Similar to the baseline, we will use a pre–post comparison method to assess health and hygiene practices among project beneficiaries, including principals and teachers, school management committees (SMCs), students, and parents. We also will assess food security status among mothers/caregivers and minimum acceptable diets among students. We will use this methodology to assess and quantify the project’s impact by tracking changes in outcomes for the same project beneficiaries over time, using measures both before and after the project. We will also compare outcomes using data collected at baseline with the changes in the same outcomes (measured in the same manner) at midline. An important precursor for this evaluation methodology is the determination of the sample size. For the pre–post comparison method, power analysis was conducted prior to the baseline evaluation implementation to determine the number of beneficiaries needed to detect differences in health and hygiene practices over time.26 During the baseline evaluation, IMPAQ sampled students and caregivers from 50 schools in Mopti and Koulikoro. Our original sample of 2,160 students and caregivers was sufficient to detect differences in hand washing practices among children. However, the CRS Mali team informed us that three out of the 50 sample schools have been closed due to insecurity in the region, thereby reducing the sample to 47 schools. After redoing the power calculations using the lower number of schools, we find that our new sample of 1880 students and caregivers are also sufficient to detect differences in hand washing practices.27,28 Cohort Comparison Method As proposed in the evaluation design, we will use a cohort comparison method to evaluate the effects of the Balanced Literacy Approach (BLA) on student literacy growth at midline. This methodology measures improvement (change) over time of beneficiaries relative to their initial state before the project started. Earlier cohorts serve as a comparison group to later cohorts. We can use this method in accordance with Education Development Center’s (EDC) BLA implementation plan. In Year 1 (2015–2016), only grade 1 26 The baseline evaluation report contains details of the power analysis conducted. 27 In the baseline, we found that the baseline average of children using handwashing practices was 0.49. To detect a change of handwashing practices from 0.49 to 0.64, we needed 900 students sampled from 50 schools. When we redo the power calculations for 47 schools, we find that we need 1457 students. We surveyed 2160 students during baseline and are now planning to survey 1880 students during the midline, which are both higher than the number of students required as per the power calculations. 28 For the power calculations, we set standard values for the level and power of the test (α = 0.05 and β = 0.8) and assumed that ρ = 0.25. IMPAQ International, LLC Page H15 McGovern-Dole III Endline Evaluation Report teachers will receive BLA intervention training. In Year 2 (2016–2017), grade 1 teachers will become grade 2 teachers and receive additional training, and new grade 1 teachers will receive BLA training. In Year 3 (2017–2018), grade 2 teachers will become grade 3 teachers, grade 1 teachers will become grade 2 teachers, and all will receive retraining; and new grade 1 teachers will receive BLA training. To implement the comparison cohort method described below, we sampled grade 1, grade 2, grade 3, and grade 4 students at baseline and we will collect data from new cohorts in the same grades. Exhibit 2 provides a graphical representation of the cohort comparison method, which we explain in detail in the following subsections. Exhibit 4: Cohort Comparison Approach to Project Evaluation Strategy Baseline Midline Endline 2015–2016 2017–2018 2019–2020 Comparison 1 4 th grade Comparison 2 3 rd grade Comparison 3 2 nd grade 4 th grade Treatment Cohort 1 1 st grade 3 rd grade Treatment Cohort 2 2 nd grade 4 th grade Treatment Cohort 3 1 st grade 3 rd grade 2-Year Program Effect 3-Year Program Effect 4-Year Program Effect We will calculate two types of project effects on literacy levels: average treatment effect on the treated (ATE) and total average treatment effect (TATE).  ATE is equivalent to the change in literacy prevalence between treatment and comparison groups after controlling for any other effects that could be influencing our results simultaneously. To obtain unbiased ATE estimates, we need to take time effects into account. Specifically, we need to subtract any changes in illiteracy prevalence in primary school children that might have arisen because of changes over time in circumstances unrelated to the project.   TATE is a weighted average of the ATE and the indirect treatment effect on the untreated (ITE). The ITE measures the indirect effect of the project on cohorts that were not selected to be taught by BLA￾trained teachers, but that belonged to schools where these BLA-trained teachers taught (spillover effects). We will underestimate the treatment’s effectiveness if we do not consider the possibility that the BLA-trained teachers might also improve the literacy level of students belonging to untreated cohorts. The treatment’s effect on the treated will be underestimated, and its effect on the untreated will remain unmeasured, which may result in incorrect policy conclusions. We will also capture IMPAQ International, LLC Page H16 McGovern-Dole III Endline Evaluation Report spillover effects using qualitative methods and triangulate the indirect treatment effects through the qualitative information as well. We will specifically ask questions related to the possibility of spillover effects during our qualitative data collection from the project team and partners. If these effects are confirmed during the midline analysis, we will consider asking additional questions during endline to contextualize these findings and may consider adding teacher FGDs. This phased-implementation approach will allow us to determine the following:  1. Two-year project effects: We find, highlighted in gray in Exhibit 4, the observations that will be used to calculate the two-year project effects at midline. Grade 2 students from Treatment Cohort 2 at midline will have been exposed to two years of teachers with BLA training (2016–2018). By comparing these students with Grade 2 students in Comparison 3 from baseline, we can estimate the two-year project effect of having exposure to a BLA trained teacher on literacy growth (Exhibit 4 provides an example of the calculations for the two-year project effects). 2. Three-year project effects: The observations that will be used to calculate the 3-year project effects are highlighted in green in Exhibit 3. Grade 3 students from Treatment Cohort 1 at midline will have been exposed to three years of teachers with BLA training (2015–2018). By comparing these students with Grade 3 students in Comparison 2 from baseline, we can estimate the three￾year project effect of having exposure to a BLA-trained teacher on literacy growth. 3. Four-year project effects: The observations that will be used to calculate the four-year project effects (three years of currently trained teachers plus the effect of staying with a trained teacher for one more year) are highlighted in yellow in Exhibit 3. Grade 4 students from Treatment Cohort 2 at endline will have been exposed to four years of teachers with BLA training (2016–2019). By comparing these students with Grade 4 students in Comparison 1 from baseline, we can estimate the four-year project effect of having exposure to a BLA-trained teacher on literacy growth. 4. Time effects: To find time effects between baseline and midline, we will compare Grade 1 students from Treatment Cohort 1 at baseline with Grade 1 students in Treatment Cohort 3 from midline, both of which would have been exposed to one year of teachers with BLA training. The only difference between these two groups is the potential time effects. Similarly, if we compare Grade 3 students from Treatment Cohort 1 at midline with Grade 3 students in Treatment Cohort 3 from endline (both of which would have been exposed to three years of teachers with BLA training), we can calculate time trends between midline and endline. 5. Spillover effects: The cohort comparison design allows us to determine spillover effects of the BLA intervention on students within BLA schools. Some of the BLA-trained teachers end up teaching the comparison groups when the teachers assigned to those grades are absent from school. Taking that fact into account is important because teacher absenteeism has been documented as a serious concern in developing countries. For example, Grade 4 students in Comparison 3 at midline will not have been taught by a BLA-trained teacher but may have benefited from the BLA intervention through spillover effects. By comparing this group with the Grade 4 students in Comparison 1 from baseline, we can determine the three-year spillover effect on literacy progress of being in a BLA school. IMPAQ International, LLC Page H17 McGovern-Dole III Endline Evaluation Report Source: IMPAQ. Methodological Limitations. This is a quasi-experimental design that relies on the assumption that we are able to capture causal changes in literacy rates by measuring changes across cohorts. Our identification strategy rests on the assumption that there are no unobserved variables that affect both the probability of being part of the intervention group and the literacy rates of children. For example, particular educational policies enacted by the government at the same year of the intervention would potentially confound the cohort comparison approach. To safeguard from these threats and ensure the validity of our methodology, we have taken three different actions exploiting the structure of the program implementation and the data available:  The inclusion of time effects controls for all year-specific, individual-shared increases in literacy outcomes for all individuals. This addresses the identification threat regarding other educational policies being enacted. The average treatment effect on the treated after 2 years of exposure to the program (ATE2) is the difference in illiteracy prevalence for children in second grade at midline and baseline after controlling for any time effects between baseline and midline, as shown in Equation 1. 𝐴𝑇𝐸2 = (𝑃𝑡+2 2 − 𝑃𝑡 2 ᇣᇧ ᇤᇧ ᇥ) two year change in prevalence − (𝑃𝑡+2 1 − 𝑃𝑡 1 ᇣᇧ ᇤᇧ ᇥ) time effect (1) The total average treatment effect on literacy levels after two years of exposure to the program (TATE2) is the weighted average of the ATE2 after two years of exposure to the program and the indirect treatment effect on the untreated (ITE) after being exposed to the project between baseline and midline. 𝐼𝑇𝐸 = (𝑃𝑡+2 4 − 𝑃𝑡 4 ᇣᇧ ᇤᇧ ᇥ) spillover effect (2) 𝑇𝐴𝑇𝐸2 = 0.5 𝐴𝑇𝐸2 + 0.5 𝐼𝑇𝐸 (3) where  𝑃𝑡+2 2 is illiteracy prevalence of children in second grade in year 3 (midline)  𝑃𝑡 2 is illiteracy prevalence of children in second grade in year 1 (baseline)  𝑃𝑡+2 1 is illiteracy prevalence of children in first grade in year 3 (midline)  𝑃𝑡 1 is illiteracy prevalence of children in first grade in year 1 (baseline)  𝑃𝑡+2 4 is illiteracy prevalence of children in fourth grade in year 3 (midline)  𝑃𝑡 4 is illiteracy prevalence of children in fourth grade in year 1 (baseline) Exhibit 5: Example of Calculations: Two-Year Program Effects IMPAQ International, LLC Page H18 McGovern-Dole III Endline Evaluation Report  Threats arising from spillover effects will be investigated through comparison across cohorts in the same school.  Additionally, our evaluation will involve a substantial data collection on different variables. These variables will provide information and will be included in our specifications to control for other factors arising from students, families, teachers, caregivers, schools, and principals. Therefore, by taking advantage of the longitudinal data, the cohort implementation of the program, and a wide set of variables, our proposed quasi-experimental design is rigorous and allows us to mitigate many of the potential issues. Descriptive Performance Analysis In addition to these quasi-experimental methods, we will also conduct descriptive performance analysis using data collected from SILC member and father surveys that we will develop at midline in addition to the existing surveys from baseline evaluation, including teacher, school principal, SMC, student, and caregiver (mother) surveys (see Data Sources for more detail about the survey instruments). In addition, we will analyze the school attendance data provided to IMPAQ by CRS through its Monitoring and Evaluation (M&E) data collection system.  Descriptive analysis of Saving and Internal Lending Communities (SILC) members’ data: Based on the data collected using a short survey, we will provide descriptive characteristics on the operation of the SILC groups. For example, we will provide information on the average group size and composition, the amount of saving done by the group, the proportion of groups contributing to the school canteens, and the average financial contributions of these groups to the school canteens. We will complement the descriptive analyses with qualitative analysis of the SILC groups to establish the connection of these groups with the program’s objectives of improving child literacy and well-being.  Comparative analysis of mothers’ and fathers’ response data: In the midline evaluation, we will survey both fathers and mothers and elicit their responses on the same key performance indicators, such as their perceptions on the importance of education. We will also ask a series of questions on their degree of decision making in the household, especially in the realm of children’s education. In many households in which a child’s father and mother are present, fathers often act as the decision makers in family matters, including children’s schooling. By comparing the responses of mothers and fathers, we can analyze the differences on educational awareness within a household and the differences in decision making, and further link them to children’s educational outcomes. Simple correlations and regression analysis can help us assess the relation between children’s educational outcomes and their fathers’ and mothers’ responses. We can also perform t-tests to see whether the differences between fathers’ and mothers’ responses belonging to the same household are also statistically significantly different.  Performance analysis using school attendance data: IMPAQ will use the attendance data provided by CRS to assess the trends in school attendance after different periods of program maturity and across the two key study areas. We will also employ any existing attendance data prior to the start of the program in 2015 to make a pre–post comparison. Our analysis will critically IMPAQ International, LLC Page H19 McGovern-Dole III Endline Evaluation Report examine the quality of the data and make recommendations to CRS, if any, about improving the data collection methods such as verifying attendance data by visiting schools unannounced.  Qualitative Evaluation Design Like the baseline, our qualitative design will combine (1) a review, analysis, and synthesis of project data and documents and (2) a qualitative rapid-assessment approach using key informant interviews (KIIs) and focus group discussions (FGDs) with selected key project stakeholders and beneficiaries at both the national and community levels. For the midline evaluation, we are adding additional stakeholders such as administrative and education officials, project team and partners, and SILC members. We explain this in more detail in the section on sampling strategy. We will update the already-developed role-specific interview and focus group protocols to question the identified key informants about their perceptions of the project implementation process, the project management, and the lessons learned. For the baseline, protocols focused on collecting information on current project realities and perceptions. For the midline, we will include questions on effectiveness and sustainability (see the evaluation questions in Appendix A). Sampling Strategy Quantitative Sampling To implement a cohort comparison method, at baseline we sampled among schools where grades 1 through 4 were taught, there were no multi-grade classrooms for grades 1–4, and teachers taught only one grade (grades 1–4) per school. Following these criteria, we initially planned to sample 10 students in each grade 1–4 from 50 primary schools at baseline (2,160 students: 1,080 girls and 1,080 boys) which factored in a 20 percent attrition rate across data collection stages. In addition to students, we surveyed their mothers/caregivers, as well as their teachers, school principals, and SMCs in our sampled schools. For the midline evaluation, because three of our originally sampled 50 schools are closed due to terrorist attacks, we will visit the 47 remaining sampled schools for the midline evaluation, surveying approximately 470 students from Grades 1 to 4 (10 students, on average, per grade in each school), for a total sample of 1880 students (940 boys and 940 girls). Although we are surveying fewer number of students, our power calculations show that our minimum detectable effect increases only slightly from IMPAQ International, LLC Page H20 McGovern-Dole III Endline Evaluation Report baseline to midline evaluation.29,30 Therefore, we can use the remainder of our sample and implement the analysis as planned during the baseline evaluation. Following the baseline evaluation, in addition to surveying mothers, teachers, principals, and SMCs, we will also survey fathers of the students in our sample and SILC group members associated with these schools. Exhibit 6 contains the updated comprehensive list of the respondents, key information collected, and sampling strategy for the midline evaluation. Exhibit 6: Sampling Strategy for Midline Evaluation31 Respondent Key Information Collected Timeline Sample Strategy Status Students Reading abilities (Annual Status of Education Report [ASER]), student perceptions of learning environment, learning habits, hunger, minimum acceptable diet, health status, and hygiene knowledge and practices Baseline (2016) 540 each from grades 1– 4 Completed Midline (2018) 627 each from grades 1–4 Completed Endline (2020) Between 450 and 470 from grades 3 and 4 only To be determined Fathers and Mothers Demographic characteristics, hygiene knowledge and Baseline (2016) 2,279 households (only mothers) Completed 29 During the baseline evaluation, the minimum detectable effect was .091 percentage points for the illiteracy rate using 50 schools. Now, with 47 schools, our minimum detectable effect increases only slightly to .094 percentage points. These calculations are based on standard values for the level and power of the test (α = 0.05 and β = 0.8) and made under the assumption that the intra-cluster correlation (ρ) is 0.25. The baseline level average illiteracy rate for Mali for children in primary school was based on UNICEF data and estimated to be 74.6%. The standard deviation on the outcome variable was 0.195. 30 Data source: http://www.unicef.org/infobycountry/mali_statistics.html; http://mali.opendataforafrica.org/xtxxjx/mali￾education-outcomes 31 The numbers updated in this table are from the midline evaluation report IMPAQ International, LLC Page H21 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Status practices, food security status, education perceptions Midline (2018) 1,663 mothers/female caregivers and 802 fathers/male caregivers Completed Endline (2020) Between 900 and 940 parents (fathers and mothers) To be determined School Principals Pre- and in-service trainings, school management, teacher monitoring and oversight, hygiene knowledge and practices, school characteristics Baseline (2016) 50 school principals Completed Midline (2018) 44 school principals Completed Endline (2020) 47 school principals32 To be determined Teachers Pre- and in-service trainings, BLA teaching practices, hygiene knowledge and practices Baseline (2016) 185 teachers Completed Midline (2018) 189 teachers Completed Endline (2020) 47 teachers from grades 1, 2, 3, and 4 each33 To be determined School Management Roles and responsibilities, SMC Baseline (2016) 48 members Completed IMPAQ International, LLC Page H22 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Status Committee— Board Members management, school and canteen management, community contribution/ support for schools and canteens, hygiene knowledge and practices Midline (2018) 45 members Completed Endline (2020) 47 members (subsample of mothers) To be determined Saving and Internal Lending Communities Members Key activities in supporting school canteens and children, changes in financial capacity due to participation in groups, and roles and responsibilities of group members Baseline (2016) Not implemented Not implemented Midline (2018) 579 (subsample of caregivers 2,465) Completed Endline (2020) 51 random members drawn from the SILC groups To be determined Source: Authors’ calculations. Qualitative Sampling In collaboration with CRS, we will again select four sites to visit—two in Mopti and two in Koulikoro. At each site selected, we will conduct FGDs with students, parents, SMC members, and SILC members. Exhibit 5 summarizes the community-level sample. At baseline, women were underrepresented in both the parent and school management committee FGDs. These groups were also mixed-gender. For the midline evaluation, where possible, we would like to separate the focus groups by gender and increase the number of women who participate. Doing so will allow respondents to speak more freely and will also allow us to compare mothers’ and fathers’ perceptions. IMPAQ International, LLC Page H23 McGovern-Dole III Endline Evaluation Report Exhibit 7: Qualitative Sampling Strategy34 Respondent Key Information Collected Timeline Sample Strategy Status National stakeholders (KII) Project objectives, project alignment with other efforts, implementation barriers, and lessons learned for future efforts and sustainability Baseline (2016) 4 stakeholders: 1 MoE (Ministry of Education), 1 CNCS (National Centre for School Canteens), 1 CRS, and 1 EDC Completed Midline (2018) 2 stakeholders: 1 MONE (Ministry of Education), and 1 CNCS (National Centre for School Canteens) Completed Endline (2020) Determined in collaboration with CRS To be determined Administrative and education officials (KII) Targeting, policies, government capacity, level of participation and ownership Baseline (2016) N/A N/A Midline (2018) 8 stakeholders, including mayors and deputy mayors Completed Endline (2020) Determined in collaboration with CRS To be determined Project Team and Partners (KII) Implementation effectiveness, staffing structure, coordination mechanisms, lessons learned Baseline (2016) N/A N/A Midline (2018) 6 stakeholders, including 1 CRS, 1 EDC, 1 CARITAS Bamako, 1 Guamina, 1 AMPRODE, 1 CARITAS Mopti Completed Endline (2020) 4–8 respondents, determined in collaboration with CRS To be determined Students (FGD) Baseline (2016) 46 students: 24 girls / 22 boys from 4 schools Completed 34 The numbers updated in this table are from the midline evaluation report IMPAQ International, LLC Page H24 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Status Aspirations and attitudes toward their schools/teachers Midline (2018) N/A N/A Endline (2020) 40–48 students: 20–24 girls/ 20–24 boys from 4 schools To be determined Parents (FGD) Perceived quality of education, parental involvement, attendance, and aspirations for their children Baseline (2016) 50 parents: 12 women / 38 men from 4 schools Completed Midline (2018) 98 parents: 56 women /42 men from 4 schools 40–48 parents: 20–24 women /20–24 men from 4 schools Completed Endline (2020) 40–48 parents: 20–24 women /20–24 men from 4 schools To be determined School Management Committees (FGD) Roles and responsibilities, training, and accomplishments to date Baseline (2016) 22 members: 3 women / 19 men from 5 committees Completed Midline (2018) 27 members: 8 women / 19 men from 4 committees Completed Endline (2020) 16–24 members: 8–12 women / 8–12 men from 4 committees To be determined Mid-Term data collection: The evaluation team lead will prepare structured questionnaires, focus group discussion and key informant guides and observation tools, building on baseline tools and adding additional questions to inform the implementation strategy and as required by monitoring indicators. Information will be collected using structured interviews, focus groups discussions and key informant interviews. The evaluator will integrate questions on program perceptions to raise issues that may improve implementation for the final program years. Midline Evaluation Data Sources We will employ the following data sources to conduct the midline evaluation of FFE III: Surveys At the midline, we will use the survey instruments we employed at baseline. These include a student survey, mother survey, teacher and school principal survey, and SMC survey. Using the same instruments from the baseline will enable us to capture relevant changes at midline with respect to project relevance, effectiveness, efficiency, and early indications of sustainability and impact. We will use the quantitative IMPAQ International, LLC Page H25 McGovern-Dole III Endline Evaluation Report data to measure the progress of the program’s objectives and indicators. However, we will work closely with CRS to update the surveys by shortening them upon USDA’s request. In addition to the existing survey instruments, as described in detail in the Descriptive Performance Analysis section, we will also develop two more surveys for the midline evaluation: SILC member survey and father survey. Reading Assessment Similar to baseline, IMPAQ will also use the Annual Status of Education Report (ASER) reading test to measure students’ reading proficiency in grades 1–4. There is a possibility that either students have access to the test from their older cohorts or teachers have become aware of the test and started preparing students for the test. To avoid any possible bias in reading outcomes, IMPAQ will update the test content using the existing versions. To be able to compare students’ reading skills between baseline and midline, we will ensure that the updated test has the same level of complexity as the one used at the baseline. Before midline data collection, IMPAQ will also conduct an adaptation workshop with a group of local reading, curriculum, and assessment experts from the Ministry of Basic Education and Literacy (MENA). We will pretest the tool to ensure that the updated ASER test is still culturally appropriate and consistent with Mali’s learning standards for second grade. Key Informant Interview and Focus Group Discussion Data As in the baseline, we will collect primary data using a national KII protocol with project stakeholders and focus group discussion guides for parents, students, and SMC members. We will update the protocols to include items related to implementation of activities to date, perceived benefits of program activities, perceived capacity for sustainability, lessons learned, and recommendations for program improvement. For midline, we will add KIIs and FGDs with administrative and education officials, the project team (including partners), and SILC members. All protocols will include questions related to relevance, effectiveness, performance and impacts, and sustainability, as well as questions targeted to the respondents’ specific roles:  Administrative and education officials: What is their level of participation and ownership? Does intervention targeting and policy reflect this?  Project team and partners: What are their perceptions of implementation effectiveness, staffing structure, and coordination mechanisms?  SILC members: How do the groups improve individual financial capacity/household conditions and school/canteen conditions? What factors contribute to their longevity?  We will work closely with CRS to hire and train experienced enumerators, choosing, to the extent possible, enumerators who were involved in the baseline data collection. We will use the same platform for programming the survey instruments at baseline and follow the same quality assurance techniques used at baseline before, during, and after the data collection. IMPAQ International, LLC Page H26 McGovern-Dole III Endline Evaluation Report School Attendance Data In addition to collecting data, CRS will also provide us with school attendance data. We will first carefully assess the quality of the data, and then implement an appropriate descriptive performance analysis. We will also provide CRS recommendations for collecting this data in the future based on our data quality assessment. Evaluation team, management and coordination: CRS will engage an external consultant or consulting firm to conduct a qualitative and quantitative mid-term evaluation. The consultant will be the same as that which led the baseline study, if the first work s/he produced met expectations. In the event expectations are not met, a new competitive recruitment process will be completed in alignment with CRS and USDA policies, with selection based on professional competencies in the areas of primary education, as well as experience and knowledge of West Africa, Mali preferred. The successful candidate or firm will be financially and legally independent of CRS and its partners to independence and ensure evaluation validity. The recruitment process and the methodology finalization will be managed through the CRS Mali MEAL Department with support from the Regional Technical Advisor for MEAL and the CRS Mali Head of Programs. Partners and key stakeholders will be invited to participate throughout the process to ensure consensus, ownership and use of findings. Evaluation Key Audience: The midterm evaluation’s primary anticipated audiences are: - USDA: overall report on program performance to date - National government actors: overall report on program performance and share program findings on literacy interventions with any initial recommendations - Program implementation team: overall report on program performance and key recommendations, including mid-course corrections, to ensure attainment of program goals and objectives and support program sustainability - Targeted communities: feedback on program performance Evaluation Timeline: Per USDA’s M&E Policy, the midterm evaluation should be completed in March 2018. However, to capture project achievements to date against goals and objectives, CRS proposes to conduct the midterm evaluation in late March-April 2018 to coincide with the school year end. CRS will share evaluation documents, including the final TOR and evaluation methodology at least one month prior to the evaluation start. The following table outlines activities and tentative timeline: Activities Proposed Timeline Hold preparatory workshop on preliminary TOR and roles and responsibilities October 2017 Share the draft TOR with USDA November 2017 Share the TOR with USDA December 2017 Recruit/Confirm contracted evaluator January – February 2018 Refine evaluation methodology and assign roles and responsibilities February, 2018 Conduct data collection and analysis March -April 2018 Conduct stakeholder validation workshop May 2018 Submit draft mid-term evaluation report to CRS May 2018 Incorporation of CRS comments in a revised report May 2018 Submit revised mid-term evaluation report to USDA June 2018 IMPAQ International, LLC Page H27 McGovern-Dole III Endline Evaluation Report Incorporation of USDA‘s comments and submission of final report June 2018 Develop action plan to address findings and recommendations Within 15 days of receiving final report Report on implementation of follow-up actions Quarterly review & report Evaluation Deliverables: The key deliverables of the midterm evaluation are as follows:  The data collection tools in French version  The abstract of the final report  The Draft midterm evaluation report in English version including table of contents, executive summary, context, methodology, findings, lessons learned and recommendations as well as the table of performance indicators with updated values.  The Final midterm evaluation report in English version  Clean and final English versions of: - Quantitative data sets in Microsoft-Excel and any other utilized format (SPSS, STATA, etc.) - Qualitative transcripts, field and interview notes, complete list of key informant interviews and focus group discussions in Microsoft-Word document  A PowerPoint presentation of key evaluation findings to share with stakeholders Use of evaluation findings and recommendations: CRS will work with implementing partners and other stakeholders to develop a response to evaluation findings and recommendations. Project management will elaborate activities for each recommendation, identify responsible party for each action, the timeline and the responsible party to verify completion. The response plan and results will be reviewed at quarterly program coordination meetings. Review of the FFE III Final Evaluation Purpose and scope: The final evaluation aims to assess project achievement of expected results through an impact evaluation using the cohort comparison method as described in the baseline study, with consideration of the theory of change, implementation, management, lessons learned and sustainability. The final evaluation will build on the baseline and midterm studies to enhance learning and understanding of the project results as follows: assess project achievements against expected results outlined in the results framework, assess relevance, effectiveness, efficiency, impact and sustainability of activities, measure outcomes and assess attribution (to the extent possible based on counterfactual data), identify lessons learned, draw conclusions and provide recommendations for future early grade literacy and food assistance interventions. Context of the review: After the confirmation of the first Coronavirus Disease 2019 (COVID-19) pandemic case in Mali, the Government of Mali implemented measures to limit the spread of the virus. Effective March 20, the Government prohibited flights originating from countries with confirmed cases of COVID￾19, including the United States. Additionally, effective March 26, schools were closed until at least May 9, 2020; workshops, meetings, and gatherings larger than 50 people were banned; and the government IMPAQ International, LLC Page H28 McGovern-Dole III Endline Evaluation Report imposed a curfew from 9:00 pm until 5:00 am daily.35 The Government also noted that they might extend these restrictions to respond to the emerging situation. Due to these restrictions, collecting primary data for the final evaluation (as initially approved) might be no longer feasible. And given that the McGovern-Dole program implementation ends in September 2020, the project is running out of time and may not be able to conduct the final evaluation at all if the final evaluation methodology is not altered significantly. Working with the time the project has left, CRS in consultation with the consulting firm proposes to use secondary data analysis and remote qualitative key informant interviews for the final evaluation. Below, a detailed description is presented on changes to the initial methodology for the final evaluation, the shift in the original timeline, as well as the potential challenges in addressing the research questions and evaluation approach. Preliminary evaluation questions: The evaluation will consider questions specifically related to intervention relevance, implementation effectiveness, project efficiency, project achievements of results against initial targets and sustainability. The exhibit below provides a full list of final evaluation research questions and identifies which questions can be fully, partially, or not be addressed with the proposed final evaluation methodology. Addressing a question “partially” implies that the question could be only answered to the extent that the data allow, which could not be as comprehensive as expected compared to the original final evaluation plan. Relevance Status 1. Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? Fully 2. Are the activities and outputs of the program consistent with the intended impacts and effects? Fully 3. Does the program meet communities and government priorities? Partially 4. Are stakeholders (management committee, parents, teachers, local authorities) satisfied with their participation in the program? Why or why not? Partially 5. Does the project align with government policies and programs (local, national)? Does the project align, and compliment other donor, other NGO and/or local organizations managed programs? Fully 6. To what extent are the objectives of the McGovern Dole III intervention consistent with beneficiaries’ expectations, the country's needs, global priorities, political partners and USDA? Partially 35 https://ml.usembassy.gov/covid-19-information/ IMPAQ International, LLC Page H29 McGovern-Dole III Endline Evaluation Report 7. To what extent were the objectives of the program valid? Fully Effectiveness Status 8. To what extent were the objectives of McGovern Dole achieved / are likely to be achieved? Fully 9. What were the major factors influencing the achievement or non- achievement of the objectives? Fully 10. Were the implementation strategies relevant and effective enough to improve: 1) enrollment and attendance among pupils particularly girls? 2) Community participation and engagement? 3) A better learning environment? Are there more effective strategies that would have agreater impact? Partially 11. What are the project’s major limitations? Partially 12. Is the staffing structure and capacity sufficient and appropriate? Is the coordination mechanism effective? What if anything should be changed? Fully 13. Has program implementation been effectively monitored? How well did the monitoring and evaluation mechanism in place help the implementation of the project? Fully Performance and Impacts Status 14. What is the overall project outcome to date? To what extent have project objectives and the yearly benchmark indicators have been achieved? What is facilitating or not the achievement of results and objectives in a timely manner? Partially 15. What evidence suggests that the BLA has contributed to improved literacy? Partially 16. Have there been changes in students’ attendance, particularly among girls? Fully (if CRS provides data) 17. Is the incentive strategy effectively promoting student attendance? Are strategy modifications needed to improve attendance? Please explain. Partially (if including principal KIIs is feasible) 18. How has the project affected girls and boys? Is there an observable difference? What? Partially (depending on available quantitative data) 19. Have community barriers to education been identified? If so, how are they being addressed? How could the project better support behavior and social change? Partially 20. How are parents encouraged to be involved in their children’s education? How might they be encouraged to be more involved (including illiterate parents)? No 21. How has teacher attendance and motivation changed? What more could be done? Partially IMPAQ International, LLC Page H30 McGovern-Dole III Endline Evaluation Report 22. How do teachers find instructional materials? How are they using them? What could be done to promote greater/more effective use? No* 23. How are community-based structures (e.g. schools, SMC, SILC) supporting project implementation? Are they on track to assume ownership of key activities beyond the life of the project? Are they satisfied with their participation? How might they be encouraged and/or supported to participate more? Partially (indirectly from other stakeholders) * 24. How have capacity building activities for SMC improved their capacities? What obstacles persist? What more should be done to ensure they will have the capacity to manage the school canteens beyond the life of theproject? Partially (indirectly from other stakeholders) * 25. What innovations, lessons learned, and good practices can be documented so far? Partially 26. What has happened as a result of the McGovern Dole program and why? What real difference has the activity made to the beneficiaries? Partially 27. Did the theory of change to improve literacy through complementary support to student attendance, literacy instruction and student attentiveness hold? Why or why not? No Sustainability Status 28. What activities and/or outcomes (both expected and unexpected) of the program are likely to be sustained? What evidence is there to suggest this? Partially 29. What is the level of ownership acquired by the stakeholders? And how do they use? How can they evolve and / or continue the benefits resulting from the action after the end of the intervention? Partially 30. What are the major factors which can influence the achievement or non- achievement of the sustainability of the project? Partially 31. How do the government’s capacities, policies, procedures, and priorities contribute to sustainability? Fully 32. What strategies should be used to obtain long lasting support from communities and local/central administration that goes beyond the time of the project? Fully 33. How did capacity building enable community-based structures (e.g. schools, SMC, SILC) to support program implementation? To what degree of participation? Partially (indirectly from other stakeholders)* 34. How has local, regional and national capacity changed regarding literacy instruction in treatment schools? School feeding programs? Student enrollment and attendance monitoring? Is there evidence that their capacity and ability to provide quality programming has improved? Fully 35. How have the national capacities, policies, procedures and priorities changed? Fully IMPAQ International, LLC Page H31 McGovern-Dole III Endline Evaluation Report 36. What innovations, lessons learned, and good practices may be taken away from the project? Partially 37. How could outcome replication or scaling up be supported by future interventions? Fully Efficiency Status 38. Were objectives achieved on time? Fully 39. Was the project implemented in the most efficient way compared to alternatives? Fully 40. Does the food supply chain (including transport and storage) minimize loss and damages? Fully 41. Were resources managed in compliance to USG and USDA policies? Fully 42. Are activities cost-efficient? Are objectives achieving on time? Is the FFE implementing in the most efficient way compared to alternatives? (Efficiency and Value for Money) Fully Final Methodology: Given that collecting primary data is no longer viable due to the continued spread of COVID-19 and related restrictions, the external evaluator will implement secondary data analysis in addition to conducting remote key informant interviews. More specifically, the external evaluator would take the following steps:  Document Review. The external evaluator will conduct an extensive review of all relevant project documents. Appendix 1 outlines a full list of documents that CRS, EDC and other partners would provide.  Remote Qualitative Interviews. To the extent possible, the external evaluator will conduct KIIs remotely with national stakeholders, project staff and implementing partners, as well as administration and education officials.  Quantitative Analysis of Secondary Data. The external evaluator will assess the quality and relevance of any existing data they receive from the project team, in answering the research questions. Based on the assessment, there will be need to 1) identify a list of indicators that could be addressed and 2) analyze the performance of activities to expected results using the relevant data. CRS recognizes that secondary data analysis is different from the impact and performance evaluations that IMPAQ originally designed to conduct. However, to the extent possible, the external evaluator will use descriptive statistics and regression analysis. This will help identify common trends and patterns that emerge using the secondary data, without directly comparing them with baseline and/or midline evaluation data. Final evaluation data sources: The table below shows the data sources that could be employed under this contingency plan to analyze the five dimensions of the project’s achievements: (1) relevance; (2) effectiveness; (3) efficiency; (4) performance and impacts; and (5) sustainability. Criteria Data Sources IMPAQ International, LLC Page H32 McGovern-Dole III Endline Evaluation Report The external evaluator will work with existing data and conduct select remote qualitative interviews. As described above, the external evaluator will conduct remote key informant interviews with national stakeholders, project staff and implementing partners, as well as administration and education officials. However, it is anticipated that it will not be feasible to conduct focus group discussions with parents, SMCs, or SILCs due to inability to reach these stakeholders by phone, Skype or other virtual means. Also, the proposed methodology will not allow to fully answer some of the evaluation questions. As an example, for the research question corresponding to the relevance criteria, “Are stakeholders (management committee, parents, teachers, local authorities) satisfied with their participation in the program? Why or why not?” the external evaluator would likely only be able to interview local authorities and principals/teachers and therefore would not have responses from parents. The table below presents the revised qualitative sampling strategy. Qualitative Sampling Strategy Respondent Key Information Collected Timeline Sample Strategy Status National stakeholders (KII) Project objectives, project alignment with other efforts, implementation barriers, and lessons learned for future efforts and sustainability Baseline (2016) 4 stakeholders: 1 MoE (Ministry of Education), 1 CNCS (National Centre for School Canteens), 1 CRS, and 1 EDC Completed Midline (2018) 2 stakeholders: 1 MONE (Ministry of Education), and 1 Completed Relevance Qualitative: KIIs: Project Staff, National Government, Admin/Educational Officials Effectiveness Qualitative: KIIs: Project Staff, National Government, Admin/Educational Officials Performance and Impacts Quantitative: CRS and EDC secondary data Qualitative: KIIs: Project Staff, National Government, Admin/Educational Officials Sustainability Qualitative: KIIs: Project Staff, National Government, Admin/Educational Officials Efficiency Qualitative: KIIs: Project Staff, National Government, Admin/Educational Officials IMPAQ International, LLC Page H33 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Status CNCS (National Centre for School Canteens) Endline (2020) 2 stakeholders: 1 MONE (Ministry of Education), and 1 CNCS (National Centre for School Canteens) To be determined Administrative and education officials (KII) Targeting, policies, government capacity, level of participation and ownership Baseline (2016) N/A N/A Midline (2018) 8 stakeholders, including mayors and deputy mayors Completed Endline (2020) 6-8 stakeholders, including mayors and deputy mayors, also including principals as possible, which will be determined in collaboration with CRS To be determined Project Team and Partners (KII) Implementation effectiveness, staffing structure, coordination mechanisms, lessons learned Baseline (2016) N/A N/A Midline (2018) 6 stakeholders, including 1 CRS, 1 EDC, 1 CARITAS Bamako, 1 Guamina, 1 AMPRODE, 1 CARITAS Mopti Completed Endline (2020) 8 stakeholders, including 3 CRS, 1 EDC, 1 CARITAS Bamako, 1 Guamina, 1 AMPRODE, 1 CARITAS Mopti To be determined USDA respondent Project goals aligned with USDA’s priorities, overall successes, challenges and lessons learned, sustainability, and compliance with USDA policies Baseline (2016) NA NA Midline (2018) NA NA Endline (2020) 1 USDA To be dtermined IMPAQ International, LLC Page H34 McGovern-Dole III Endline Evaluation Report Respondent Key Information Collected Timeline Sample Strategy Status Teachers (KII) Attendance, motivation, instructional materials, satisfaction with BLA approach Endline (2020) 4 respondents, determined in collaboration with CRS, as feasible To be determined Parents – including SMC and SILC members (KII) Perceived quality of education, parental involvement, attendance, and aspirations for their children, Roles and responsibilities, training, and accomplishments to date Activities in relationship to financial capacity/ household conditions, school and canteen conditions, ownership and sustainability Endline (2020) 4-8 respondents, determined in collaboration with CRS, as feasible To be determined Additionally, this proposed methodology will not allow the evaluation team to assess and understand the four-year impact of the program on the learning achievement of children because of the inability to conduct primary survey data collection. Instead, the external evaluator will examine existing data to understand general patterns and trends in learning; but will not be able to assign causality to the observed changes. Evaluation team, management and coordination: As regards the recruitment of the external consultant, the selection will be on the basis of a job well done on the baseline study and the midterm evaluation, its renewal will depend of her/his past work. In the event prior work does not meet expectations, CRS will relaunch the recruitment process to identify a new consultant. The successful candidate or firm will be financially and legally independent of CRS and its partners to maintain independence and ensure evaluation validity. The recruitment process and methodology finalization will be managed through the CRS Mali MEAL Department with support from the Regional Technical Advisor for MEAL and the CRS Mali Head of Programs. Partners and stakeholders will participate to ensure consensus, ownership and use of the FFE III findings. Evaluation Key Audience: The final evaluation’s primary anticipated audiences are: - USDA: overall report on program performance to date - National government actors: overall report on program performance and share program findings on literacy interventions with any initial recommendations - Program implementation team: overall report on program performance - Targeted communities: feedback on program performance Evaluation Timelines: The table below shows the proposed new timeline as compared to the original timeline for the final evaluation. IMPAQ International, LLC Page H35 McGovern-Dole III Endline Evaluation Report Key stakeholders will be informed of the overall evaluation results. This will also be based on a participatory process to address quality, validity, utility and mutual ownership of the findings and recommendations. Challenges and Limitations: The proposed methodology has limitations and challenges, largely due to continued impacts from COVID-19, timing, and a potential lack of primary data collection. The following limitations and challenges are anticipated. Deliverables Originally Planned Proposed Timeline Document Review N/A June, 2020 Evaluation design plan with updated data collection tools March 31, 2020 June, 2020 Fieldwork activities, including training and data collection April 14 – 24, 2020 N/A Remote qualitative interviews N/A July, 2020 Secondary data management N/A July, 2020 Stakeholder validation workshop June 12, 2020 August, 2020 Endline evaluation package to CRS July 10, 2020 August, 2020 Endline evaluation report to USDA July 24, 2020 September, 2020 Potential Limitation/Challenges Mitigation Strategies Inability to attribute causal impact of the program on learning achievement If comparable learning outcome data from EDC are available for different grades and for baseline, midline, and endline, there might be a possibility to use the same quantitative evaluation design methodology to understand impact Inability to construct comparable performance indicators Instead of measuring changes over time, the external evaluator will aim to provide a holistic picture of program implementation using existing quantitative data and qualitative reports in order to offer recommendations for future implementation and sustainability IMPAQ International, LLC Page H36 McGovern-Dole III Endline Evaluation Report External Consultant/Firm Retention CRS will engage an external consultant/firm through an open solicitation. The successful candidate will be financially and legally independent of CRS and demonstrate relevant regional knowledge, analytical capacity, language skills and experience in evaluations for development education programs, with a preference given to previous experience evaluating school feeding and/or literacy programs. Other criteria to consider for the selection of the evaluation firm include experience working in Mali, French language proficiency, experience conducting process and impact evaluations, qualitative and quantitative data analysis, etc. The external consultant/firm identified through a competitive bid process during the baseline is envisioned to be retained during the subsequent midterm and final evaluations to ensure methodology consistency and planning. However, final retention for the midterm and the final evaluation is contingent on satisfactory completion of all deliverables during the preceding evaluation as assessed by CRS and USDA. Should an external consultant/firm not meet satisfactory completion of the preceding deliverables, CRS will open the selection process to a competitive bidding process as conducted during the baseline. Application Requirements Application must include a letter of motivation, CVs and an overview of the proposed work plan (2 pages), including the following aspects: 1. Initial reactions to this evaluation TOR 2. Suggested revised work plan and methodology 3. List of publications relevant to education, nutrition and microfinance Inability to address all research questions as planned for the endline evaluation The external evaluator will employ existing data creatively to answer the research questions as rigorously as possible, even though this limitation cannot be entirely mitigated Potential inability to conduct remote focus group discussions with parents, SMC members, and SILC members In collaboration with CRS, the external evaluator. will plan to add KIIs with principals as possible to get a school-level perspective over the phone. Any local interviews will be conducted by local consultant in Mali who speaks local languages Potential inability to conduct remote key informant interviews with teachers by phone If interviewing principals by phone is feasible, the external evaluator would develop their interview guide to also serve as a proxy for teachers IMPAQ International, LLC Page H37 McGovern-Dole III Endline Evaluation Report IMPAQ International, LLC Page H1 McGovern-Dole III Endline Evaluation Report Annex 1- Results Frameworks MGD 1.2: Improved Attentiveness MGD 2.1: Improved Knowledge of Safe Food Preparation and Storage Practices MGD 2.5: Increased Access to Preventative Health Interventions MGD 1.1.1: More Consistent Teacher Attendance MGD 1.1.3: Improved Literacy Instructional Materials MGD 1.1.4: Increased Skills & Knowledge of Teachers MGD 1.1.5: Increased Skills & Knowledge of School Administrators MGD 1.2.1: Reduced Short￾Term Hunger MGD 1.3.1: Increased Economic & Cultural Incentives (Or Decreased Disincentives) MGD 1.3.2: Reduced Health￾Related Absences MGD 1.3.3: Improved School Infrastructure MGD 1.3.4: Increased Student Enrollment MGD 1.3.5: Increased Community Understanding of benefits of education Output: Teacher recognition Output: Distribution: Improved Literacy Materials Output: Training: Teachers Output: Training: School Administrators MGD S02: Increased Use of Health & Dietary Practices CRS assist SMC to access funds through mayors’ office to meet school needs Output: Enrollment Campaigns Output: Training: School Management Committees Training: Commodity management, Food Preparation and Storage Practices Distribution: De￾worming medication, vitamins & minerals Output: Teacher recognition Output: Student Recognition Foundational Results MGD SO2: Increased Use of Health and Dietary Practices Output: Capacity Building: Local, regional, national level MGD 1.4.1: Increased Capacity of Government Institutions MGD 1.4.3: Increased Government Support Output: Capacity Building: Local, regional, national level Output: Take Home Rations MGD 1.4.4: Increased Engagement of Local Organizations and Community Groups Output: Form Savings and Lending Groups Output: Training: Government Officials Output: Training: School Management Committees MGD 1.2.1.1, 1.3.1.1: Increased Access to Food (School Feeding) Output: Provide School Meals MGD 1.1: Improved Quality of Literacy Instruction MGD 1.3: Improved Student Attendance MGD SO1: Improved Literacy of School-Age Children IMPAQ International, LLC Page H2 McGovern-Dole III Endline Evaluation Report Annex 2 - Key Project Indicators The key questions in Appendix 2 are based on the project’s performance indicators. For the indicators listed below, values must be reported at the baseline, midterm and final. CRS will provide the external consultant/firm with the USDA approved performance monitoring plan (PMP) that offers the definition, data source, and method/approach for these indicators. As with the evaluation plan, the PMP may be modified upon suggestion of the external consultant/firm or CRS and approval of USDA. MGD SO1: Improved Literacy of School-Age Children 1. Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (Male/Female) 2. Average number of days present to teach per teacher 3. Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance 4. Percent of girl students reporting they feel encouraged to participate in class by their teachers 5. Percent of boy students reporting they feel encouraged to participate in class by their teachers 6. Number of school administrators and officials in target schools who demonstrate use of new techniques or tools as a result of USDA assistance 7. Percent of students in target schools who are identified as "attentive" during class/instruction. 8. Percent of students in target schools who indicate that they are "not hungry" (Male/Female/Receives THR and School Meal/Receives only School Meal) 9. Number of students (males/females) regularly (80%) attending USDA supported classrooms/ schools (Male/Female). 10. Average number of days per student of school attended 11. Average number of days missed per student per school year due to student health issues (Male/Female) 12. Number of school improvement activities completed as a result of funding received from the local mayor's office 13. Percent of school-aged children enrolled in school (Male/Female) 14. Number of targeted school communities having benefited from community-level barrier analysis on barriers to school enrollment 15. Number of community-level barrier analyses conducted 16. Percent of community members demonstrating knowledge of educational benefits   MGD SO2: Increased Use of Health and Dietary Practices  17. Percent of SMC demonstrating knowledge of safe food preparation and storage practices as a result of USDA assistance 18. Avg. number of days missed per student per school year due to student health issues.  IMPAQ International, LLC Page H3 McGovern-Dole III Endline Evaluation Report Annex 3: List of Project Documents Project documents include but are not limited to:  The McGovern Dole project document, including terms of reference  Semi-annual and annual reports to US Department of Agriculture  Implementing partners Quarterly reports  Joint Mission Reports  Success stories  Different types of data, including o Total number of students in allschools o School attendance o Number of days of canteen’s functioning o Total number of class days o Community contributions o Distribution: Daily meals, THR, Vit A and deworming o Distribution of illustrated card o Saving and Internal Lending Community data o Distribution of teaching materials o Teacher’s recognition o Educational practices and follow-ups o Tutoring program o EGRA evaluation data (students and teachers)  Annual data quality assessment report  Rapid Assessment report in 2019  Earl Grade Reading Assessment evaluation reports  Performance Monitoring Plan and performance indicators IMPAQ International, LLC Page H4 McGovern-Dole III Endline Evaluation Report Annex 4: Attachment E - Performance Indicators Activity Indicator Target for FY 2016 Target for FY 2017 Target for FY 2018 Target for FY 2019 Target for FY 2020 Target for FY 2021 Capacity Building: Local, regional, national level Number of capacity building events organized for local, regional or national education structures as a result of USDA assistance (National Level; Trainings and Workshops) 1 2 2 2 1 0 Number of modules with a specific supplementary training material developed as a result of USDA assistance 1 2 2 0 0 0 Number of capacity building events organized for local, regional or national education structures as a result of USDA assistance (National Level; Monitoring Visits) 0 2 2 2 1 0 Number of local, regional or national education officials participating in sustainability events 0 35 35 35 35 0 Number of capacity building events organized for local, regional, or national education structures as a result of USDA assistance (District Level; Training, 0 01 01 02 02 0 IMPAQ International, LLC Page H5 McGovern-Dole III Endline Evaluation Report Meetings, and Workshops) Number of mayors’ offices staff trained 0 0 0 0 154 0 Distributio n: De￾worming medication , vitamins & minerals Number of students receiving deworming medication(s) 66,933 69,342 64,252 35,213 36,155 0 Number of students receiving Vitamin A tablets 66,933 69,342 64,252 35,231 36,155 0 Number of de￾worming treatments provided 66,933 138,684 96,378 70,426 72,310 0 Number of Vitamin A supplements provided 66,933 138,684 96,378 70,426 72,310 0 Enrolment campaigns Number of community-level barrier analyses conducted 0 264 251 13 13 0 Number of enrolment campaigns conducted 224 224 224 224 0 0 Number of students enrolled in schools receiving USDA assistance 66,933 69,342 64,252 73,193 75,023 0 Number of students enrolled in schools receiving USDA assistance (female) 34,002 35,226 32,649 36,977 37,901 0 IMPAQ International, LLC Page H6 McGovern-Dole III Endline Evaluation Report Number of students enrolled in schools receiving USDA assistance (male) 32,931 34,116 31,603 36,216 37,122 0 Number of students regularly (80%) attending USDA supported classrooms/schools 0 52,007 51,402 62,214 63,770 0 Number of students regularly (80%) attending USDA supported classrooms/schools (female) 0 28,181 26,119 31,430 32,216 0 Number of students regularly (80%) attending USDA supported classrooms/schools (male) 0 27,293 25,282 30,784 31,553 0 Number of target communities benefitting from community-level barrier analyses 0 264 251 0 0 0 Number of target communities benefitting from enrollment campaigns 0 224 224 224 0 0 Form Savings and Lending Groups Number of Savings and Internal Lending Community (SILC) groups supported as a result of USDA assistance 242 427 427 527 527 0 IMPAQ International, LLC Page H7 McGovern-Dole III Endline Evaluation Report Average amount of contribution per Savings and Internal Lending Community (SILC) group to school canteens (per year in USD). 10 12 15 17 20 0 Number of Savings and Internal Lending Community (SILC) groups contributing to their school canteen 171 192 213 256 300 0 Number of individuals actively participating in Savings and Internal Lending Community (SILC) groups as a result of USDA assistance 3,993 7,515 7,985 9,315 9,315 0 Number of household members benefitting from the creation of Savings and Internal Lending Community (SILC) groups formed as a result of USDA assistance 31,944 56,364 53,546 51,328 48,509 0 Number of Savings and Internal Lending Community (SILC) group exchange visits conducted 0 62 62 0 0 0 Number of new Savings and Internal Lending Community (SILC) groups created as a result of USDA assistance 0 185 0 100 0 0 IMPAQ International, LLC Page H8 McGovern-Dole III Endline Evaluation Report Number of Private Service Providers (PSP) trained and certified as a result of USDA assistance 0 37 0 0 0 0 Distributio n: Improved Literacy Materials Number of balanced literacy kits distributed to schools (Bamanankan) 25 11 22 11 11 0 Number of balanced literacy kits distributed to schools (Dogo-so) 0 4 8 4 4 0 Number of balanced literacy kits distributed to schools (French) 249 249 498 249 249 0 0 0 0 0 0 0 Number of schools receiving school supplies and materials as a result of USDA assistance 264 264 251 264 264 0 Number of students benefiting from the distribution of school supplies and materials 12,635 21,100 26,861 27,398 27,946 0 Number of teachers that have received literacy instructional materials as a result of USDA assistance 215 214 429 214 214 0 Number of textbooks and other teaching and learning materials provided as 2,365 2,354 4,719 2,354 2,354 IMPAQ International, LLC Page H9 McGovern-Dole III Endline Evaluation Report a result of USDA assistance 0 Provide School Meals Number of daily school meals (breakfast, snack, lunch) provided to school-age children as a result of USDA assistance 0 11,094,72 0 8,995 280 11,710,88 0 12,010,430 0 Number of individuals benefiting directly from USDA-funded interventions 0 69,342 67,538 76,560 79,540 0 Number of individuals benefiting directly from USDA-funded interventions (continuing) 0 0 57,986 59,274 67,432 0 Number of individuals benefiting directly from USDA-funded interventions (female) 0 35,226 33,975 38,375 39,930 0 Number of individuals benefiting directly from USDA-funded interventions (male) 0 34,116 33,563 38,185 39,610 0 Number of individuals benefiting directly from USDA-funded interventions (new) 0 69,342 9,552 17,286 12,108 0 Number of individuals 0 IMPAQ International, LLC Page H10 McGovern-Dole III Endline Evaluation Report benefiting indirectly from USDA-funded interventions 138,684 128,504 146,386 152,746 0 Number of school￾aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance 0 69,342 64,252 73,193 76,373 0 Number of school￾aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (continuing) 0 0 54,614 62,946 64,250 0 Number of school￾aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (female) 0 35,226 32,649 36,977 38,579 0 Number of school￾aged children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance (male) 0 34,116 31,603 36,216 37,794 0 Number of school￾aged children receiving daily school meals 0 69,342 9,638 10,247 11,853 IMPAQ International, LLC Page H11 McGovern-Dole III Endline Evaluation Report (breakfast, snack, lunch) as a result of USDA assistance (new) 0 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance 0 69,342 64,252 73,193 76,373 0 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (continuing) 0 0 54,614 62, 946 64,250 0 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (female) 0 35,226 32,649 36,977 38,579 0 Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance (male) 0 34,116 31,603 36,216 37,794 0 Number of social assistance beneficiaries participating in productive safety nets as a result of 0 69,342 9,638 10,247 11,853 0 IMPAQ International, LLC Page H12 McGovern-Dole III Endline Evaluation Report USDA assistance (new) Percent of school-age children receiving a minimum acceptable diet — 29% 30% 35% 0 Percent of school-age children receiving a minimum acceptable diet (female) 29% 30.50% 35.50 0 Percent of school-age children receiving a minimum acceptable diet (male) 28% 29.50 34.50 0 Total quantity of commodities (MT) provided for school meals as a result of USDA assistance 0 2,060 1,310 2,330 2,390 0 Student recognition Number of students whose parents received illustrated report cards distributed to literate and illiterate parents 0 69,342 64,252 68,093 69,795 0 Number of students who receive certificates that recognize academic achievement 0 5,280 5,020 5,280 5,280 0 Percent of parents who acknowledge receipt of their child's (or children's) report card 0 00 95 00 95 0 Take home rations Number of individuals receiving take-home rations as a result of USDA assistance 0 18,677 41,220 15,932 600 0 IMPAQ International, LLC Page H13 McGovern-Dole III Endline Evaluation Report Number of individuals receiving take-home rations as a result of USDA assistance (continuing) 0 0 12,555 12,725 526 0 Number of individuals receiving take-home rations as a result of USDA assistance (female students) 0 9,112 21,434 8,228 256 0 Number of individuals receiving take-home rations as a result of USDA assistance (male students) 0 8,574 19,786 7, 596 236 0 Number of individuals receiving take-home rations as a result of USDA assistance (new) 0 18,677 28,665 3,207 74 0 Number of individuals receiving take-home rations as a result of USDA assistance (Others) 0 991 918 2,798 2,824 0 Number of take￾home rations provided as a result of USDA assistance 0 195,914 201,470 206,124 178,636 0 Total quantity of commodities (MT) distributed as family rations to cooks as a 0 130 100 70 IMPAQ International, LLC Page H14 McGovern-Dole III Endline Evaluation Report result of USDA assistance 70 0 Total quantity of commodities (MT) provided for take￾home rations as a result of USDA assistance 0 130 300 340 230 0 Teacher recognition Number of Annual Results Presentations held 0 14 14 14 14 0 Number of teachers that receive performance awards at the Annual Results Presentations 40 40 40 40 40 0 Number of teachers who have received feedback from supervision visits 158 234 300 270 144 0 Percent of teachers who have received feedback from supervision visits 60 52 44 60 64 0 Training: Commodit y manageme nt, Food Preparation and Storage Practices Number of government staff in relevant ministries/offices trained in commodity management, food preparation and storage practices 14 14 0 0 0 0 Number of individuals trained in commodity management, food 1,324 1,324 0 280 280 IMPAQ International, LLC Page H15 McGovern-Dole III Endline Evaluation Report preparation and storage practices at the community-level 0 Number of individuals who receive Training-of￾Trainer training in commodity management, food preparation and storage practices 87 87 0 0 0 0 Number of school canteen cooks trained in safe food preparation and storage 0 991 502 582 0 0 Number of training sessions in commodity management, food preparation and storage practices conducted at the community-level 1 2 2 2 2 0 Training: Governme nt Officials Number of government officials certified as Teacher Trainers 0 36 0 0 0 0 Number of government officials trained in measuring literacy and using the Early Grade Reading Assessment (EGRA) tool 26 26 26 26 26 0 Number of trained government officials participating in the 26 26 26 26 26 IMPAQ International, LLC Page H16 McGovern-Dole III Endline Evaluation Report Early Grade Reading Assessment (EGRA) 0 Number of training sessions conducted for government officials on measuring literacy and using the Early Grade Reading Assessment (EGRA) tool 2 1 2 1 0 0 Number of training sessions conducted for Teacher Trainers on the Balanced Literacy Approach 2 2 2 1 1 0 Training: School Manageme nt Committee s Number of Action Plans created by School Management Committees as a result of USDA assistance 0 66 132 264 0 0 Number of Community Giant Scoreboards created as a result of USDA assistance 0 264 0 13 0 0 Number of matching grants awarded to eligible School Management Committees 0 0 66 88 44 0 Number of Parent￾Teacher Associations (PTAs) or similar "school" governance structures supported 264 264 251 291 291 0 IMPAQ International, LLC Page H17 McGovern-Dole III Endline Evaluation Report as a result of USDA assistance Number of School Management Committee members trained on MONE modules 0 1,324 1,324 200 200 0 Value of public and private sector investments leveraged as a result of USDA assistance 390,377 129,600 217,013 304,544 296,475 0 Value of public and private sector investments leveraged as a result of USDA assistance (Host Government – national and local) 388,667 23,760 68,640 107,580 47,520 0 Value of public and private sector investments leveraged as a result of USDA assistance (Other Public including SMC and SILC groups cash contribution) 1,710 105,840 148,373 196,964 248,955 0 Training: School Administra tors Number of school administrators and officials in target schools who demonstrate use of new techniques or 34 80 149 195 225 0 IMPAQ International, LLC Page H18 McGovern-Dole III Endline Evaluation Report tools as a result of USDA assistance Number of school administrators and officials trained or certified as a result of USDA assistance 319 311 303 319 319 0 Number of training sessions for school administrators conducted as a result of USDA assistance 2 2 1 1 1 0 Training: Teachers Number of teachers/educators/te aching assistants in target schools who demonstrate use of new and quality teaching techniques or tools 53 108 216 180 108 0 Number of teachers/educators/te aching assistants trained or certified as a result of USDA assistance 264 452 678 452 226 0 Number of training sessions provided to teachers as a result of USDA assistance 2 3 4 2 1 0 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text 2 5 10 15 20 0 Percent of students who, by the end of 2 5 10 15 20 0 IMPAQ International, LLC Page H19 McGovern-Dole III Endline Evaluation Report two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (female) Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (male) 2 5 10 15 20 0 Training: Literacy Assistance Volunteers Number of literacy assistance volunteers trained 0 0 0 324 324 0 Number of Grade 1-3 students supported through the literacy assistance program 0 0 0 2,592 2,592 0 Percent of students receiving literacy assistance who attain the mean score of his/her class in reading 0 0 0 50% 50% 0